You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "kou (via GitHub)" <gi...@apache.org> on 2024/08/05 09:28:17 UTC
[PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
kou opened a new pull request, #43553:
URL: https://github.com/apache/arrow/pull/43553
### Rationale for this change
Statistics are useful for fast query processing. Many query engines
use statistics to optimize their query plan.
Apache Arrow format doesn't have statistics but other formats that can
be read as Apache Arrow data may have statistics. For example, Apache
Parquet C++ can read Apache Parquet file as Apache Arrow data and
Apache Parquet file may have statistics.
One of the Arrow C data interface use cases is the following:
1. Module A reads Apache Parquet file as Apache Arrow data
2. Module A passes the read Apache Arrow data to module B through the
Arrow C data interface
3. Module B processes the passed Apache Arrow data
If module A can pass the statistics associated with the Apache Parquet
file to module B through the Arrow C data interface, module B can use
the statistics to optimize its query plan.
### What changes are included in this PR?
Add the specification to pass statistics through the Arrow C data interface based on the discussion on the `dev@` mailing list: https://lists.apache.org/thread/z0jz2bnv61j7c6lbk7lympdrs49f69cx
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1706056246
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. Producers
+passes statistics as a map Arrow array that uses the schema through
+the Arrow C data interface.
+
+Here is the schema for a statistics map Arrow Array:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``int32``
+ - ``true``
+ - The column index or null if the statistics refer to whole table
+ or record batch.
Review Comment:
```suggestion
- The column index, or null if the statistics refer to whole
table or record batch.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1772916746
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,218 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`ipc-message-format
+metadata for each message`. If you're interested in the specification
+for passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ values: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``values``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``values``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
Review Comment:
`ARROW:null_count:{exact,approximate}` are missing.
I'll add them later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1896959199
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,339 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+Use case
+--------
+
+One of :ref:`c-stream-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C stream interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C stream interface, module B can
+use the statistics to optimize its query plan.
+
+For example, DuckDB uses this approach but DuckDB couldn't use
+statistics because there wasn't the standardized way to pass
+statistics.
+
+.. seealso::
+
+ `duckdb::ArrowTableFunction::ArrowScanBind() in DuckDB 1.1.3
+ <https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L373-L403>`_
+
+Goals
+-----
+
+TODO: Remove the C data interface limitation?
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+TODO: Remove the C data interface limitation?
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+TODO: Should we deprecate the current ADBC's statistics API and
+redesign with this specification?
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+
+ The column index is computed as the same rule used by
+ :ref:`ipc-recordbatch-message`.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
Review Comment:
Thanks!
This is also pointed out by @felipecrv :
* https://github.com/apache/arrow/pull/45058#pullrequestreview-2520677048
* https://github.com/apache/arrow/pull/45058#discussion_r1895832598
https://github.com/apache/arrow/pull/45058 uses "statistics name" instead of "statistics key".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2550534233
Based on the mailing list discussion, we don't limit this specification to only the Arrow C data interface.
We standardize only statistics schema.
I close this in favor of GH-45058.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2516238797
Revision: c1f0e2de42fc20cdb97fde06e50db73e0cba41e1
Submitted crossbow builds: [ursacomputing/crossbow @ actions-e43e80bd7f](https://github.com/ursacomputing/crossbow/branches/all?query=actions-e43e80bd7f)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/12153724789/job/33892278091)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1704373291
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
Review Comment:
What about the Arrow IPC format? Can you add a sentence here that explains why we do not recommend using this to pass statistics over Arrow IPC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2272566705
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1731509894
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,219 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`ipc-message-format
+metadata for each message`. If you're interested in the specification
+for passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ map<
+ key: int32,
+ items: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
Review Comment:
That looks reasonable to me!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed pull request #43553: GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface
URL: https://github.com/apache/arrow/pull/43553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1877531987
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
Review Comment:
(Sorry... I couldn't work on this today... I hope that I can work on this tomorrow...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2516236511
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1879550285
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
Review Comment:
I chose this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1876534342
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
Review Comment:
I think it's fair to mention that this proposal doesn't cover the API used to request these arrays or the transport mechanism used in that API. Mentioning any particular transport/request mechanism is probably more confusing than helpful (and likely to change following this proposal as formalizing that request/transport will be valuable!).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2269505186
@pdet please take a look and add comments if you have any, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1896967602
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,339 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+Use case
+--------
+
+One of :ref:`c-stream-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C stream interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C stream interface, module B can
+use the statistics to optimize its query plan.
+
+For example, DuckDB uses this approach but DuckDB couldn't use
+statistics because there wasn't the standardized way to pass
+statistics.
+
+.. seealso::
+
+ `duckdb::ArrowTableFunction::ArrowScanBind() in DuckDB 1.1.3
+ <https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L373-L403>`_
+
+Goals
+-----
+
+TODO: Remove the C data interface limitation?
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+TODO: Remove the C data interface limitation?
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+TODO: Should we deprecate the current ADBC's statistics API and
+redesign with this specification?
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+
+ The column index is computed as the same rule used by
+ :ref:`ipc-recordbatch-message`.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by type code not name. So
+ producers can use any valid name for fields.
+
+ TODO: Should we standardize field names?
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for individual statistics instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+For example, you can use your product name as namespace
+such as ``MY_PRODUCT:my_statistics:exact``.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float64``
+ - The average size in bytes of a row in the target column. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``: TODO: Should we use ``int64`` instead?
+ - The average size in bytes of a row in the target column. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target column. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target column. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target column. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target column. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target column. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target column. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target column. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target column. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target column. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target column. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
+
+If you find a missing statistics key that is usable for multiple
+systems, please propose it on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Examples
+========
+
+Here are some examples to help you understand.
+
+Simple record batch
+-------------------
+
+Schema::
+
+ vendor_id: int32
+ passenger_count: int64
+
+Data::
+
+ vendor_id: [5, 1, 5, 1, 5]
+ passenger_count: [1, 1, 2, 0, null]
+
+Statistics schema::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<int64>
+ >
+ >
+
+Statistics array::
Review Comment:
Let's continue this in the current PR: https://github.com/apache/arrow/pull/45058/files#r1896967442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2272515336
> I was just slightly confused for one second that the second mapping is the value `items` in the first mapping.
Ah, it makes sense. I'll improve it. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2268606727
:warning: GitHub issue #38837 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2518886012
Revision: 1cdc8b1deb5fbddc5aaa5a01befe53584f7fff79
Submitted crossbow builds: [ursacomputing/crossbow @ actions-39aa7166b5](https://github.com/ursacomputing/crossbow/branches/all?query=actions-39aa7166b5)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/12171415682/job/33948264186)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1871755575
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
Review Comment:
Do we really want this? What's the point of allowing arbitrary field names? This will make interoperability more difficult.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
Review Comment:
Why is this float64? It doesn't seem to make sense. The maximum is obviously an integer.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
Review Comment:
The C Data Interface already exposes the `null_count`, what use is this for?
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
Review Comment:
"the target" is the array, right?
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
Review Comment:
"By index"? Do you mean by data type instead?
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
Review Comment:
We should perhaps explain what the "column index" means when the data is nested.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
Review Comment:
Is "float" different from "float64"?
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
Review Comment:
What does "variants" mean here? Individual statistics?
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
Review Comment:
What does "instead of using values" means here?
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
Review Comment:
So there can be user-defined statistics? This document should be clear about it.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
Review Comment:
What's the point? The length is already known.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
+
+If you find a missing statistics key that is usable for multiple
+systems, please propose it on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Examples
+========
+
+Here are some examples to help you understand.
+
+C++
Review Comment:
This should not be part of the general spec, can it be removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1872669437
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
Review Comment:
Maybe we should combine it with the comment about defining "column index" above...we could extend the numbering to cover nested columns (possibly down the line)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1879556583
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
Review Comment:
Let's use it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2516382656
http://crossbow.voltrondata.com/pr_docs/43553/format/CDataInterfaceStatistics.html
I've added an example in C++. I'll start a vote for this proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1879551959
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1871749972
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
Review Comment:
I think what we have actually defined here is a way to represent statistics as an Arrow array (i.e., there's no real connection to the C data interface...these can and should be Julia arrays, C++ arrays, Arrow C interface arrays, serialized as IPC, as appropriate).
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
Review Comment:
I'm not sure I would change this sentence, but as a personal note I hope that we do unify the two approaches. In the short term I suppose ADBC can have logic in its driver and client APIs to translate between the two.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
Review Comment:
Perhaps out of scope for this proposal, but this wouldn't allow for statistics of a nested column? (Unless the value type of something like `ARROW:xxx:max_value` could be a struct for a nested column). (We use nested column statistics in GeoParquet to do filter pushdown on points).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1706048953
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
Review Comment:
```suggestion
* Establish a standard way to pass statistics through the Arrow C data
interface.
* Provide this in a manner that enables compatibility and ease of
implementation for existing users of the Arrow C data interface.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1706056246
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. Producers
+passes statistics as a map Arrow array that uses the schema through
+the Arrow C data interface.
+
+Here is the schema for a statistics map Arrow Array:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``int32``
+ - ``true``
+ - The column index or null if the statistics refer to whole table
+ or record batch.
Review Comment:
```suggestion
- The zero-based column index, or null if the statistics
describe the whole table or record batch.
```
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. Producers
+passes statistics as a map Arrow array that uses the schema through
+the Arrow C data interface.
+
+Here is the schema for a statistics map Arrow Array:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``int32``
+ - ``true``
+ - The column index or null if the statistics refer to whole table
+ or record batch.
Review Comment:
The index is zero-based, correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "Tmonster (via GitHub)" <gi...@apache.org>.
Tmonster commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2273382769
> @Tmonster I think the proposed `Statistics keys` already cover the table statistics we use/produce. Could you double-check?
DuckDB has equivalent statistics for the distinct counts, row counts, min value, and max value. So looks good to me 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1728163687
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,219 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`ipc-message-format
+metadata for each message`. If you're interested in the specification
+for passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ map<
+ key: int32,
+ items: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
Review Comment:
An (one element) array of maps. But we can simplify this. I didn't notice it. Thanks.
How about the following? It removes the outer `map`:
column: int32,
values: map<
key: dictionary<
indices: int32,
dictionary: utf8
>,
items: dense_union<...all needed types...>,
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1776958001
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,218 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`ipc-message-format
+metadata for each message`. If you're interested in the specification
+for passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ values: map<
Review Comment:
`values` isn't a good name because it's a map array of statistics key and statistics value pairs.
`statistics` (or `items`) may be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2268609841
I'm not a native English speaker. Wording suggestions are very welcome.
I'll add examples after I implement a convenient API to C++.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1706058360
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. Producers
+passes statistics as a map Arrow array that uses the schema through
+the Arrow C data interface.
+
+Here is the schema for a statistics map Arrow Array:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``int32``
+ - ``true``
+ - The column index or null if the statistics refer to whole table
+ or record batch.
+ * - items
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separated table for details.
+
+Here is the schema for the statistics map:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<int32, utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. See also the separated description for
+ statistics key.
Review Comment:
```suggestion
approximate value. Also see the separate description below for
statistics key.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1706058786
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. Producers
+passes statistics as a map Arrow array that uses the schema through
+the Arrow C data interface.
+
+Here is the schema for a statistics map Arrow Array:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``int32``
+ - ``true``
+ - The column index or null if the statistics refer to whole table
+ or record batch.
+ * - items
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separated table for details.
+
+Here is the schema for the statistics map:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<int32, utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. See also the separated description for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. See also the separated description for
+ statistics key.
Review Comment:
```suggestion
byte width statistic. Also see the separate description below
for statistics key.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1731960160
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,219 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`ipc-message-format
+metadata for each message`. If you're interested in the specification
+for passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ map<
+ key: int32,
+ items: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
Review Comment:
I've updated this part!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1872876442
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
Review Comment:
It doesn't make sense for the maximum approximation to be float64. The fractional part is entirely nonsensical.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2537553771
The discussion thread: https://lists.apache.org/thread/b6chzlyn95rztoybs39b6olz907g12gj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1706308428
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. Producers
+passes statistics as a map Arrow array that uses the schema through
+the Arrow C data interface.
+
+Here is the schema for a statistics map Arrow Array:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``int32``
+ - ``true``
+ - The column index or null if the statistics refer to whole table
+ or record batch.
+ * - items
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separated table for details.
+
+Here is the schema for the statistics map:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<int32, utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. See also the separated description for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. See also the separated description for
+ statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
Review Comment:
They are the same as ADBC's one: https://github.com/apache/arrow-adbc/blob/05fa60d643c66b572d426ab28aa78fc52e9520e8/c/include/arrow-adbc/adbc.h#L524-L570
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. Producers
+passes statistics as a map Arrow array that uses the schema through
+the Arrow C data interface.
+
+Here is the schema for a statistics map Arrow Array:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``int32``
+ - ``true``
+ - The column index or null if the statistics refer to whole table
+ or record batch.
Review Comment:
Correct.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
Review Comment:
Thanks. I should have done it.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
Review Comment:
Good point. This may fit the Arrow IPC format. (A producer sends data and statistics as 2 separated the Arrow IPC format data.)
But the Arrow IPC format can use more approaches. For example, the Arrow IPC format can have metadata for each record batch data: https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format (The Arrow C data can't have metadata for `ArrowArray`.)
The Arrow IPC format can be used with other mechanisms such as Arrow Flight and ADBC.
So this may not be the best approach for the Arrow IPC format. We should discuss this use case with the Arrow IPC format separately.
I'll add something to here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1706052981
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. Producers
+passes statistics as a map Arrow array that uses the schema through
+the Arrow C data interface.
Review Comment:
```suggestion
This specification provides only the schema for statistics. The
producer passes statistics through the Arrow C data interface as an
Arrow map array that uses this schema.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2518883645
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2516563764
Ah, should we have one more implementation before a vote for this proposal...?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1872670934
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
Review Comment:
I think the point is that consumers shouldn't rely on particular field names, given the index/type is supposed to fully define it already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1875486051
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
Review Comment:
I think the confusion is that I think Kou is trying to say that this isn't the "one true way" to embed statistics inside Flight or an IPC file (or in other words, this isn't a spec for adding Parquet-like statistics to IPC), but of course there's nothing stopping you from using those APIs to transmit statistics data for some other purpose. (Correct me if I'm wrong here @kou.) Again, because the original motivation was the DuckDB use case and so trying to solve "statistics for IPC files" was out of scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1875498336
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
Review Comment:
Thanks! It's correct.
(I'll be able to work on this proposal tomorrow. I'll update the current proposal and start a discussion on the mailing list tomorrow.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2534647955
> I think the C Data Interface framing came from the original use case (which I would like to not lose sight of); allowing engines like DuckDB to have a way to get statistics when given a C Data Interface stream, so that they can properly do query planning. Even if we reframe this as just a definition for the format of the statistics it might be good to mention the original use case so there's proper context, to help commenters who aren't familiar with query planning.
I've added the original DuckDB use case.
DuckDB may be able to get statistics without `ArrowArrayStream` (DuckDB may be able to call separate API to get statistics) because `duckdb::TableFunction` has `table_function_cardinality_t cardinality` and `table_statistics_t statistics`.
See also: https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L525-L527
I'll start a discussion on the mailing list tomorrow.
(I wanted to do it today...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "pdet (via GitHub)" <gi...@apache.org>.
pdet commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2270654634
The format and contents LGTM! I was just slightly confused for one second that the second mapping is the value `items` in the first mapping.
@Tmonster I think the proposed `Statistics keys` already cover the table statistics we use/produce. Could you double-check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2272566945
The preview URL: http://crossbow.voltrondata.com/pr_docs/43553/format/CDataInterfaceStatistics.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2272568117
Revision: 64dea6b68d7b64c62712020589303fdd055e5d67
Submitted crossbow builds: [ursacomputing/crossbow @ actions-07dffbaf08](https://github.com/ursacomputing/crossbow/branches/all?query=actions-07dffbaf08)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/10277870502/job/28440588437)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1872885905
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
Review Comment:
Any Arrow array can be transmitted over Flight or IPC, including statistics arrays. I agree with @lidavidm about explaining the context and original motivation, but I don't think we should exclude IPC or Flight here. People can use statistics arrays over these protocols if they find it useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1896964885
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,339 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+Use case
+--------
+
+One of :ref:`c-stream-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C stream interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C stream interface, module B can
+use the statistics to optimize its query plan.
+
+For example, DuckDB uses this approach but DuckDB couldn't use
+statistics because there wasn't the standardized way to pass
+statistics.
+
+.. seealso::
+
+ `duckdb::ArrowTableFunction::ArrowScanBind() in DuckDB 1.1.3
+ <https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L373-L403>`_
+
+Goals
+-----
+
+TODO: Remove the C data interface limitation?
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+TODO: Remove the C data interface limitation?
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+TODO: Should we deprecate the current ADBC's statistics API and
+redesign with this specification?
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+
+ The column index is computed as the same rule used by
+ :ref:`ipc-recordbatch-message`.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by type code not name. So
+ producers can use any valid name for fields.
+
+ TODO: Should we standardize field names?
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for individual statistics instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+For example, you can use your product name as namespace
+such as ``MY_PRODUCT:my_statistics:exact``.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float64``
+ - The average size in bytes of a row in the target column. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``: TODO: Should we use ``int64`` instead?
+ - The average size in bytes of a row in the target column. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target column. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target column. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target column. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target column. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target column. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target column. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target column. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target column. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target column. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target column. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
+
+If you find a missing statistics key that is usable for multiple
+systems, please propose it on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Examples
+========
+
+Here are some examples to help you understand.
+
+Simple record batch
+-------------------
+
+Schema::
+
+ vendor_id: int32
+ passenger_count: int64
+
+Data::
+
+ vendor_id: [5, 1, 5, 1, 5]
+ passenger_count: [1, 1, 2, 0, null]
+
+Statistics schema::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<int64>
+ >
+ >
+
+Statistics array::
Review Comment:
Ah, it makes sense.
I used physical representation (columnar representation) here but logical representation (row based representation) may be easy to understand.
How about showing both of them? (We keep the current representation and add row based representation like you suggested.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1896745819
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,339 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+Use case
+--------
+
+One of :ref:`c-stream-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C stream interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C stream interface, module B can
+use the statistics to optimize its query plan.
+
+For example, DuckDB uses this approach but DuckDB couldn't use
+statistics because there wasn't the standardized way to pass
+statistics.
+
+.. seealso::
+
+ `duckdb::ArrowTableFunction::ArrowScanBind() in DuckDB 1.1.3
+ <https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L373-L403>`_
+
+Goals
+-----
+
+TODO: Remove the C data interface limitation?
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+TODO: Remove the C data interface limitation?
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+TODO: Should we deprecate the current ADBC's statistics API and
+redesign with this specification?
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+
+ The column index is computed as the same rule used by
+ :ref:`ipc-recordbatch-message`.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by type code not name. So
+ producers can use any valid name for fields.
+
+ TODO: Should we standardize field names?
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
Review Comment:
Nit is I would find the term "statistics type" rather than "statistics key" easier to understand
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,339 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+Use case
+--------
+
+One of :ref:`c-stream-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C stream interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C stream interface, module B can
+use the statistics to optimize its query plan.
+
+For example, DuckDB uses this approach but DuckDB couldn't use
+statistics because there wasn't the standardized way to pass
+statistics.
+
+.. seealso::
+
+ `duckdb::ArrowTableFunction::ArrowScanBind() in DuckDB 1.1.3
+ <https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L373-L403>`_
+
+Goals
+-----
+
+TODO: Remove the C data interface limitation?
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+TODO: Remove the C data interface limitation?
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+TODO: Should we deprecate the current ADBC's statistics API and
+redesign with this specification?
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+
+ The column index is computed as the same rule used by
+ :ref:`ipc-recordbatch-message`.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
Review Comment:
Maybe we could be more explicitly about what the key means (it seems like it is the "statistics type" )?
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,339 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+Use case
+--------
+
+One of :ref:`c-stream-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C stream interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C stream interface, module B can
+use the statistics to optimize its query plan.
+
+For example, DuckDB uses this approach but DuckDB couldn't use
+statistics because there wasn't the standardized way to pass
+statistics.
+
+.. seealso::
+
+ `duckdb::ArrowTableFunction::ArrowScanBind() in DuckDB 1.1.3
+ <https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L373-L403>`_
+
+Goals
+-----
+
+TODO: Remove the C data interface limitation?
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+TODO: Remove the C data interface limitation?
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+TODO: Should we deprecate the current ADBC's statistics API and
+redesign with this specification?
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+
+ The column index is computed as the same rule used by
+ :ref:`ipc-recordbatch-message`.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by type code not name. So
+ producers can use any valid name for fields.
+
+ TODO: Should we standardize field names?
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for individual statistics instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+For example, you can use your product name as namespace
+such as ``MY_PRODUCT:my_statistics:exact``.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float64``
+ - The average size in bytes of a row in the target column. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``: TODO: Should we use ``int64`` instead?
+ - The average size in bytes of a row in the target column. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target column. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target column. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target column. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target column. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target column. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target column. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target column. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target column. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target column. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target column. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
+
+If you find a missing statistics key that is usable for multiple
+systems, please propose it on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Examples
+========
+
+Here are some examples to help you understand.
+
+Simple record batch
+-------------------
+
+Schema::
+
+ vendor_id: int32
+ passenger_count: int64
+
+Data::
+
+ vendor_id: [5, 1, 5, 1, 5]
+ passenger_count: [1, 1, 2, 0, null]
+
+Statistics schema::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<int64>
+ >
+ >
+
+Statistics array::
Review Comment:
I didn't understand this example. I thought the statistics were structs, so I would have expected the data to look something like this (perhaps we could give the "logical contents" and then the specific array encoding):
```
[
// first struct element
{
column: null, # record batch
statistics: {
"ARROW:row_count:exact": 0
}
},
{
column: 0, # vendor_id
statistics: {
"ARROW:null_count:exact": 0,
"ARROW:distinct_count:exact": 2,
"ARROW:max_value:exact": 5,
"ARROW:min_value:exact": 1,
}
},
...
]
```
I can help work out the example if people think this is a good idea
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,339 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+Use case
+--------
+
+One of :ref:`c-stream-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C stream interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C stream interface, module B can
+use the statistics to optimize its query plan.
+
+For example, DuckDB uses this approach but DuckDB couldn't use
+statistics because there wasn't the standardized way to pass
+statistics.
+
+.. seealso::
+
+ `duckdb::ArrowTableFunction::ArrowScanBind() in DuckDB 1.1.3
+ <https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L373-L403>`_
+
+Goals
+-----
+
+TODO: Remove the C data interface limitation?
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+TODO: Remove the C data interface limitation?
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+TODO: Should we deprecate the current ADBC's statistics API and
+redesign with this specification?
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+
+ The column index is computed as the same rule used by
+ :ref:`ipc-recordbatch-message`.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by type code not name. So
+ producers can use any valid name for fields.
+
+ TODO: Should we standardize field names?
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for individual statistics instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+For example, you can use your product name as namespace
+such as ``MY_PRODUCT:my_statistics:exact``.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float64``
+ - The average size in bytes of a row in the target column. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``: TODO: Should we use ``int64`` instead?
+ - The average size in bytes of a row in the target column. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target column. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target column. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target column. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target column. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target column. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target column. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target column. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target column. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target column. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target column. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
+
+If you find a missing statistics key that is usable for multiple
+systems, please propose it on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Examples
+========
+
+Here are some examples to help you understand.
+
+Simple record batch
+-------------------
+
+Schema::
+
+ vendor_id: int32
+ passenger_count: int64
+
+Data::
+
+ vendor_id: [5, 1, 5, 1, 5]
+ passenger_count: [1, 1, 2, 0, null]
+
+Statistics schema::
Review Comment:
It is probably too late, but I figured I would point out another format we use for statistics in DataFusion is a columnar form rather than a nested structure
https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/trait.PruningStatistics.html
For example to represent this data in DataFusion it would be
| `vendor_id::null_count` | `vendor_id::min` | `vendor_id::max` | ... | `passenger_count::max` |
|--------|--------|--------|--------|--------|
| 0 | 1 | 5 | ... | 2 |
The benefit of this encoding is that it can be used to quickly evaluate predicates on ranges (e.g. figure out if `vendor_id = 6` could ever be true)
We use this format to read statistics from the parquet files (see [`ParquetStatisticsConverter`](https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/statistics/struct.StatisticsConverter.html) to rule out row groups)
It does result in potentially very wide schemas however
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
Review Comment:
We have had trouble in the arrow-rs implementation or other places where schema's contain names but they aren't standardized
For example, we have an embedded field name for `List`s and some implementations use `"item"` and some `"element"` which causes spurious schema mismatch errors
Therefore I also recommend removing field names unless there is some good reason for doing so (it isn't clear to me from the text why there are arbitrary field namds)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1704320155
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
Review Comment:
```suggestion
1. Module A reads Apache Parquet file as Apache Arrow data.
2. Module A passes the read Apache Arrow data to module B through the
Arrow C data interface.
3. Module B processes the passed Apache Arrow data.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1704317397
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
Review Comment:
```suggestion
be read as Apache Arrow data may have statistics. For example, the
Apache Parquet C++ implementation can read an Apache Parquet file as
Apache Arrow data and the Apache Parquet file may have statistics.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1704394998
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
Review Comment:
Maybe link to https://arrow.apache.org/adbc/current/format/specification.html#statistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1728057129
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,219 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`ipc-message-format
+metadata for each message`. If you're interested in the specification
+for passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ map<
+ key: int32,
+ items: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
Review Comment:
Does this mean we get an array of maps or a single map? Because we really only need a single map. Any consumer of these statistics will loop through the values and gather only the statistic values that it cares about by skipping the keys that it doesn't understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1706056549
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. Producers
+passes statistics as a map Arrow array that uses the schema through
+the Arrow C data interface.
+
+Here is the schema for a statistics map Arrow Array:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``int32``
+ - ``true``
+ - The column index or null if the statistics refer to whole table
+ or record batch.
+ * - items
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separated table for details.
Review Comment:
```suggestion
the separate table below for details.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1896963935
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,339 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+Use case
+--------
+
+One of :ref:`c-stream-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C stream interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C stream interface, module B can
+use the statistics to optimize its query plan.
+
+For example, DuckDB uses this approach but DuckDB couldn't use
+statistics because there wasn't the standardized way to pass
+statistics.
+
+.. seealso::
+
+ `duckdb::ArrowTableFunction::ArrowScanBind() in DuckDB 1.1.3
+ <https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L373-L403>`_
+
+Goals
+-----
+
+TODO: Remove the C data interface limitation?
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+TODO: Remove the C data interface limitation?
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+TODO: Should we deprecate the current ADBC's statistics API and
+redesign with this specification?
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+
+ The column index is computed as the same rule used by
+ :ref:`ipc-recordbatch-message`.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by type code not name. So
+ producers can use any valid name for fields.
+
+ TODO: Should we standardize field names?
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for individual statistics instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+For example, you can use your product name as namespace
+such as ``MY_PRODUCT:my_statistics:exact``.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float64``
+ - The average size in bytes of a row in the target column. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``: TODO: Should we use ``int64`` instead?
+ - The average size in bytes of a row in the target column. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target column. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target column. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target column. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target column. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target column. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target column. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target column. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target column. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target column. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target column. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
+
+If you find a missing statistics key that is usable for multiple
+systems, please propose it on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Examples
+========
+
+Here are some examples to help you understand.
+
+Simple record batch
+-------------------
+
+Schema::
+
+ vendor_id: int32
+ passenger_count: int64
+
+Data::
+
+ vendor_id: [5, 1, 5, 1, 5]
+ passenger_count: [1, 1, 2, 0, null]
+
+Statistics schema::
Review Comment:
Thanks for sharing the existing statistics representation.
This is also shared by @tustvold in the initial discussion: https://lists.apache.org/thread/hd79kp59796sqmwo24vmsnbk8t6x7h8g
We chose the current approach than the DataFusion approach because the current approach will be easy to add support for not pre-defined (application-specific) statistics names (statistics types): https://lists.apache.org/thread/g362w785ntcdp0hjjqr06lk5gl4bfrj0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1896966737
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
Review Comment:
Thanks. I didn't notice that the schema comparison case.
In this case, I think that we don't need to care about the case. Because we have a dynamic part in the proposed statistics schema. (This dense union part.) In general, two statistics schema will not be same with filed names standardization. For example, one statistics schema use `dense_union<int64, float64>` but another schema use `dense_union<int64>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1870677307
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
+
+If you find a missing statistics key that is usable for multiple
+systems, please propose it on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Examples
+========
+
+Here are some examples to help you understand.
Review Comment:
Should we provide concrete examples of data in the statistics array? I'm not sure if this is the best place. An impatient reader may be interested to see what content to expect from a quick glance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2520641276
Just to make sure I'm not missing something, this specification only covers the serialization of statistics via the C data interface and does not provide any mechanism as part of the existing C data interface to ask for the statistics of e.g. an `ArrowArrayStream`. The acquisition of statistics has to happen "out-of-band" via an application-specific API, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1876520469
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
Review Comment:
I think using the flattened index is probably the easiest (better than using complex types to represent statistic values or a field reference with more than one component)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1879553630
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
+
+If you find a missing statistics key that is usable for multiple
+systems, please propose it on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Examples
+========
+
+Here are some examples to help you understand.
+
+C++
Review Comment:
I've added one simple record batch case.
I'll add more examples later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2268612421
Revision: 22336f4d30e12cae6efb5961e4f822992576bcc5
Submitted crossbow builds: [ursacomputing/crossbow @ actions-28c2a45b3d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-28c2a45b3d)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/10246272610/job/28343021429)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2268606513
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1706048953
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
Review Comment:
```suggestion
* Establish a standard way to pass statistics through the Arrow C data
interface.
* Provide this in a manner that enables compatibility and ease of
implementation for users of the Arrow C data interface.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1706053799
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,192 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, Apache
+Parquet C++ can read Apache Parquet file as Apache Arrow data and
+Apache Parquet file may have statistics.
+
+One of the Arrow C data interface use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface
+3. Module B processes the passed Apache Arrow data
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Provide the standard way to pass statistics through the Arrow C data
+ interface to avoid reinventing the wheel.
+* The standard way must be easy to use with the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. Producers
+passes statistics as a map Arrow array that uses the schema through
+the Arrow C data interface.
+
+Here is the schema for a statistics map Arrow Array:
Review Comment:
```suggestion
Here is the schema for a statistics Arrow map array:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1872666633
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
Review Comment:
There's precedent in Apache Calcite where metadata handlers return float64. The exact value is not important and these are approximations used as part of query planning.
Example: https://calcite.apache.org/javadocAggregate/org/apache/calcite/rel/metadata/BuiltInMetadata.MaxRowCount.html#getMaxRowCount()
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
+
+If you find a missing statistics key that is usable for multiple
+systems, please propose it on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Examples
+========
+
+Here are some examples to help you understand.
+
+C++
Review Comment:
Given we have other comments asking for concrete examples, I feel like we should keep this content somewhere.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
Review Comment:
These stats apply for a stream of data, so this would be the null_count for the dataset behind the stream as a whole.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
Review Comment:
Ditto as above. These are summary statistics, not statistics for a single array (which would be mostly useless).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1872670000
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
Review Comment:
In retrospect since I think nothing in ADBC actually implements the statistics properly we may choose to deprecate the existing API and down the line reintroduce it using the new schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #43553:
URL: https://github.com/apache/arrow/pull/43553#issuecomment-2522248187
> Just to make sure I'm not missing something, this specification only covers the serialization of statistics via the C data interface and does not provide any mechanism as part of the existing C data interface to ask for the statistics of e.g. an `ArrowArrayStream`. The acquisition of statistics has to happen "out-of-band" via an application-specific API, right?
@CurtHagenlocher correct, that's what the discussion settled on (though as mentioned, that means that the original use case isn't fully covered anymore!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43553:
URL: https://github.com/apache/arrow/pull/43553#discussion_r1872626610
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
Review Comment:
I missed the nested case.
How about using the same rule used by the RecordBatch message?
https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
Review Comment:
Ah, yes. You're right. I should have used "by type code" not "by index".
```suggestion
consumers can access to proper field by type code not name. So
```
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
Review Comment:
"instead of using flags" means that we use "max" statistics key + "exact" flag for exact max statistics and "max" statistics key + "approximate" flag for approximate max statistics.
ADBC uses this style: https://github.com/apache/arrow-adbc/blob/95f55c73e916a9b47bc287d10296e8f561284d42/c/include/arrow-adbc/adbc.h#L1761
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
Review Comment:
It's not known yet.
In general, a statistics array is passed before we receive a target record batch.
We don't know the number of rows in a target record batch at the time.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
Review Comment:
Good point. I missed nested type case.
How about using the same rule used by the RecordBatch message?
https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message
It may be better that we use `field` not `column` like RecordBatch message does.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
Review Comment:
Right.
But I'm not sure whether this approach is the best approach not for the Arrow C data interface...
See also: https://github.com/apache/arrow/pull/43553/files#r1704373291
I think that our discussion focused on only the Arrow C data interface:
https://lists.apache.org/thread/6m9xrhfktnt0nnyss7qo333n0cl76ypc
> Out of scope:
>
> * Transmit statistics through not the C data interface
>
> Examples:
> * Transmit statistics through Apache Arrow IPC file
> * Transmit statistics through Apache Arrow Flight
With the C data interface, we have only limited approaches for this. For example, separated Arrow array like this approach, metadata in `ArrowSchema` and so on.
But we may have more approaches with other interfaces. Other approaches may be better than this approach with other approaches.
I'm OK with expanding scope of this specification. Is it OK that we restart this discussion with not only the C data interface but also all interfaces?
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
Review Comment:
Yes. I think that I borrowed "variants" from a suggestion from others but I can't find it...
Let's use "indivisual statistics":
```suggestion
We assign different statistics keys for individual statistics instead of using
```
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
Review Comment:
It makes sense:
```suggestion
statistics keys. User-defined statistics must not use it.
For example, you can use your product name as namespace
such as `MY_PRODUCT:my_statistics:exact`.
```
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
Review Comment:
@lidavidm Do you have any opinion for unifying this specification and the ADBC's statistics related APIs?
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
Review Comment:
Hmm. I think that it's difficult that we standardize field names. Because we don't define concrete field types in this specification.
We can use `str(type_code)` for field name. Does it make sense?
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
Review Comment:
Right. Let's clarify it:
```suggestion
- The average size in bytes of a row in the target column. (exact)
```
(I'll change others like this.)
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
Review Comment:
Oh, I should have used "float64" like others:
```suggestion
- ``float64``
```
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
Review Comment:
It's the number of `null` for the array. In this case, the array is a statistics array not a target column/record batch.
This is for a target column/record batch not a statistics array.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
Review Comment:
It's correct for the "exact" case. But I think that float64 is better for the "approximate" case. Because an approximate max byte width may be float for string array.
##########
docs/source/format/CDataInterfaceStatistics.rst:
##########
@@ -0,0 +1,271 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _c-data-interface-statistics:
+
+=====================================================
+Passing statistics through the Arrow C data interface
+=====================================================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+One of :ref:`c-data-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C data interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B through the Arrow C data interface, module B can use
+the statistics to optimize its query plan.
+
+Goals
+-----
+
+* Establish a standard way to pass statistics through the Arrow C data
+ interface.
+* Provide this in a manner that enables compatibility and ease of
+ implementation for existing users of the Arrow C data interface.
+
+Non-goals
+---------
+
+* Provide a common way to pass statistics that can be used for
+ other interfaces such Arrow Flight too.
+
+For example, ADBC has `the statistics related APIs
+<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__.
+This specification doesn't replace them.
+
+This specification may fit some use cases of :ref:`format-ipc` not the
+Arrow data interface. But we don't recommend this specification for
+the Arrow IPC format for now. Because we may be able to define better
+specification for the Arrow IPC format. The Arrow IPC format has some
+different features compared with the Arrow C data interface. For
+example, the Arrow IPC format can have :ref:`metadata for each message
+<ipc-message-format>`. If you're interested in the specification for
+passing statistics through the Arrow IPC format, please start a
+discussion on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+.. _c-data-interface-statistics-schema:
+
+Schema
+======
+
+This specification provides only the schema for statistics. The
+producer passes statistics through the Arrow C data interface as an
+Arrow map array that uses this schema.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<
+ indices: int32,
+ dictionary: utf8
+ >,
+ items: dense_union<...all needed types...>,
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<indices: int32, dictionary: utf8>``
+ - ``false``
+ - Statistics key is string. Dictionary is used for
+ efficiency. Different keys are assigned for exact value and
+ approximate value. Also see the separate description below for
+ statistics key.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. Also see the separate description below
+ for statistics key.
+
+ We don't standardize field names for the dense union because
+ consumers can access to proper field by index not name. So
+ producers can use any valid name for fields.
+
+.. _c-data-interface-statistics-key:
+
+Statistics key
+--------------
+
+Statistics key is string. ``dictionary<int32, utf8>`` is used for
+efficiency.
+
+We assign different statistics keys for variants instead of using
+flags. For example, we assign different statistics keys for exact
+value and approximate value.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a key.
+
+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float``
+ - The average size in bytes of a row in the target. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table or record batch. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table or record
+ batch. (approximate)
+
+If you find a missing statistics key that is usable for multiple
+systems, please propose it on the `Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Examples
+========
+
+Here are some examples to help you understand.
+
+C++
Review Comment:
OK. I'll remove it.
I'll add an example data record batch and statistics array for it instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org