You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2026/01/15 15:43:46 UTC
[PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
pitrou opened a new pull request, #48870:
URL: https://github.com/apache/arrow/pull/48870
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
Please remove this line and the above text before creating your pull request.
### Rationale for this change
### What changes are included in this PR?
### Are these changes tested?
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
**This PR contains a "Critical Fix".** (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737360141
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
Review Comment:
I think those are in a good place (I found them when I was writing the IPC reader and tested them). Those files don't necessarily exercise specific types of failures and many of the basic ones (e.g., a string array with an out of range offset) were covered before there were any fuzz failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2732723770
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
Review Comment:
I agree this would be nice, or perhaps a corpus of IPC streams that should fail validation for specific reasons.
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
```suggestion
For APIs that read serialized data (e.g., an IPC reader or function that deserializes
IPC-formatted bytes into an object), it is **(extremely?) recommended** that validation
is performed. If the ability to turn off validation is exposed as an option for performance
reasons when reading trusted data, it is **recommended** that validation is still performed
by default and the security implications of skipping validation are documented.
```
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
Review Comment:
```suggestion
either trusts or validates serialized Arrow data it receives from other sources. Many
```
Should we distinguish between "Arrow data" and data that may have come from a file? As a user you have to trust any Arrow library `Array` objects you receive (the act of validating an invalid `Array` object may cause a crash).
Serialized data is separate I think because it could be corrupted in transport and whoever produced it did not already have the ability to execute code (to produce any C/Rust/C++/Other language version of an ArrowArray, you need the ability to execute code).
Mostly I think this line should be more specific about where users should be calling `ValidateFull()` or another validation API (I don't think the average pyarrow user would be any more secure for calling this when receiving a Table from some other Python library).
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if an element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value slot in the values buffer can be ignored
+for all purposes. It is therefore tempting, when creating an array with null
+values, to not initialize the corresponding value slots.
+
+However, this then introduces a serious security risk if the Arrow data is
+serialized and published (e.g. using IPC or Flight) such that it can be
+accessed by untrusted users. Indeed, the uninitialized value slot can
+reveal data left by a previous memory allocation made in the same process.
+Depending on the application, this data could contain confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any
+data uninitialized in a buffer if the array might be sent to, or read by, an
+untrusted third-party, even when the uninitialized data is logically
+irrelevant. The easiest way to do this is to zero-initialize any buffer that
+will not be populated in full.
+
+If it is determined, through benchmarking, that zero-initialization imposes
+an excessive performance cost, a library or application may instead decide
+to use uninitialized memory internally as an optimization; but it should then
+ensure all such uninitialized values are cleared before passing the Arrow data
+to another system.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+----------------
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer, as it is by construction impossible to guard against dangerous
+behavior in this case.
+
+Advice for implementors
+-----------------------
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still
+**recommended** that you cursorily validate it for soundness (for example that
+the right number of buffers is passed for a given datatype), as a trusted producer
+can have bugs anyway.
+
+
+IPC Format
+==========
+
+The :ref:`IPC format <_ipc-message-format>` is a serialization format for the
+columnar format with associated metadata. Reading an IPC stream or file from
+an untrusted source comes with similar caveats as reading the Arrow columnar
+format.
+
+The additional signalisation and metadata in the IPC format come with
+their own risks. For example, buffer offsets and sizes encoded in IPC messages
+may be out of bounds for the IPC stream; Flatbuffers-encoded metadata payloads
+may carry incorrect offsets pointing outside of the designated metadata area.
+
+Advice for users
+----------------
+
+Your Arrow library choice should be safe against an invalid IPC stream. However,
+if the IPC stream is structurally valid, it will typically not validate the Arrow
+data itself. Therefore, as with the columnar format, it is **extremely recommended**
+that you validate the Arrow data read from an untrusted IPC stream.
+
+Advice for implementors
+-----------------------
+
+It is **extremely recommended** to run dedicated validation checks when decoding
+the IPC format, to make sure that the decoding can not induce unwanted behavior.
+Failing those checks should return a well-known error to the caller, not crash.
+
+
+Extension Types
+===============
+
+Extension types typically register a custom deserialization hook so that they
+can be automatically recreated when reading from an external source (for example
+using IPC). The deserialization hook has to decode the extension type's parameters
+from a string or binary payload specific to the extension type.
+:ref:`Typical examples <opaque_extension>` use a bespoke JSON representation
+with object fields representing the various parameters.
+
+When reading data from an untrusted source, any registered deserialization hook
+could be called with an arbitrary payload. It is therefore of primary importance
+that the hook be safe to call on invalid, potentially malicious, data. This mandates
+the use of a robust metadata serialization schema (such as JSON, but not Python's
+[``pickle``](https://docs.python.org/3/library/pickle.html), for example).
+
+Advice for users and implementors
+---------------------------------
+
+When designing an extension type, it is **extremely recommended** to choose a
+metadata serialization format that is robust against potentially malicious
+data.
+
+When implementing an extension type, it is **recommended** to ensure that the
+deserialization hook is able to detect, and error out, if the serialized metadata
+payload is invalid.
+
+
+Testing for robustness
+======================
+
+Advice for implementors
+-----------------------
+
+For APIs that may process untrusted inputs, it is **extremely recommended**
Review Comment:
```suggestion
For APIs that may process untrusted inputs, it is **extremely recommended**
```
I think this applies to both serialized and unserialized inputs
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if an element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value slot in the values buffer can be ignored
+for all purposes. It is therefore tempting, when creating an array with null
+values, to not initialize the corresponding value slots.
+
+However, this then introduces a serious security risk if the Arrow data is
+serialized and published (e.g. using IPC or Flight) such that it can be
+accessed by untrusted users. Indeed, the uninitialized value slot can
+reveal data left by a previous memory allocation made in the same process.
+Depending on the application, this data could contain confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any
+data uninitialized in a buffer if the array might be sent to, or read by, an
+untrusted third-party, even when the uninitialized data is logically
+irrelevant. The easiest way to do this is to zero-initialize any buffer that
+will not be populated in full.
+
+If it is determined, through benchmarking, that zero-initialization imposes
+an excessive performance cost, a library or application may instead decide
+to use uninitialized memory internally as an optimization; but it should then
+ensure all such uninitialized values are cleared before passing the Arrow data
+to another system.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+----------------
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer, as it is by construction impossible to guard against dangerous
+behavior in this case.
+
+Advice for implementors
+-----------------------
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still
+**recommended** that you cursorily validate it for soundness (for example that
+the right number of buffers is passed for a given datatype), as a trusted producer
+can have bugs anyway.
+
+
+IPC Format
+==========
+
+The :ref:`IPC format <_ipc-message-format>` is a serialization format for the
+columnar format with associated metadata. Reading an IPC stream or file from
+an untrusted source comes with similar caveats as reading the Arrow columnar
+format.
+
+The additional signalisation and metadata in the IPC format come with
+their own risks. For example, buffer offsets and sizes encoded in IPC messages
+may be out of bounds for the IPC stream; Flatbuffers-encoded metadata payloads
+may carry incorrect offsets pointing outside of the designated metadata area.
+
+Advice for users
+----------------
+
+Your Arrow library choice should be safe against an invalid IPC stream. However,
+if the IPC stream is structurally valid, it will typically not validate the Arrow
+data itself. Therefore, as with the columnar format, it is **extremely recommended**
+that you validate the Arrow data read from an untrusted IPC stream.
+
+Advice for implementors
+-----------------------
+
+It is **extremely recommended** to run dedicated validation checks when decoding
+the IPC format, to make sure that the decoding can not induce unwanted behavior.
+Failing those checks should return a well-known error to the caller, not crash.
+
+
+Extension Types
+===============
+
+Extension types typically register a custom deserialization hook so that they
+can be automatically recreated when reading from an external source (for example
+using IPC). The deserialization hook has to decode the extension type's parameters
+from a string or binary payload specific to the extension type.
+:ref:`Typical examples <opaque_extension>` use a bespoke JSON representation
+with object fields representing the various parameters.
+
+When reading data from an untrusted source, any registered deserialization hook
+could be called with an arbitrary payload. It is therefore of primary importance
+that the hook be safe to call on invalid, potentially malicious, data. This mandates
+the use of a robust metadata serialization schema (such as JSON, but not Python's
+[``pickle``](https://docs.python.org/3/library/pickle.html), for example).
Review Comment:
```suggestion
the use of a robust metadata serialization schema (such as JSON, but not Python's
[``pickle``](https://docs.python.org/3/library/pickle.html), or R's
[``serialize()``](https://stat.ethz.ch/R-manual/R-devel/library/base/html/serialize.html)
for example).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3846448512
Revision: 4a70d617e1739cc1e0491987a062a56fe4ae66e1
Submitted crossbow builds: [ursacomputing/crossbow @ actions-84d9160d85](https://github.com/ursacomputing/crossbow/branches/all?query=actions-84d9160d85)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/21666842755/job/62464393642)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2763032638
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
> Unfortunately, its extensive use of unsafe also means that the JVM doesn't protect us.
Wasn't there a talk of switching to the newer FFM facility? Perhaps that hasn't happened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737319566
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if an element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value slot in the values buffer can be ignored
+for all purposes. It is therefore tempting, when creating an array with null
+values, to not initialize the corresponding value slots.
+
+However, this then introduces a serious security risk if the Arrow data is
+serialized and published (e.g. using IPC or Flight) such that it can be
+accessed by untrusted users. Indeed, the uninitialized value slot can
+reveal data left by a previous memory allocation made in the same process.
+Depending on the application, this data could contain confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any
+data uninitialized in a buffer if the array might be sent to, or read by, an
+untrusted third-party, even when the uninitialized data is logically
+irrelevant. The easiest way to do this is to zero-initialize any buffer that
+will not be populated in full.
+
+If it is determined, through benchmarking, that zero-initialization imposes
+an excessive performance cost, a library or application may instead decide
+to use uninitialized memory internally as an optimization; but it should then
+ensure all such uninitialized values are cleared before passing the Arrow data
+to another system.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+----------------
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer, as it is by construction impossible to guard against dangerous
+behavior in this case.
+
+Advice for implementors
+-----------------------
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still
+**recommended** that you cursorily validate it for soundness (for example that
+the right number of buffers is passed for a given datatype), as a trusted producer
+can have bugs anyway.
+
+
+IPC Format
+==========
+
+The :ref:`IPC format <_ipc-message-format>` is a serialization format for the
+columnar format with associated metadata. Reading an IPC stream or file from
+an untrusted source comes with similar caveats as reading the Arrow columnar
+format.
+
+The additional signalisation and metadata in the IPC format come with
+their own risks. For example, buffer offsets and sizes encoded in IPC messages
+may be out of bounds for the IPC stream; Flatbuffers-encoded metadata payloads
+may carry incorrect offsets pointing outside of the designated metadata area.
+
+Advice for users
+----------------
+
+Your Arrow library choice should be safe against an invalid IPC stream. However,
+if the IPC stream is structurally valid, it will typically not validate the Arrow
+data itself. Therefore, as with the columnar format, it is **extremely recommended**
+that you validate the Arrow data read from an untrusted IPC stream.
+
+Advice for implementors
+-----------------------
+
+It is **extremely recommended** to run dedicated validation checks when decoding
+the IPC format, to make sure that the decoding can not induce unwanted behavior.
+Failing those checks should return a well-known error to the caller, not crash.
+
+
+Extension Types
+===============
+
+Extension types typically register a custom deserialization hook so that they
+can be automatically recreated when reading from an external source (for example
+using IPC). The deserialization hook has to decode the extension type's parameters
+from a string or binary payload specific to the extension type.
+:ref:`Typical examples <opaque_extension>` use a bespoke JSON representation
+with object fields representing the various parameters.
+
+When reading data from an untrusted source, any registered deserialization hook
+could be called with an arbitrary payload. It is therefore of primary importance
+that the hook be safe to call on invalid, potentially malicious, data. This mandates
+the use of a robust metadata serialization schema (such as JSON, but not Python's
+[``pickle``](https://docs.python.org/3/library/pickle.html), for example).
Review Comment:
Is https://stat.ethz.ch the canonical source for the R documentation or is there a more official website?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737316833
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
Well, we would be recommending something that we don't do currently :-)
(and the reason we don't do it is that it can be costly, e.g. `ValidateFull` is O(n) for some datatypes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3800710611
I've addressed most review comments and expanded the document quite a bit:
1) did a bit of rewording, for hopefully better clarity
2) added a paragraph about invalid values (such as utf8 in a String array)
3) added a section about deserialization of registered extension types
4) added a section about robustness testing for implementations
5) added a stub "non-Arrow formats" section
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3804553448
Revision: 29ccd3bbd024f264f201f3f54f4e191a443f51c7
Submitted crossbow builds: [ursacomputing/crossbow @ actions-2ed1d5b844](https://github.com/ursacomputing/crossbow/branches/all?query=actions-2ed1d5b844)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/21394853552/job/61590533223)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2695738346
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
+and published such that it can be accessed by untrusted users. Indeed, the
+uninitialized value slot can reveal data left by a previous memory allocation
+made in the same process. Depending on the application, this data could contain
+confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any data
+uninitialized in a buffer if the array might be sent to, or read by, a untrusted
+third-party, even when the uninitialized data is logically irrelevant. The
+easiest way to do this, though perhaps not the most efficient, is to zero-initialize
+any buffer that will not be populated in full.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+''''''''''''''''
+
+If you produce a C Data Interface structure for data that nevertheless comes
+from an untrusted source (for example because you are writing a proxy to
+an arbitrary third-party service), it is **recommended** that you validate
+the data first, as the consumer may assume that the data is valid already.
+
Review Comment:
```suggestion
In addition to invalid pointers, some array types have offsets, sizes, and buffer indices that might be out-of-bounds. The library producing arrays through the
C data interface might be performing only very light validation of these values.
```
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
+and published such that it can be accessed by untrusted users. Indeed, the
+uninitialized value slot can reveal data left by a previous memory allocation
+made in the same process. Depending on the application, this data could contain
+confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any data
+uninitialized in a buffer if the array might be sent to, or read by, a untrusted
+third-party, even when the uninitialized data is logically irrelevant. The
+easiest way to do this, though perhaps not the most efficient, is to zero-initialize
+any buffer that will not be populated in full.
Review Comment:
Worth pointing out something about query engines and dataframe libraries deciding to not do so for internal/intermediate values in computations but applying a canonicalization pass when data leaves the system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "Alex-PLACET (via GitHub)" <gi...@apache.org>.
Alex-PLACET commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2732075861
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, you need to 1) read the
+values' offsets from array buffer #2, and 2) read the range of bytes
+delimited by these offsets in array buffer #3. If the offsets are invalid
+(deliberately or not), then step 2) can access memory outside of the buffers'
+range.
Review Comment:
Can you clarify the steps, like aving a real numbered list ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3755508833
Revision: 593babb2a061da2954fc133a89bb64eea20131c3
Submitted crossbow builds: [ursacomputing/crossbow @ actions-4f7018459b](https://github.com/ursacomputing/crossbow/branches/all?query=actions-4f7018459b)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/21037284450/job/60488681144)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3846437295
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737316833
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
Well, we would be recommending something that we don't do currently :-)
(and the reason we do this is that it can be costly, e.g. `ValidateFull` is O(n) for some datatypes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737312952
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
Review Comment:
> Should we distinguish between "Arrow data" and data that may have come from a file?
Ah, in the context of this document "Arrow data" really means the columnar data itself. The language-specific objects should be out of the picture.
Perhaps there's a way to make this clearer?
> As a user you have to trust any Arrow library Array objects you receive
Hmm, not really? If I call an Arrow IPC reader, I can trust that the returned "Array" (or "RecordBatch") object satisfies some basic, implementation-specific, conditions (such as having valid buffer pointers), but I cannot trust upfront that the data that it points to is valid Arrow data.
Hence the recommendation for users to explicitly validate Arrow data coming from untrusted sources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2732450314
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
Review Comment:
And, yes, it should return an error as the beginning of the sentence says.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737915187
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
https://github.com/apache/arrow-nanoarrow/blob/090b85e7de759aca86201396615c3e8dbddb3bb8/src/nanoarrow/ipc/reader.c#L468-L485
At the lower-level API a user has to choose the validation level (I should probably link to this article in the docstring)
https://github.com/apache/arrow-nanoarrow/blob/090b85e7de759aca86201396615c3e8dbddb3bb8/src/nanoarrow/nanoarrow_ipc.h#L430-L444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3853729878
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2769137993
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data is ingested that cannot be fully trusted.
+
+In other words, this document pertains to data passed in a standardized serialized
+form (such as a IPC stream), as opposed to an implementation-specific native
+representation (such as ``arrow::Array`` in C++).
+
+.. note::
+ Implementation-specific concerns, such as bad API usage, are out of scope
+ for this document. Please refer to the implementation's own documentation.
+
+
+Who should read this
+====================
+
+You should read this document if you belong to either of these two categories:
+
+1. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined below;
+
+2. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
Review Comment:
Hmm, that changes the meaning quite a bit. I wanted to stress that "users" don't directly implement the Arrow specs, they use language-specific abstraction layers provided by an implementation. Perhaps I should just use these words :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2769381979
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data is ingested that cannot be fully trusted.
+
+In other words, this document pertains to data passed in a standardized serialized
+form (such as a IPC stream), as opposed to an implementation-specific native
+representation (such as ``arrow::Array`` in C++).
+
+.. note::
+ Implementation-specific concerns, such as bad API usage, are out of scope
+ for this document. Please refer to the implementation's own documentation.
+
+
+Who should read this
+====================
+
+You should read this document if you belong to either of these two categories:
+
+1. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined below;
+
+2. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
Review Comment:
The new wording looks good to me 👍
> 1. *users* of Arrow: that is, developers of third-party libraries or applications
> that don't implement directly implement the Arrow formats or protocols, but
> instead call language-specific APIs provided by an Arrow library
> (as defined below);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3842795350
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3842800257
I pushed some updates:
* expand on fuzz regression files and integration files
* mention that API misuse is out of scope
* mention R's `serialize`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3800968497
Rendered document preview at https://s3.amazonaws.com/arrow-data/pr_docs/48870/format/Security.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2698167993
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
Review Comment:
I think the core point of this paragraph may get a little lost in the specific details. I would suggest we start by explicitly stating the root cause of potential safety concerns. Something like
```suggestion
Arrow is a low level memory format, and the contents of Arrow
buffers are often combined and treated as pointers into the process
memory space. Invalid Arrow data may cause invalid memory accesses
(potentially crashing the process) or permit access to non-Arrow data
(potentially allowing an attacker to exfiltrate confidential information).
For example, reading and interpreting Arrow data involves reading into several buffers,
sometimes in non-trivial ways. This may for instance involve data-dependent
indirect addressing: to read a value from a Binary array, you need to
1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
these offsets in buffer #3. If the offsets are invalid (deliberately or not),
then step 2) can access invalid memory.
```
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
Review Comment:
I suggest we also make the point about performance here to give context about why
validation is not always performed
Perhaps something like this:
"Arrow implementations often assume Arrays follow the specification
to provide high speed processing. It is extremely important that
your application either trusts or validates the Arrays it receives from
other sources.
Many Arrow implementations provide APIs to do such validation.
In terms of APIs, the Rust implementation **always** validates data from external sources, unless the validation is explicitly turned off with APIs marked as `unsafe` (a special Rust keyword).
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
+and published such that it can be accessed by untrusted users. Indeed, the
+uninitialized value slot can reveal data left by a previous memory allocation
+made in the same process. Depending on the application, this data could contain
+confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any data
+uninitialized in a buffer if the array might be sent to, or read by, a untrusted
+third-party, even when the uninitialized data is logically irrelevant. The
+easiest way to do this, though perhaps not the most efficient, is to zero-initialize
+any buffer that will not be populated in full.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+''''''''''''''''
+
+If you produce a C Data Interface structure for data that nevertheless comes
+from an untrusted source (for example because you are writing a proxy to
Review Comment:
I don't think this is any different than the other APIs -- basically "if you don't trust the producer source, you should always explicitly validate the arrays before processing them"
This doesn't seem any different for the C Data Interface than for the other APIs (like IPC files. etc)
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
Review Comment:
Avoiding a double negative might make this read better
```suggestion
process' address space. As such, in-memory Arrow data should be accessed with care.
```
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
+and published such that it can be accessed by untrusted users. Indeed, the
+uninitialized value slot can reveal data left by a previous memory allocation
+made in the same process. Depending on the application, this data could contain
+confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any data
+uninitialized in a buffer if the array might be sent to, or read by, a untrusted
+third-party, even when the uninitialized data is logically irrelevant. The
+easiest way to do this, though perhaps not the most efficient, is to zero-initialize
+any buffer that will not be populated in full.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+''''''''''''''''
+
+If you produce a C Data Interface structure for data that nevertheless comes
+from an untrusted source (for example because you are writing a proxy to
+an arbitrary third-party service), it is **recommended** that you validate
+the data first, as the consumer may assume that the data is valid already.
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer.
+
+Advice for implementors
+'''''''''''''''''''''''
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still **recommended**
+that you validate it for soundness, as a trusted producer can have bugs anyway.
+
+IPC Format
Review Comment:
As above, I think we could combine this into the section about validating data from untrusted sources, and give C Data Interface and IPC Format as examples of potentially untrusted sources.
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
+and published such that it can be accessed by untrusted users. Indeed, the
+uninitialized value slot can reveal data left by a previous memory allocation
+made in the same process. Depending on the application, this data could contain
+confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any data
+uninitialized in a buffer if the array might be sent to, or read by, a untrusted
+third-party, even when the uninitialized data is logically irrelevant. The
+easiest way to do this, though perhaps not the most efficient, is to zero-initialize
+any buffer that will not be populated in full.
Review Comment:
Perhaps we can emphasize that all bytes in an Arrow array, regardless if they are "reachable", are readable by other libraries and users. Thus they should contain no potentially sensitive data (like uninitialized values).
And therefore, if query engines choose to use uninitialized memory internally as an optimization, they should ensure all such uninitialized values are cleared before passing the Arrays to another system
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2732350252
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
Review Comment:
What does `execute` mean when data is invalid? Be able to return error or throw, for example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737332660
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if an element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value slot in the values buffer can be ignored
+for all purposes. It is therefore tempting, when creating an array with null
+values, to not initialize the corresponding value slots.
+
+However, this then introduces a serious security risk if the Arrow data is
+serialized and published (e.g. using IPC or Flight) such that it can be
+accessed by untrusted users. Indeed, the uninitialized value slot can
+reveal data left by a previous memory allocation made in the same process.
+Depending on the application, this data could contain confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any
+data uninitialized in a buffer if the array might be sent to, or read by, an
+untrusted third-party, even when the uninitialized data is logically
+irrelevant. The easiest way to do this is to zero-initialize any buffer that
+will not be populated in full.
+
+If it is determined, through benchmarking, that zero-initialization imposes
+an excessive performance cost, a library or application may instead decide
+to use uninitialized memory internally as an optimization; but it should then
+ensure all such uninitialized values are cleared before passing the Arrow data
+to another system.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+----------------
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer, as it is by construction impossible to guard against dangerous
+behavior in this case.
+
+Advice for implementors
+-----------------------
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still
+**recommended** that you cursorily validate it for soundness (for example that
+the right number of buffers is passed for a given datatype), as a trusted producer
+can have bugs anyway.
+
+
+IPC Format
+==========
+
+The :ref:`IPC format <_ipc-message-format>` is a serialization format for the
+columnar format with associated metadata. Reading an IPC stream or file from
+an untrusted source comes with similar caveats as reading the Arrow columnar
+format.
+
+The additional signalisation and metadata in the IPC format come with
+their own risks. For example, buffer offsets and sizes encoded in IPC messages
+may be out of bounds for the IPC stream; Flatbuffers-encoded metadata payloads
+may carry incorrect offsets pointing outside of the designated metadata area.
+
+Advice for users
+----------------
+
+Your Arrow library choice should be safe against an invalid IPC stream. However,
+if the IPC stream is structurally valid, it will typically not validate the Arrow
+data itself. Therefore, as with the columnar format, it is **extremely recommended**
+that you validate the Arrow data read from an untrusted IPC stream.
+
+Advice for implementors
+-----------------------
+
+It is **extremely recommended** to run dedicated validation checks when decoding
+the IPC format, to make sure that the decoding can not induce unwanted behavior.
+Failing those checks should return a well-known error to the caller, not crash.
+
+
+Extension Types
+===============
+
+Extension types typically register a custom deserialization hook so that they
+can be automatically recreated when reading from an external source (for example
+using IPC). The deserialization hook has to decode the extension type's parameters
+from a string or binary payload specific to the extension type.
+:ref:`Typical examples <opaque_extension>` use a bespoke JSON representation
+with object fields representing the various parameters.
+
+When reading data from an untrusted source, any registered deserialization hook
+could be called with an arbitrary payload. It is therefore of primary importance
+that the hook be safe to call on invalid, potentially malicious, data. This mandates
+the use of a robust metadata serialization schema (such as JSON, but not Python's
+[``pickle``](https://docs.python.org/3/library/pickle.html), for example).
+
+Advice for users and implementors
+---------------------------------
+
+When designing an extension type, it is **extremely recommended** to choose a
+metadata serialization format that is robust against potentially malicious
+data.
+
+When implementing an extension type, it is **recommended** to ensure that the
+deserialization hook is able to detect, and error out, if the serialized metadata
+payload is invalid.
+
+
+Testing for robustness
+======================
+
+Advice for implementors
+-----------------------
+
+For APIs that may process untrusted inputs, it is **extremely recommended**
Review Comment:
A more general guideline that isn't put forth in this document (but perhaps we should?) is that it's preferrable to validate untrusted data at the Arrow deserialization/conversion boundary (for example deserializing IPC or decoding Parquet). As a consequence, all already deserialized data should already be trusted.
Can you think of counter-examples?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "raboof (via GitHub)" <gi...@apache.org>.
raboof commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2695128361
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
Review Comment:
```suggestion
However, this then introduces a serious security risk if the Arrow data is serialized
```
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
Review Comment:
```suggestion
For example, if an element of a primitive Arrow array is marked null through its
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2768353693
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data is ingested that cannot be fully trusted.
+
+In other words, this document pertains to data passed in a standardized serialized
+form (such as a IPC stream), as opposed to an implementation-specific native
+representation (such as ``arrow::Array`` in C++).
+
+.. note::
+ Implementation-specific concerns, such as bad API usage, are out of scope
+ for this document. Please refer to the implementation's own documentation.
+
+
+Who should read this
+====================
+
+You should read this document if you belong to either of these two categories:
+
+1. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined below;
+
+2. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
Review Comment:
```suggestion
1. *users* of Arrow: that is, developers of third-party libraries or applications
that consume data using Arrow formats or protocols created by another application.
2. *implementors* of Arrow libraries: that is, libraries that provide APIs
abstraction away from the details of the Arrow formats and protocols; such
libraries include, but are not limited to, the official Arrow implementations documented on
https://arrow.apache.org.
```
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
BTW the way the Rust API works is that by default, data read from IPC is explicitly validated (which is indeed quite expensive)
It is possible to turn this validation off via an `unsafe` API (the `unsafe` bit means the calling application has to explicitly disable the validation to acknowledge they are trusting the source):
https://docs.rs/arrow-ipc/57.2.0/arrow_ipc/reader/struct.FileDecoder.html#method.with_skip_validation
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data is ingested that cannot be fully trusted.
+
+In other words, this document pertains to data passed in a standardized serialized
+form (such as a IPC stream), as opposed to an implementation-specific native
+representation (such as ``arrow::Array`` in C++).
+
+.. note::
+ Implementation-specific concerns, such as bad API usage, are out of scope
+ for this document. Please refer to the implementation's own documentation.
+
+
+Who should read this
+====================
+
+You should read this document if you belong to either of these two categories:
+
+1. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined below;
+
+2. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives under serialized form
+from other sources. Many Arrow implementations provide APIs to validate Arrow
+data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if an element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value slot in the values buffer can be ignored
+for all purposes. It is therefore tempting, when creating an array with null
+values, to not initialize the corresponding value slots.
+
+However, this then introduces a serious security risk if the Arrow data is
+serialized and published (e.g. using IPC or Flight) such that it can be
+accessed by untrusted users. Indeed, the uninitialized value slot can
+reveal data left by a previous memory allocation made in the same process.
+Depending on the application, this data could contain confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any
+data uninitialized in a buffer if the array might be sent to, or read by, an
+untrusted third-party, even when the uninitialized data is logically
+irrelevant. The easiest way to do this is to zero-initialize any buffer that
+will not be populated in full.
+
+If it is determined, through benchmarking, that zero-initialization imposes
+an excessive performance cost, a library or application may instead decide
+to use uninitialized memory internally as an optimization; but it should then
+ensure all such uninitialized values are cleared before passing the Arrow data
+to another system.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+----------------
+
+You should **never** consume a C Data Interface structure from an untrusted
Review Comment:
This is good and clear 👍
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data is ingested that cannot be fully trusted.
+
+In other words, this document pertains to data passed in a standardized serialized
+form (such as a IPC stream), as opposed to an implementation-specific native
+representation (such as ``arrow::Array`` in C++).
+
+.. note::
+ Implementation-specific concerns, such as bad API usage, are out of scope
+ for this document. Please refer to the implementation's own documentation.
+
+
+Who should read this
+====================
+
+You should read this document if you belong to either of these two categories:
+
+1. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined below;
+
+2. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives under serialized form
+from other sources. Many Arrow implementations provide APIs to validate Arrow
+data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if an element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value slot in the values buffer can be ignored
+for all purposes. It is therefore tempting, when creating an array with null
+values, to not initialize the corresponding value slots.
+
+However, this then introduces a serious security risk if the Arrow data is
+serialized and published (e.g. using IPC or Flight) such that it can be
+accessed by untrusted users. Indeed, the uninitialized value slot can
+reveal data left by a previous memory allocation made in the same process.
+Depending on the application, this data could contain confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any
+data uninitialized in a buffer if the array might be sent to, or read by, an
+untrusted third-party, even when the uninitialized data is logically
+irrelevant. The easiest way to do this is to zero-initialize any buffer that
+will not be populated in full.
+
+If it is determined, through benchmarking, that zero-initialization imposes
+an excessive performance cost, a library or application may instead decide
+to use uninitialized memory internally as an optimization; but it should then
+ensure all such uninitialized values are cleared before passing the Arrow data
+to another system.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+----------------
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer, as it is by construction impossible to guard against dangerous
+behavior in this case.
+
+Advice for implementors
+-----------------------
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still
+**recommended** that you validate it for soundness (for example that the right
+number of buffers is passed for a given datatype), as a trusted producer can
+have bugs anyway.
+
+
+IPC Format
+==========
+
+The :ref:`IPC format <_ipc-message-format>` is a serialization format for the
+columnar format with associated metadata. Reading an IPC stream or file from
+an untrusted source comes with similar caveats as reading the Arrow columnar
+format.
+
+The additional signalisation and metadata in the IPC format come with
+their own risks. For example, buffer offsets and sizes encoded in IPC messages
+may be out of bounds for the IPC stream; Flatbuffers-encoded metadata payloads
+may carry incorrect offsets pointing outside of the designated metadata area.
+
+Advice for users
+----------------
+
+Your Arrow library choice should be safe against an invalid IPC stream. However,
+if the IPC stream is structurally valid, it will typically not validate the Arrow
+data itself. Therefore, as with the columnar format, it is **extremely recommended**
+that you validate the Arrow data read from an untrusted IPC stream.
Review Comment:
I think this paragraph makes assumptions about implementations (e.g what the default validation setting is). I think it would be clearer to just provide the guidance of always validate the data
```suggestion
Arrow libraries will typically ensure IPC streams are structurally valid
but may not also validate the underlying Array data. It is **extremely recommended**
that you use the appropriate APIs to validate the Arrow data read from an untrusted IPC stream.
```
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data is ingested that cannot be fully trusted.
+
+In other words, this document pertains to data passed in a standardized serialized
+form (such as a IPC stream), as opposed to an implementation-specific native
+representation (such as ``arrow::Array`` in C++).
+
+.. note::
+ Implementation-specific concerns, such as bad API usage, are out of scope
+ for this document. Please refer to the implementation's own documentation.
Review Comment:
Here is a suggestion to make the introduction perhaps read better. Not needed, just a suggestion
```suggestion
This document describes security considerations when reading Arrow
data from untrusted sources.
This document specifically focuses on data passed in a standardized serialized
form (such as a IPC stream), as opposed to an implementation-specific native
representation (such as ``arrow::Array`` in C++).
.. note::
Implementation-specific concerns, such as bad API usage, are out of scope
for this document. Please refer to the implementation's own documentation.
```
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
Review Comment:
It seems like links to the invalid fuzz data has been added above
I personally think this PR is already hugely valuable, even without such a list.
Thus I suggest we merge this PR and get it published. We can file an issue to track adding type specific security risks as a follow on
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data is ingested that cannot be fully trusted.
+
+In other words, this document pertains to data passed in a standardized serialized
+form (such as a IPC stream), as opposed to an implementation-specific native
+representation (such as ``arrow::Array`` in C++).
+
+.. note::
+ Implementation-specific concerns, such as bad API usage, are out of scope
+ for this document. Please refer to the implementation's own documentation.
+
+
+Who should read this
+====================
+
+You should read this document if you belong to either of these two categories:
+
+1. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined below;
+
+2. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives under serialized form
+from other sources. Many Arrow implementations provide APIs to validate Arrow
+data for soundness.
Review Comment:
I think this may cause some confusion about what "trusts" means. Here is a suggestion to clarify
```suggestion
Arrow implementations often assume inputs follow the specification to provide
high speed processing. It is **extremely recommended** that your application
explicitly validates any Arrow data it receives under serialized form
from untrusted sources. Many Arrow implementations provide explicit APIs to
perform such validation.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2732134892
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, you need to 1) read the
+values' offsets from array buffer #2, and 2) read the range of bytes
+delimited by these offsets in array buffer #3. If the offsets are invalid
+(deliberately or not), then step 2) can access memory outside of the buffers'
+range.
Review Comment:
Why not, but this is just a simple example. I'll try to avoid the mixup of buffer numbers and step numbers, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737589366
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if an element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value slot in the values buffer can be ignored
+for all purposes. It is therefore tempting, when creating an array with null
+values, to not initialize the corresponding value slots.
+
+However, this then introduces a serious security risk if the Arrow data is
+serialized and published (e.g. using IPC or Flight) such that it can be
+accessed by untrusted users. Indeed, the uninitialized value slot can
+reveal data left by a previous memory allocation made in the same process.
+Depending on the application, this data could contain confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any
+data uninitialized in a buffer if the array might be sent to, or read by, an
+untrusted third-party, even when the uninitialized data is logically
+irrelevant. The easiest way to do this is to zero-initialize any buffer that
+will not be populated in full.
+
+If it is determined, through benchmarking, that zero-initialization imposes
+an excessive performance cost, a library or application may instead decide
+to use uninitialized memory internally as an optimization; but it should then
+ensure all such uninitialized values are cleared before passing the Arrow data
+to another system.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+----------------
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer, as it is by construction impossible to guard against dangerous
+behavior in this case.
+
+Advice for implementors
+-----------------------
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still
+**recommended** that you cursorily validate it for soundness (for example that
+the right number of buffers is passed for a given datatype), as a trusted producer
+can have bugs anyway.
+
+
+IPC Format
+==========
+
+The :ref:`IPC format <_ipc-message-format>` is a serialization format for the
+columnar format with associated metadata. Reading an IPC stream or file from
+an untrusted source comes with similar caveats as reading the Arrow columnar
+format.
+
+The additional signalisation and metadata in the IPC format come with
+their own risks. For example, buffer offsets and sizes encoded in IPC messages
+may be out of bounds for the IPC stream; Flatbuffers-encoded metadata payloads
+may carry incorrect offsets pointing outside of the designated metadata area.
+
+Advice for users
+----------------
+
+Your Arrow library choice should be safe against an invalid IPC stream. However,
+if the IPC stream is structurally valid, it will typically not validate the Arrow
+data itself. Therefore, as with the columnar format, it is **extremely recommended**
+that you validate the Arrow data read from an untrusted IPC stream.
+
+Advice for implementors
+-----------------------
+
+It is **extremely recommended** to run dedicated validation checks when decoding
+the IPC format, to make sure that the decoding can not induce unwanted behavior.
+Failing those checks should return a well-known error to the caller, not crash.
+
+
+Extension Types
+===============
+
+Extension types typically register a custom deserialization hook so that they
+can be automatically recreated when reading from an external source (for example
+using IPC). The deserialization hook has to decode the extension type's parameters
+from a string or binary payload specific to the extension type.
+:ref:`Typical examples <opaque_extension>` use a bespoke JSON representation
+with object fields representing the various parameters.
+
+When reading data from an untrusted source, any registered deserialization hook
+could be called with an arbitrary payload. It is therefore of primary importance
+that the hook be safe to call on invalid, potentially malicious, data. This mandates
+the use of a robust metadata serialization schema (such as JSON, but not Python's
+[``pickle``](https://docs.python.org/3/library/pickle.html), for example).
+
+Advice for users and implementors
+---------------------------------
+
+When designing an extension type, it is **extremely recommended** to choose a
+metadata serialization format that is robust against potentially malicious
+data.
+
+When implementing an extension type, it is **recommended** to ensure that the
+deserialization hook is able to detect, and error out, if the serialized metadata
+payload is invalid.
+
+
+Testing for robustness
+======================
+
+Advice for implementors
+-----------------------
+
+For APIs that may process untrusted inputs, it is **extremely recommended**
Review Comment:
I can't think of counter examples (other than functions like `from_buffers()` that assemble untrusted buffers into an array). I think this is the distinction I was trying to get at, however we want to phrase it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2761498167
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
arrow-java barely has a validation API in the first place :/
Unfortunately, its extensive use of unsafe also means that the JVM doesn't protect us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3846991941
It looks the `preview-docs` build now fails for an unrelated reason.
In any case, do reviewers agree that this is good enough to go, or do you think there should be further changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2728599425
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
+and published such that it can be accessed by untrusted users. Indeed, the
+uninitialized value slot can reveal data left by a previous memory allocation
+made in the same process. Depending on the application, this data could contain
+confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any data
+uninitialized in a buffer if the array might be sent to, or read by, a untrusted
+third-party, even when the uninitialized data is logically irrelevant. The
+easiest way to do this, though perhaps not the most efficient, is to zero-initialize
+any buffer that will not be populated in full.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+''''''''''''''''
+
+If you produce a C Data Interface structure for data that nevertheless comes
+from an untrusted source (for example because you are writing a proxy to
+an arbitrary third-party service), it is **recommended** that you validate
+the data first, as the consumer may assume that the data is valid already.
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer.
+
+Advice for implementors
+'''''''''''''''''''''''
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still **recommended**
+that you validate it for soundness, as a trusted producer can have bugs anyway.
+
+IPC Format
Review Comment:
Yes -- I was thinking that if the guidance is the same for IPC and C Data Interface, calling them out separately makes this documentation more verbose than it could be
The high level overview in my mind is:
1. If you read Arrow data (in any format) from an untrusted source, it can be a memory and security concern. Thus you should always verify such data.
2. Implementers should provide a way to let users opt in / out of the verification depending on the source and their security threat model.
If there are specific things to suggest remembering to check that is specific for specific formats such as the FlagBuffers offsets for IPC based formats, then we could call them out in those sections.
I am happy to propose some new wording if you like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2769185664
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data is ingested that cannot be fully trusted.
+
+In other words, this document pertains to data passed in a standardized serialized
+form (such as a IPC stream), as opposed to an implementation-specific native
+representation (such as ``arrow::Array`` in C++).
+
+.. note::
+ Implementation-specific concerns, such as bad API usage, are out of scope
+ for this document. Please refer to the implementation's own documentation.
+
+
+Who should read this
+====================
+
+You should read this document if you belong to either of these two categories:
+
+1. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined below;
+
+2. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
Review Comment:
I've tried to improve the wording a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #48870:
URL: https://github.com/apache/arrow/pull/48870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3800712950
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3804543689
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3842805061
Revision: 66dce281848cc1b0b4598c59fc2304cd1473ccc1
Submitted crossbow builds: [ursacomputing/crossbow @ actions-900406a8a2](https://github.com/ursacomputing/crossbow/branches/all?query=actions-900406a8a2)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/21641589415/job/62382470081)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3846579129
Revision: 131d70fa848626197ad6942b2a7cba16f8ef553d
Submitted crossbow builds: [ursacomputing/crossbow @ actions-4dc8dcf3d1](https://github.com/ursacomputing/crossbow/branches/all?query=actions-4dc8dcf3d1)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/21667725096/job/62467402999)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-4251637817
Here is a follow on PR that proposes a definition of bug vs security vulnerability:
- https://github.com/apache/arrow/issues/49760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2728477711
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
+and published such that it can be accessed by untrusted users. Indeed, the
+uninitialized value slot can reveal data left by a previous memory allocation
+made in the same process. Depending on the application, this data could contain
+confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any data
+uninitialized in a buffer if the array might be sent to, or read by, a untrusted
+third-party, even when the uninitialized data is logically irrelevant. The
+easiest way to do this, though perhaps not the most efficient, is to zero-initialize
+any buffer that will not be populated in full.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+''''''''''''''''
+
+If you produce a C Data Interface structure for data that nevertheless comes
+from an untrusted source (for example because you are writing a proxy to
Review Comment:
Hmm, that's true. I might just remove if it muddies the message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737451211
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
Review Comment:
I agree, we'd need a more deliberately generated corpus of invalid files. Unfortunately I don't think we have any tools to help us do that currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737554204
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
Can you point me to the relevant piece of code in nanoarrow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2732698268
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
+and published such that it can be accessed by untrusted users. Indeed, the
+uninitialized value slot can reveal data left by a previous memory allocation
+made in the same process. Depending on the application, this data could contain
+confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any data
+uninitialized in a buffer if the array might be sent to, or read by, a untrusted
+third-party, even when the uninitialized data is logically irrelevant. The
+easiest way to do this, though perhaps not the most efficient, is to zero-initialize
+any buffer that will not be populated in full.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+''''''''''''''''
+
+If you produce a C Data Interface structure for data that nevertheless comes
+from an untrusted source (for example because you are writing a proxy to
+an arbitrary third-party service), it is **recommended** that you validate
+the data first, as the consumer may assume that the data is valid already.
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer.
+
+Advice for implementors
+'''''''''''''''''''''''
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still **recommended**
+that you validate it for soundness, as a trusted producer can have bugs anyway.
+
+IPC Format
Review Comment:
Yes, there's no way to meaningfully validate an `ArrowArray` other than perhaps to check pointers for unexpected NULL. This is because the `ArrowArray` does not provide buffer lengths (the consumer has to infer those based on the `ArrowSchema` and/or the buffer data of previous buffers). This is roughly the same as receiving an Arrow C++ or arrow-rs array from another library: the consumer has to assume it was correctly produced to avoid a crash and a malicious consumer can always attempt to read byte ranges it isn't supposed to based on the buffer pointers).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "Alex-PLACET (via GitHub)" <gi...@apache.org>.
Alex-PLACET commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2732075861
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, you need to 1) read the
+values' offsets from array buffer #2, and 2) read the range of bytes
+delimited by these offsets in array buffer #3. If the offsets are invalid
+(deliberately or not), then step 2) can access memory outside of the buffers'
+range.
Review Comment:
Can you clarify the steps, like a real numbered list ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2760277753
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
It seems that Arrow Java doesn't validate automatically either, though I can't say for sure. @lidavidm @jbonofre
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3843087121
Revision: 4a70d617e1739cc1e0491987a062a56fe4ae66e1
Submitted crossbow builds: [ursacomputing/crossbow @ actions-7625da4b11](https://github.com/ursacomputing/crossbow/branches/all?query=actions-7625da4b11)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/21643496578/job/62389307858)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2769386034
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,278 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes security considerations when reading Arrow
+data from untrusted sources. It focuses specifically on data passed in a
+standardized serialized form (such as a IPC stream), as opposed to an
+implementation-specific native representation (such as ``arrow::Array`` in C++).
+
+.. note::
+ Implementation-specific concerns, such as bad API usage, are out of scope
+ for this document. Please refer to the implementation's own documentation.
+
+
+Who should read this
+====================
+
+You should read this document if you belong to either of these two categories:
+
+1. *users* of Arrow: that is, developers of third-party libraries or applications
+ that don't implement directly implement the Arrow formats or protocols, but
Review Comment:
implement is repeated twice here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3755496185
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3800722054
Revision: 40c916ba460a79ce950adbf6865c1667a15aad8a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-c3adcc06fb](https://github.com/ursacomputing/crossbow/branches/all?query=actions-c3adcc06fb)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/21366855485/job/61500872666)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2728617890
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
+and published such that it can be accessed by untrusted users. Indeed, the
+uninitialized value slot can reveal data left by a previous memory allocation
+made in the same process. Depending on the application, this data could contain
+confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any data
+uninitialized in a buffer if the array might be sent to, or read by, a untrusted
+third-party, even when the uninitialized data is logically irrelevant. The
+easiest way to do this, though perhaps not the most efficient, is to zero-initialize
+any buffer that will not be populated in full.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+''''''''''''''''
+
+If you produce a C Data Interface structure for data that nevertheless comes
+from an untrusted source (for example because you are writing a proxy to
+an arbitrary third-party service), it is **recommended** that you validate
+the data first, as the consumer may assume that the data is valid already.
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer.
+
+Advice for implementors
+'''''''''''''''''''''''
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still **recommended**
+that you validate it for soundness, as a trusted producer can have bugs anyway.
+
+IPC Format
Review Comment:
Well, the guidance is not the same. For the C Data Interface, it is simply impossible (to my knowledge) to protect against rogue raw pointers. For IPC, validation is possible to ensure safe operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3853743538
Revision: 61a3e8111efeef775552a2a2e0815cbe6d5c8899
Submitted crossbow builds: [ursacomputing/crossbow @ actions-19f42b185d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-19f42b185d)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/21713677373/job/62623594342)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3854484408
Thanks a lot for the reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737296057
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
Review Comment:
Hmm, actually, a corpus of IPC streams already exists in https://github.com/apache/arrow-testing/tree/master/data/arrow-ipc-stream . Perhaps we should move them to a dedicated subdirectory for readability, and then mention them in this document?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3843078608
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2728482166
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
+Arrow specifications. Some of these concerns will apply directly to users of
+Arrow through existing implementations. Others should only be relevant for the
+implementors of Arrow libraries: by this, we mean libraries that provide APIs
+abstracting away from the details of the Arrow formats and protocols.
+
+Columnar Format
+===============
+
+The Arrow :ref:`columnar format <_format_columnar>` involves direct access to the
+process' address space. As such, in-memory Arrow data should not be accessed
+without care.
+
+Invalid data
+------------
+
+Reading and interpreting Arrow data involves reading into several buffers,
+sometimes in non-trivial ways. This may for instance involve data-dependent
+indirect addressing: to read a value from a Binary array, you need to
+1) read its offsets in buffer #2, and 2) read the range of bytes delimited by
+these offsets in buffer #3. If the offsets are invalid (deliberately or not),
+then step 2) can access invalid memory (potentially crashing the process) or
+memory unrelated to Arrow (potentially allowing an attacker to exfiltrate
+confidential data).
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+If you receive Arrow in-memory data from an untrusted source, it is
+**extremely recommended** that you first validate the data for structural
+soundness before reading it. Many Arrow implementations provide APIs to do
+such validation.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide APIs to validate Arrow data, so that users
+can assert whether data coming from untrusted sources can be safely accessed.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if a element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value in the values buffer can be ignored for all
+purposes. It is therefore tempting, when creating an array with null values, to
+not initialize the corresponding value slots.
+
+However, this then introduces a serious security if the Arrow data is serialized
+and published such that it can be accessed by untrusted users. Indeed, the
+uninitialized value slot can reveal data left by a previous memory allocation
+made in the same process. Depending on the application, this data could contain
+confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any data
+uninitialized in a buffer if the array might be sent to, or read by, a untrusted
+third-party, even when the uninitialized data is logically irrelevant. The
+easiest way to do this, though perhaps not the most efficient, is to zero-initialize
+any buffer that will not be populated in full.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+''''''''''''''''
+
+If you produce a C Data Interface structure for data that nevertheless comes
+from an untrusted source (for example because you are writing a proxy to
+an arbitrary third-party service), it is **recommended** that you validate
+the data first, as the consumer may assume that the data is valid already.
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer.
+
+Advice for implementors
+'''''''''''''''''''''''
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still **recommended**
+that you validate it for soundness, as a trusted producer can have bugs anyway.
+
+IPC Format
Review Comment:
"This" means the IPC format section?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2732448983
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
Review Comment:
Execute as in call a function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737513975
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
+Uninitialized data
+------------------
+
+A less obvious pitfall is when some parts of an Arrow array are left uninitialized.
+For example, if an element of a primitive Arrow array is marked null through its
+validity bitmap, the corresponding value slot in the values buffer can be ignored
+for all purposes. It is therefore tempting, when creating an array with null
+values, to not initialize the corresponding value slots.
+
+However, this then introduces a serious security risk if the Arrow data is
+serialized and published (e.g. using IPC or Flight) such that it can be
+accessed by untrusted users. Indeed, the uninitialized value slot can
+reveal data left by a previous memory allocation made in the same process.
+Depending on the application, this data could contain confidential information.
+
+Advice for users and implementors
+'''''''''''''''''''''''''''''''''
+
+When creating a Arrow array, it is **recommended** that you never leave any
+data uninitialized in a buffer if the array might be sent to, or read by, an
+untrusted third-party, even when the uninitialized data is logically
+irrelevant. The easiest way to do this is to zero-initialize any buffer that
+will not be populated in full.
+
+If it is determined, through benchmarking, that zero-initialization imposes
+an excessive performance cost, a library or application may instead decide
+to use uninitialized memory internally as an optimization; but it should then
+ensure all such uninitialized values are cleared before passing the Arrow data
+to another system.
+
+.. note::
+ Sending Arrow data out of the current process can happen *indirectly*,
+ for example if you produce it over the C Data Interface and the consumer
+ persists it using the IPC format on some public storage.
+
+
+C Data Interface
+================
+
+The C Data Interface contains raw pointers into the process' address space.
+It is generally not possible to validate that those pointers are legitimate;
+read from such a pointer may crash or access unrelated or bogus data.
+
+Advice for users
+----------------
+
+You should **never** consume a C Data Interface structure from an untrusted
+producer, as it is by construction impossible to guard against dangerous
+behavior in this case.
+
+Advice for implementors
+-----------------------
+
+When consuming a C Data Interface structure, you can assume that it comes from
+a trusted producer, for the reason explained above. However, it is still
+**recommended** that you cursorily validate it for soundness (for example that
+the right number of buffers is passed for a given datatype), as a trusted producer
+can have bugs anyway.
+
+
+IPC Format
+==========
+
+The :ref:`IPC format <_ipc-message-format>` is a serialization format for the
+columnar format with associated metadata. Reading an IPC stream or file from
+an untrusted source comes with similar caveats as reading the Arrow columnar
+format.
+
+The additional signalisation and metadata in the IPC format come with
+their own risks. For example, buffer offsets and sizes encoded in IPC messages
+may be out of bounds for the IPC stream; Flatbuffers-encoded metadata payloads
+may carry incorrect offsets pointing outside of the designated metadata area.
+
+Advice for users
+----------------
+
+Your Arrow library choice should be safe against an invalid IPC stream. However,
+if the IPC stream is structurally valid, it will typically not validate the Arrow
+data itself. Therefore, as with the columnar format, it is **extremely recommended**
+that you validate the Arrow data read from an untrusted IPC stream.
+
+Advice for implementors
+-----------------------
+
+It is **extremely recommended** to run dedicated validation checks when decoding
+the IPC format, to make sure that the decoding can not induce unwanted behavior.
+Failing those checks should return a well-known error to the caller, not crash.
+
+
+Extension Types
+===============
+
+Extension types typically register a custom deserialization hook so that they
+can be automatically recreated when reading from an external source (for example
+using IPC). The deserialization hook has to decode the extension type's parameters
+from a string or binary payload specific to the extension type.
+:ref:`Typical examples <opaque_extension>` use a bespoke JSON representation
+with object fields representing the various parameters.
+
+When reading data from an untrusted source, any registered deserialization hook
+could be called with an arbitrary payload. It is therefore of primary importance
+that the hook be safe to call on invalid, potentially malicious, data. This mandates
+the use of a robust metadata serialization schema (such as JSON, but not Python's
+[``pickle``](https://docs.python.org/3/library/pickle.html), for example).
Review Comment:
That is the most canonical source I know about but this can also just be `serialize()` without the link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737508651
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
Ah, Rust and nanoarrow both do this by default. It seems odd to me that the default behaviour of reading a file would be a mode that could cause a crash but perhaps this is normal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2737693394
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
Review Comment:
I think you covered the distinction I was getting at below. Perhaps I am just trying to describe the current state of the Arrow ecosystem, which is that no function that accepts an array calls `ValidateFull()` on their input (i.e., functions assume valid input and it is the responsibility of producers to produce valid output). Serialized data has the highest chance of generating invalid output, which is why validating it is reccomended.
My other point about the distinction between producers of IPC vs. producers of Arrays is that most producers of Arrays can already cause a crash or read any address in the process if they want to (so there's perhaps no escalation of risk).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #48870:
URL: https://github.com/apache/arrow/pull/48870#issuecomment-3804135300
cc @paleolimbot @alex-placet @wgtmac @lidavidm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2763050061
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+This document describes potential security concerns with the various Arrow
+specifications in contexts where data cannot be fully trusted.
+
+
+Who should read this
+====================
+
+This document targets two categories of readers:
+
+1. *implementors* of Arrow libraries: that is, libraries that provide APIs
+ abstraction away from the details of the Arrow formats and protocols; such
+ libraries include the official Arrow implementations documented on
+ https://arrow.apache.org, but not only.
+
+2. *users* of Arrow: that is, developers of third-party libraries or applications
+ that use some of the Arrow formats or protocols by calling into Arrow libraries
+ as defined above.
+
+
+Columnar Format
+===============
+
+Invalid data
+------------
+
+The Arrow :ref:`columnar format <_format_columnar>` is an efficient binary
+representation with a focus on performance and efficiency. While the format
+does not store raw pointers, the contents of Arrow buffers are often
+combined and converted to pointers into the process' address space.
+Invalid Arrow data may therefore cause invalid memory accesses
+(potentially crashing the process) or access to non-Arrow data
+(potentially allowing an attacker to exfiltrate confidential information).
+
+For instance, to read a value from a Binary array, one needs to 1) read the
+values' offsets from the array's offsets buffer, and 2) read the range of bytes
+delimited by these offsets in the array's data buffer. If the offsets are
+invalid (deliberately or not), then step 2) can access memory outside of the
+data buffer's range.
+
+Another instance of invalid data lies in the values themselves. For example,
+a String array is only allowed to contain valid UTF-8 data, but an untrusted
+source might have emitted invalid UTF-8 under the disguise of a String array.
+An unsuspecting algorithm that is only specified for valid UTF-8 inputs might
+lead to dangerous behavior (for example by reading memory out of bounds when
+looking for an UTF-8 character boundary).
+
+Fortunately, knowing its schema, it is possible to validate Arrow data up front,
+so that reading this data will not pose any danger later on.
+
+.. TODO:
+ For each layout, we should list the associated security risks and the recommended
+ steps to validate (perhaps in Columnar.rst)
+
+Advice for users
+''''''''''''''''
+
+Arrow implementations often assume inputs follow the specification to provide
+high speed processing. It is **extremely recommended** that your application
+either trusts or validates the Arrow data it receives from other sources. Many
+Arrow implementations provide APIs to validate Arrow data for soundness.
+
+.. TODO: link to some validation APIs for the main implementations here?
+
+Advice for implementors
+'''''''''''''''''''''''
+
+It is **recommended** that you provide dedicated APIs to validate Arrow arrays
+and/or record batches. Users will be able to utilize those APIs to assert whether
+data coming from untrusted sources can be safely accessed.
+
+A typical validation API must return a well-defined error, not crash, if the
+given Arrow data is invalid; it must always be safe to execute regardless of
+whether the data is valid or not.
+
Review Comment:
There's been talk, but no action; the bigger issue is that it would mean dropping Java <24 or so. Given we still have people pop up occasionally trying to use Java 8, that may be a hard sell; I think Spark has only just moved to Java 17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48868: [Doc] Document security model for the Arrow formats [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #48870:
URL: https://github.com/apache/arrow/pull/48870#discussion_r2698399828
##########
docs/source/format/Security.rst:
##########
@@ -0,0 +1,150 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_security:
+
+***********************
+Security Considerations
+***********************
+
+How to read this
+================
+
+Hereafter we try list potential security concerns when dealing with the various
Review Comment:
```suggestion
Hereafter we try to list potential security concerns when dealing with the various
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org