[Discuss][IPC] Deprecate Tensor and SparseTensor messages?

[Antoine Pitrou <an...@python.org>]

Hello,

While the Arrow IPC format is primarily meant for RecordBatch transport, 
it also has dedicated messages for transporting N-dimensional Tensor and 
SparseTensor data.

The Tensor message was added in 2017 
(https://github.com/apache/arrow/pull/435), while the SparseTensor 
message was added in 2019 (https://github.com/apache/arrow/pull/3651).

Both have not had any significant additions since 2020. Furthermore:

* They are still marked experimental.
* They are not exercised by the the cross-implementation integration 
test suite.
* Convenience IPC reading APIs are generally architected around 
RecordBatch iteration, requiring more manual plumbing to read Tensor and 
SparseTensor messages from an IPC stream.
* They are generally not reachable from Flight RPC APIs, or other 
high-level APIs such the C++ Dataset API.
* A code search on GitHub reveals no apparent third-party usage of these 
message types.

Nowadays, Arrow has canonical extension types for fixed-shape and 
variable-shaped tensors which allow passing (dense) N-dimensional 
tensors as regular RecordBatch columns. We could also add canonical 
extension types for the different kinds of sparse tensors, if there is 
demand for it.

Despite not seeing much usage, these message types carry their 
maintenance baggage. Recently, we've had a number of security reports 
around the lack of validation of untrusted Tensor and SparseTensor 
messages in Arrow C++, requiring core developer intervention to analyze 
and fix those issues.

I would like to suggest that we start deprecating these messages types 
in the IPC format, and that we recommend deprecating the corresponding 
APIs in all major Arrow implementations. What do you think?

Regards

Antoine.

Re: [Discuss][IPC] Deprecate Tensor and SparseTensor messages?

[Weston Pace <we...@gmail.com>]

I would support deprecation.  I've never seen legitimate use of these
messages, only confusion.

On Wed, Jul 1, 2026 at 9:25 AM Rok Mihevc <ro...@gmail.com> wrote:

> I would support removing Tensor and SparseTensors from IPC
> because FixedShapeTensorArray and VariableShapeTensorArray
> seem like a better way forward and can be made to provide the
> same functionality.
> Also securing IPC messages is quite difficult given complexity of
> tensor data structures especially once one factors in that every
> IPC implementation needs to think about this.
>
> It would of course also be good to do some research downstream
> to see if we have downsream users who depend on Tensor via IPC.
>
> Rok
>
> On Wed, Jul 1, 2026 at 5:59 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hello,
> >
> > While the Arrow IPC format is primarily meant for RecordBatch transport,
> > it also has dedicated messages for transporting N-dimensional Tensor and
> > SparseTensor data.
> >
> > The Tensor message was added in 2017
> > (https://github.com/apache/arrow/pull/435), while the SparseTensor
> > message was added in 2019 (https://github.com/apache/arrow/pull/3651).
> >
> > Both have not had any significant additions since 2020. Furthermore:
> >
> > * They are still marked experimental.
> > * They are not exercised by the the cross-implementation integration
> > test suite.
> > * Convenience IPC reading APIs are generally architected around
> > RecordBatch iteration, requiring more manual plumbing to read Tensor and
> > SparseTensor messages from an IPC stream.
> > * They are generally not reachable from Flight RPC APIs, or other
> > high-level APIs such the C++ Dataset API.
> > * A code search on GitHub reveals no apparent third-party usage of these
> > message types.
> >
> > Nowadays, Arrow has canonical extension types for fixed-shape and
> > variable-shaped tensors which allow passing (dense) N-dimensional
> > tensors as regular RecordBatch columns. We could also add canonical
> > extension types for the different kinds of sparse tensors, if there is
> > demand for it.
> >
> > Despite not seeing much usage, these message types carry their
> > maintenance baggage. Recently, we've had a number of security reports
> > around the lack of validation of untrusted Tensor and SparseTensor
> > messages in Arrow C++, requiring core developer intervention to analyze
> > and fix those issues.
> >
> > I would like to suggest that we start deprecating these messages types
> > in the IPC format, and that we recommend deprecating the corresponding
> > APIs in all major Arrow implementations. What do you think?
> >
> > Regards
> >
> > Antoine.
> >
> >
>

Re: [Discuss][IPC] Deprecate Tensor and SparseTensor messages?

[Rok Mihevc <ro...@gmail.com>]

I would support removing Tensor and SparseTensors from IPC
because FixedShapeTensorArray and VariableShapeTensorArray
seem like a better way forward and can be made to provide the
same functionality.
Also securing IPC messages is quite difficult given complexity of
tensor data structures especially once one factors in that every
IPC implementation needs to think about this.

It would of course also be good to do some research downstream
to see if we have downsream users who depend on Tensor via IPC.

Rok

On Wed, Jul 1, 2026 at 5:59 PM Antoine Pitrou <an...@python.org> wrote:

>
> Hello,
>
> While the Arrow IPC format is primarily meant for RecordBatch transport,
> it also has dedicated messages for transporting N-dimensional Tensor and
> SparseTensor data.
>
> The Tensor message was added in 2017
> (https://github.com/apache/arrow/pull/435), while the SparseTensor
> message was added in 2019 (https://github.com/apache/arrow/pull/3651).
>
> Both have not had any significant additions since 2020. Furthermore:
>
> * They are still marked experimental.
> * They are not exercised by the the cross-implementation integration
> test suite.
> * Convenience IPC reading APIs are generally architected around
> RecordBatch iteration, requiring more manual plumbing to read Tensor and
> SparseTensor messages from an IPC stream.
> * They are generally not reachable from Flight RPC APIs, or other
> high-level APIs such the C++ Dataset API.
> * A code search on GitHub reveals no apparent third-party usage of these
> message types.
>
> Nowadays, Arrow has canonical extension types for fixed-shape and
> variable-shaped tensors which allow passing (dense) N-dimensional
> tensors as regular RecordBatch columns. We could also add canonical
> extension types for the different kinds of sparse tensors, if there is
> demand for it.
>
> Despite not seeing much usage, these message types carry their
> maintenance baggage. Recently, we've had a number of security reports
> around the lack of validation of untrusted Tensor and SparseTensor
> messages in Arrow C++, requiring core developer intervention to analyze
> and fix those issues.
>
> I would like to suggest that we start deprecating these messages types
> in the IPC format, and that we recommend deprecating the corresponding
> APIs in all major Arrow implementations. What do you think?
>
> Regards
>
> Antoine.
>
>

Re: [Discuss][IPC] Deprecate Tensor and SparseTensor messages?

[Matt Topol <zo...@gmail.com>]

I agree with Curt that we should probably do an official vote to go ahead
with the deprecation. That said, I'd be in favor of it

--Matt

On Mon, Aug 31, 2026, 11:54 AM Curt Hagenlocher <cu...@hagenlocher.org>
wrote:

> My vague expectation is that there should be a vote. From what I can
> tell, the only complete implementation is the C++ one -- which means
> that the feature wouldn't even meet today's bar for format changes.
>
> There is a recent request for sparse tensor support in Rust at
> https://github.com/apache/arrow-rs/issues/10462 .
>
> On Mon, Aug 31, 2026 at 8:28 AM Antoine Pitrou <an...@python.org> wrote:
> >
> >
> > Hello,
> >
> > Just a ping on this for people who haven't seen the original discussion.
> >
> > Some questions:
> > 1) is anybody opposed to such a deprecation?
> > 2) do you think a formal vote is required?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 01/07/2026 à 17:59, Antoine Pitrou a écrit :
> > >
> > > Hello,
> > >
> > > While the Arrow IPC format is primarily meant for RecordBatch
> transport,
> > > it also has dedicated messages for transporting N-dimensional Tensor
> and
> > > SparseTensor data.
> > >
> > > The Tensor message was added in 2017
> > > (https://github.com/apache/arrow/pull/435), while the SparseTensor
> > > message was added in 2019 (https://github.com/apache/arrow/pull/3651).
> > >
> > > Both have not had any significant additions since 2020. Furthermore:
> > >
> > > * They are still marked experimental.
> > > * They are not exercised by the the cross-implementation integration
> > > test suite.
> > > * Convenience IPC reading APIs are generally architected around
> > > RecordBatch iteration, requiring more manual plumbing to read Tensor
> and
> > > SparseTensor messages from an IPC stream.
> > > * They are generally not reachable from Flight RPC APIs, or other
> > > high-level APIs such the C++ Dataset API.
> > > * A code search on GitHub reveals no apparent third-party usage of
> these
> > > message types.
> > >
> > > Nowadays, Arrow has canonical extension types for fixed-shape and
> > > variable-shaped tensors which allow passing (dense) N-dimensional
> > > tensors as regular RecordBatch columns. We could also add canonical
> > > extension types for the different kinds of sparse tensors, if there is
> > > demand for it.
> > >
> > > Despite not seeing much usage, these message types carry their
> > > maintenance baggage. Recently, we've had a number of security reports
> > > around the lack of validation of untrusted Tensor and SparseTensor
> > > messages in Arrow C++, requiring core developer intervention to analyze
> > > and fix those issues.
> > >
> > > I would like to suggest that we start deprecating these messages types
> > > in the IPC format, and that we recommend deprecating the corresponding
> > > APIs in all major Arrow implementations. What do you think?
> > >
> > > Regards
> > >
> > > Antoine.
> >
>

Re: [Discuss][IPC] Deprecate Tensor and SparseTensor messages?

[Curt Hagenlocher <cu...@hagenlocher.org>]

My vague expectation is that there should be a vote. From what I can
tell, the only complete implementation is the C++ one -- which means
that the feature wouldn't even meet today's bar for format changes.

There is a recent request for sparse tensor support in Rust at
https://github.com/apache/arrow-rs/issues/10462 .

On Mon, Aug 31, 2026 at 8:28 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> Hello,
>
> Just a ping on this for people who haven't seen the original discussion.
>
> Some questions:
> 1) is anybody opposed to such a deprecation?
> 2) do you think a formal vote is required?
>
> Regards
>
> Antoine.
>
>
> Le 01/07/2026 à 17:59, Antoine Pitrou a écrit :
> >
> > Hello,
> >
> > While the Arrow IPC format is primarily meant for RecordBatch transport,
> > it also has dedicated messages for transporting N-dimensional Tensor and
> > SparseTensor data.
> >
> > The Tensor message was added in 2017
> > (https://github.com/apache/arrow/pull/435), while the SparseTensor
> > message was added in 2019 (https://github.com/apache/arrow/pull/3651).
> >
> > Both have not had any significant additions since 2020. Furthermore:
> >
> > * They are still marked experimental.
> > * They are not exercised by the the cross-implementation integration
> > test suite.
> > * Convenience IPC reading APIs are generally architected around
> > RecordBatch iteration, requiring more manual plumbing to read Tensor and
> > SparseTensor messages from an IPC stream.
> > * They are generally not reachable from Flight RPC APIs, or other
> > high-level APIs such the C++ Dataset API.
> > * A code search on GitHub reveals no apparent third-party usage of these
> > message types.
> >
> > Nowadays, Arrow has canonical extension types for fixed-shape and
> > variable-shaped tensors which allow passing (dense) N-dimensional
> > tensors as regular RecordBatch columns. We could also add canonical
> > extension types for the different kinds of sparse tensors, if there is
> > demand for it.
> >
> > Despite not seeing much usage, these message types carry their
> > maintenance baggage. Recently, we've had a number of security reports
> > around the lack of validation of untrusted Tensor and SparseTensor
> > messages in Arrow C++, requiring core developer intervention to analyze
> > and fix those issues.
> >
> > I would like to suggest that we start deprecating these messages types
> > in the IPC format, and that we recommend deprecating the corresponding
> > APIs in all major Arrow implementations. What do you think?
> >
> > Regards
> >
> > Antoine.
>

Re: [Discuss][IPC] Deprecate Tensor and SparseTensor messages?

[Antoine Pitrou <an...@python.org>]

Hello,

Just a ping on this for people who haven't seen the original discussion.

Some questions:
1) is anybody opposed to such a deprecation?
2) do you think a formal vote is required?

Regards

Antoine.


Le 01/07/2026 à 17:59, Antoine Pitrou a écrit :
> 
> Hello,
> 
> While the Arrow IPC format is primarily meant for RecordBatch transport,
> it also has dedicated messages for transporting N-dimensional Tensor and
> SparseTensor data.
> 
> The Tensor message was added in 2017
> (https://github.com/apache/arrow/pull/435), while the SparseTensor
> message was added in 2019 (https://github.com/apache/arrow/pull/3651).
> 
> Both have not had any significant additions since 2020. Furthermore:
> 
> * They are still marked experimental.
> * They are not exercised by the the cross-implementation integration
> test suite.
> * Convenience IPC reading APIs are generally architected around
> RecordBatch iteration, requiring more manual plumbing to read Tensor and
> SparseTensor messages from an IPC stream.
> * They are generally not reachable from Flight RPC APIs, or other
> high-level APIs such the C++ Dataset API.
> * A code search on GitHub reveals no apparent third-party usage of these
> message types.
> 
> Nowadays, Arrow has canonical extension types for fixed-shape and
> variable-shaped tensors which allow passing (dense) N-dimensional
> tensors as regular RecordBatch columns. We could also add canonical
> extension types for the different kinds of sparse tensors, if there is
> demand for it.
> 
> Despite not seeing much usage, these message types carry their
> maintenance baggage. Recently, we've had a number of security reports
> around the lack of validation of untrusted Tensor and SparseTensor
> messages in Arrow C++, requiring core developer intervention to analyze
> and fix those issues.
> 
> I would like to suggest that we start deprecating these messages types
> in the IPC format, and that we recommend deprecating the corresponding
> APIs in all major Arrow implementations. What do you think?
> 
> Regards
> 
> Antoine.