You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rok (via GitHub)" <gi...@apache.org> on 2023/08/15 02:41:45 UTC

[GitHub] [arrow] rok opened a new pull request, #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

rok opened a new pull request, #37166:
URL: https://github.com/apache/arrow/pull/37166

   ### Rationale for this change
   
   For use cases where underlying datatype and number of dimensions in tensors are equal but not the actual shape we want to add a `VariableShapeTensorType`.
   See https://github.com/apache/arrow/issues/24868 and https://github.com/huggingface/datasets/issues/5272
   
   ### What changes are included in this PR?
   
   This introduces definition of `arrow.variable_shape_tensor` extension and it's C++ implementation.
   
   ### Are these changes tested?
   
   Yes.
   
   ### Are there any user-facing changes?
   
   This introduces new extension type to the user.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] paleolimbot commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1310524404


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,76 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.

Review Comment:
   Is it necessary for `ndim` to be known in advance? If it is not (i.e., if the storage type is defined instead as a `List<uint32>[ndim]`), we can return this type from the ADBC Postgres driver (here: https://github.com/apache/arrow-adbc/blob/main/c/driver/postgresql/postgres_copy_reader.h#L471-L534 ).
   
   If it is essential that the number of dimensions is known at the schema level it is OK too (we just won't be able to actually use it in the postgres driver).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1335975252


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,107 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape over dimensions that are guaranteed to stay
+    constant over of all tensors in the array if all their ragged dimension
+    sizes were replaced by 0.
+    An array containing tensor with shape (2, 3, 4) and uniform dimensions
+    (0, 2) would have uniform shape (2, 0, 4).
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object including number of
+  dimensions of the contained tensors as an integer with key **"ndim"**

Review Comment:
   Changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1344158707


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where

Review Comment:
   Changed to `int32`. (Shapes and strides in `Tensor` are typically `int64`)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1348794676


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensor's dimensions which are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with null. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, null, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``

Review Comment:
   @jorisvandenbossche feel free to accept this suggestion or alter it.
   22 @pitrou 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1708235013

   We could split this into two types:
   `VariableShapeTensor` that has `ndim=const`, stores permutations and raged dimension list per array. This is less general but uses `fixed_size_list` for shape and saves storage cost on not storing strides. It does incur computation cost when strides need to be calculated. This layout would map to [tf.RaggedTenor](https://www.tensorflow.org/api_docs/python/tf/RaggedTensor). Storage would be:
    ```cpp
    struct_({field("shape", fixed_size_list(uint32(), ndim)), field("data", list(value_type))})
    ```
   
   `VariableDimensionTensor` that has `ndim!=const` and stores strides (or permutations) per row. This is more general, but would add extra storage cost by storing strides (and keeping shapes in `list`). It would require less computation as strides would not have to be calculated at use time. This layout would map well to [torch.Nested](https://pytorch.org/docs/stable/nested.html), but would be more general as it would allow arbitrary ndim per row.
    ```cpp
    struct_({field("shape", list(uint32()), field("strides", list(int64()), field("data", list(value_type))})
   ```
   
   cc @pitrou 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] paleolimbot commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1317269300


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,76 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.

Review Comment:
   Yes, there is potentially a different `ndim` for each item in the array. I imagine that in practice this does not frequently occur but at the time we resolve the Arrow output type we don't have any actual data to inspect to guess. Opening that can of worms would be hard but we might have to do it for other reasons, too (e.g., guessing decimal output precision/bitwidth since that can very by row as well in Postgres).
   
   I don't want the discussion to get *too* hung up on this point if it makes life more difficult. If I had to choose between allowing `ndim` to vary among items in the array and consistency with the `fixed_shape_tensor()`, I would pick consistency with the fixed shape tensor! There are other considerations for returning arrays from Postgres to Arrow (for example, if `ndim` is 1, a more intuitive output type would be a plain `List`); my initial comment of "we won't be able to use this" is probably not true.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1693515812

   Looking at [PyTorch `NestedTensor`](https://github.com/pytorch/pytorch/blob/3a3cf0e09d475df9237c95ebd14debf650e0c038/aten/src/ATen/native/nested/README.md#nestedtensors) it stores strides per tensor as opposed to our proposal which stores just a permutation for the entire array which can then be used to derive strides (using tensor shape) per tensor. In PyTorch if strides are not provided to the [constructor](https://github.com/pytorch/pytorch/blob/3c11184ca8fe35de70d1147f0f9d9beea8dc7e48/aten/src/ATen/NestedTensorImpl.cpp#L211C1-L221) they are apparently calculated assuming row-major/C-contiguous layout.
   https://github.com/pytorch/pytorch/blob/3a3cf0e09d475df9237c95ebd14debf650e0c038/aten/src/ATen/NestedTensorImpl.cpp#L104C19-L130
   This makes me assume strides are stored for caching (and not to allow variable data layout per tensor).
   
   Should we adjust to store permutations (or strides) per row too or rather leave that for another extension (`GeneralVariableShapeTensorType`)? @lhoestq @mariosasko @AlenkaF @jorisvandenbossche @ezyang
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347405370


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,109 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensors dimensions are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with 0s. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, 0, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "dim_names": ["C", "H", "W"] }``
+
+  - Example with ``uniform_shape`` metadata for a set of color images
+    with variable width:
+
+    ``{ "dim_names": ["H", "W", "C"], "uniform_shape": [400, 0, 3] }``
+
+  - Example of permuted 3-dimensional tensor:
+
+    ``{ "permutation": [2, 0, 1] }``
+
+    This is the physical layout shape and the shape of the logical
+    layout would given an individual tensor of shape [100, 200, 500]
+    be ``[500, 100, 200]``.
+
+.. note::
+
+  With the exception of ``permutation``, the parameters and storage
+  of VariableShapeTensor relate to the *physical* storage of the tensor.
+
+  For example, consider a tensor with:
+    shape = [10, 20, 30]
+    dim_names = [x, y, z]
+    permutations = [2, 0, 1]
+
+  This means the logical tensor has names [z, x, y] and shape [30, 10, 20].
+
+  Elements in a variable shape tensor extension array are stored
+  in row-major/C-contiguous order.

Review Comment:
   ```suggestion
   .. note::
      Values inside each **data** tensor element are stored in row-major/C-contiguous
      order according to the corresponding **shape**.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1344026774


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape of the dimensions that are guaranteed to stay
+    constant over all tensors in the array, with the shape of the ragged dimensions
+    set to 0.
+    An array containing tensor with shape (2, 3, 4) and uniform dimensions
+    (0, 2) would have uniform shape (2, 0, 4).
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object, that optionally includes
+  dimension names with keys **"dim_names"**, ordering of
+  dimensions with key **"permutation"**, indices of dimensions whose sizes
+  are guaranteed to remain constant with key **"uniform_dimensions"** and
+  shape of those dimensions with key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "dim_names": ["C", "H", "W"] }``
+
+  - Example with ``uniform_dimensions`` metadata for a set of color images
+    with variable width:

Review Comment:
   Indeed `uniform_dimensions` is a "weaker" version of `uniform_shape` information. We could have a case where we know dimensions of images won't change, but we don't know the shape until runtime. Could the information that dimension is uniform still be used for optimization at runtime? I imagine no. What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347591001


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensors dimensions are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with 0s. If ``uniform_shape`` is not provided it is assumed

Review Comment:
   Switched language to `null`. This is similar to what pytorch and tensorflow do (they use `None` in python).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] ezyang commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "ezyang (via GitHub)" <gi...@apache.org>.
ezyang commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1703643362

   This has been a contentious point for us internally. CC'ing @jbschlosser @cpuhrsch
   
   If you want a generic container format that can represent anything that morally is a list of regular dense tensors, the most expressive way to do it is store individual strides per tensor. This is particularly useful if you have two jagged dimensions (e.g., list of varying size images on both height and width); if you store sizes/strides per batch this all just works fine.
   
   However, it's a bit of a pain to store all the metadata, because most of the time you're not taking full advantage of all the expressivity; something a bit more compact will be more efficient, both in terms of memory usage and in terms of algorithmic efficiency. For example, it's common to have only one ragged dimension, so you have batch x ragged x some number of dense dimensions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1695511354

   @AlenkaF Storing permutations would allow per tensor memory layout, which I'm not sure is really needed in practice and could be confusing.
   Storing strides (and perhaps requiring uniform permutation) could be useful as consumers wouldn't need to calculate strides when using tensors. Complexity wouldn't really increase.
   Either way the size of type would increase by `n tensors x ndim x sizeof(int64)`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1744769338

   @pitrou do you think it'd be a good idea to split out the type definition into a separate PR or should we merge this in one piece?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1343976973


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape of the dimensions that are guaranteed to stay
+    constant over all tensors in the array, with the shape of the ragged dimensions
+    set to 0.
+    An array containing tensor with shape (2, 3, 4) and uniform dimensions
+    (0, 2) would have uniform shape (2, 0, 4).
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object, that optionally includes
+  dimension names with keys **"dim_names"**, ordering of
+  dimensions with key **"permutation"**, indices of dimensions whose sizes
+  are guaranteed to remain constant with key **"uniform_dimensions"** and
+  shape of those dimensions with key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "dim_names": ["C", "H", "W"] }``
+
+  - Example with ``uniform_dimensions`` metadata for a set of color images
+    with variable width:
+
+    ``{ "dim_names": ["H", "W", "C"], "uniform_dimensions": [1] }``
+
+  - Example of permuted 3-dimensional tensor:
+
+    ``{ "permutation": [2, 0, 1] }``
+
+    This is the physical layout shape and the shape of the logical
+    layout would given an individual tensor of shape [100, 200, 500]
+    be ``[500, 100, 200]``.
+
+.. note::
+
+  With the exception of permutation all other parameters and storage
+  of VariableShapeTensor define the *physical* storage of the tensor.

Review Comment:
   ```suggestion
     With the exception of ``permutation``, the parameters and storage
     of VariableShapeTensor relate to the *physical* storage of the tensor.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1343980148


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape of the dimensions that are guaranteed to stay
+    constant over all tensors in the array, with the shape of the ragged dimensions

Review Comment:
   If they are just wording details then we don't need to inform the ML, IMHO.
   However, if we make any semantic changes (such as the ``uniform_dimensions`` questions below), then it would need mentioning on the ML.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347320811


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are

Review Comment:
   Added in `uniform_shape`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1678350689

   :warning: GitHub issue #24868 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1344085172


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape of the dimensions that are guaranteed to stay
+    constant over all tensors in the array, with the shape of the ragged dimensions
+    set to 0.
+    An array containing tensor with shape (2, 3, 4) and uniform dimensions
+    (0, 2) would have uniform shape (2, 0, 4).
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object, that optionally includes
+  dimension names with keys **"dim_names"**, ordering of
+  dimensions with key **"permutation"**, indices of dimensions whose sizes
+  are guaranteed to remain constant with key **"uniform_dimensions"** and
+  shape of those dimensions with key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "dim_names": ["C", "H", "W"] }``
+
+  - Example with ``uniform_dimensions`` metadata for a set of color images
+    with variable width:

Review Comment:
   I have no idea. Perhaps potential users of this type do? :-)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1343969434


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where

Review Comment:
   I have not followed this, but was there any discussion of signed vs. unsigned here? Most of the Arrow spec uses signed integers even for numbers that are always positive (and unless your tensors are huge I'm not sure having one less bit to represent the shape's dimensions is a problem?).



##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where

Review Comment:
   I have not followed this, but was there any discussion of signed vs. unsigned here? Most of the Arrows spec use signed integers even for numbers that are always positive (and unless your tensors are huge I'm not sure having one less bit to represent the shape's dimensions is a problem?).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1348794015


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensor's dimensions which are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with null. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, null, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``

Review Comment:
   That's a fair point! Empty string feels like the safer choice here. See my suggested change below.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347320516


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape of the dimensions that are guaranteed to stay
+    constant over all tensors in the array, with the shape of the ragged dimensions
+    set to 0.
+    An array containing tensor with shape (2, 3, 4) and uniform dimensions
+    (0, 2) would have uniform shape (2, 0, 4).
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object, that optionally includes
+  dimension names with keys **"dim_names"**, ordering of
+  dimensions with key **"permutation"**, indices of dimensions whose sizes
+  are guaranteed to remain constant with key **"uniform_dimensions"** and
+  shape of those dimensions with key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "dim_names": ["C", "H", "W"] }``
+
+  - Example with ``uniform_dimensions`` metadata for a set of color images
+    with variable width:

Review Comment:
   Current language was demanding both be given at the same time. So that's imo equivalent to just having `uniform_shape`, so let's simplify to `uniform_shape` only.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1352520207


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensor's dimensions which are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with null. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, null, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``

Review Comment:
   Personally I would say this paragraph is a bit superfluous, because that all follows from the main spec logically and from those parameters being optional?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1335860070


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,107 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape over dimensions that are guaranteed to stay
+    constant over of all tensors in the array if all their ragged dimension
+    sizes were replaced by 0.

Review Comment:
   ```suggestion
     * **uniform_shape** = shape of the dimensions that are guaranteed to stay
       constant over all tensors in the array, with the shape of the ragged dimensions
       set to 0.
   ```
   
   (I had a hard time understanding the sentence, is my suggestion still correct / intended meaning?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1708307003

   Hmm, is there a strong use case for `VariableDimensionTensor`? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1748763563

   I've moved implementation to #38008 and changed the language to only ask for `uniform_shape` as discussed above.
   @jorisvandenbossche & @pitrou could you please do a sanity check before we merge?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347399122


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,109 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensors dimensions are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with 0s. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, 0, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "dim_names": ["C", "H", "W"] }``
+
+  - Example with ``uniform_shape`` metadata for a set of color images
+    with variable width:
+
+    ``{ "dim_names": ["H", "W", "C"], "uniform_shape": [400, 0, 3] }``
+
+  - Example of permuted 3-dimensional tensor:
+
+    ``{ "permutation": [2, 0, 1] }``
+
+    This is the physical layout shape and the shape of the logical
+    layout would given an individual tensor of shape [100, 200, 500]
+    be ``[500, 100, 200]``.

Review Comment:
   ```suggestion
       For example, if the physical **shape** of an individual tensor
       is ``[100, 200, 500]``, this permutation would denote a logical shape
       of ``[500, 100, 200]``.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1348794676


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensor's dimensions which are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with null. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, null, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``

Review Comment:
   @jorisvandenbossche feel free to accept this suggestion or alter it.
   cc @pitrou 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1343971847


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape of the dimensions that are guaranteed to stay
+    constant over all tensors in the array, with the shape of the ragged dimensions

Review Comment:
   The term "ragged" always feels obscure to me, how about "non-uniform"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1750388089

   @jorisvandenbossche can we merge this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1716759683

   Seeing how ragged/jagged dimensions are used in [pytorch](https://pytorch.org/torchrec/torchrec.sparse.html#module-torchrec.sparse.jagged_tensor) and [tensorflow](https://www.tensorflow.org/api_docs/python/tf/RaggedTensor) I've added some language and an example [here](https://github.com/apache/arrow/pull/37166/files#diff-b54425cb176b53e51925c13a4d4e85cf7d03d4e1226e6d5bf4d7ae09923db8b3R197-R218). For now this would be purely metadata that would help converting arrow <-> jagged/ragged.
   
   I'll also send the updated doc to the mailing list to see if there's more feedback to be had.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1305610018


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,60 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList`` of the tensor shape where
+    the size of the list is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+  * **ndim** = the number of dimensions of the tensor.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = same as `arrow.fixed_shape_tensor`
+
+  * **permutation** = same as in `arrow.fixed_shape_tensor`
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object including number of
+  dimensions of the contained tensors as an integer with key **"ndim"**
+  plus optional dimension names with keys **"dim_names"** and ordering of
+  the dimensions with key **"permutation"**.
+
+  - Example: ``{ "ndim": 2}``
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "ndim": 3, "dim_names": ["C", "H", "W"]}``
+
+  - Example of permuted 3-dimensional tensor:
+
+    ``{ "ndim": 3, "permutation": [2, 0, 1]}``
+
+    This is the physical layout shape and the shape of the logical
+    layout would given an individual tensor of shape [100, 200, 500]
+    be ``[500, 100, 200]``.

Review Comment:
   Good point! `ndim` is provided by storage type and serializing `ndim` would be redundant. I'll refactor it out of JSON. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche merged PR #37166:
URL: https://github.com/apache/arrow/pull/37166


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1352675201


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensor's dimensions which are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with null. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, null, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``

Review Comment:
   I suppose so. So how about just:
   ```suggestion
     Minimal metadata is an empty string.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1303654040


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,60 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList`` of the tensor shape where
+    the size of the list is equal to the number of dimensions of the
+    tensor.

Review Comment:
   FixedSizeList of what?  Int32?  Int16?  Any integer type?



##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,60 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList`` of the tensor shape where
+    the size of the list is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+  * **ndim** = the number of dimensions of the tensor.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = same as `arrow.fixed_shape_tensor`
+
+  * **permutation** = same as in `arrow.fixed_shape_tensor`
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object including number of
+  dimensions of the contained tensors as an integer with key **"ndim"**
+  plus optional dimension names with keys **"dim_names"** and ordering of
+  the dimensions with key **"permutation"**.
+
+  - Example: ``{ "ndim": 2}``
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "ndim": 3, "dim_names": ["C", "H", "W"]}``
+
+  - Example of permuted 3-dimensional tensor:
+
+    ``{ "ndim": 3, "permutation": [2, 0, 1]}``
+
+    This is the physical layout shape and the shape of the logical
+    layout would given an individual tensor of shape [100, 200, 500]
+    be ``[500, 100, 200]``.

Review Comment:
   The way the description for this type (and for the fixed size tensor type actually) reads it makes it seem like there should be a `value_type` parameter in the JSON.  However, I think this parameter is actually expressed by the choice of storage type right?  E.g. if the storage type is `struct<data: List<Float64>, shape: FixedSizeList<3>>` then one can infer that the `value_type` is `Float64`.
   
   However, couldn't you also infer that `ndim` is 3?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1335885638


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,107 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape over dimensions that are guaranteed to stay
+    constant over of all tensors in the array if all their ragged dimension
+    sizes were replaced by 0.
+    An array containing tensor with shape (2, 3, 4) and uniform dimensions
+    (0, 2) would have uniform shape (2, 0, 4).
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object including number of
+  dimensions of the contained tensors as an integer with key **"ndim"**

Review Comment:
   It's not needed and I'd omit it for now. I'll add a minimal example. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1348787269


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensor's dimensions which are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with null. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, null, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``

Review Comment:
   ```suggestion
     Minimal metadata is an empty string.
   
     - In case of minimal metadata we assume that all dimensions are non-uniform
       and the ordering of dimensions is the same as the physical layout.
       Shape of individual tensors is given by ``shape`` ``FixedSizeList<int32>[ndim]`` 
       field which also implies number of dimensions with its length ``ndim``.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1343974885


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape of the dimensions that are guaranteed to stay
+    constant over all tensors in the array, with the shape of the ragged dimensions
+    set to 0.
+    An array containing tensor with shape (2, 3, 4) and uniform dimensions
+    (0, 2) would have uniform shape (2, 0, 4).
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object, that optionally includes
+  dimension names with keys **"dim_names"**, ordering of
+  dimensions with key **"permutation"**, indices of dimensions whose sizes
+  are guaranteed to remain constant with key **"uniform_dimensions"** and
+  shape of those dimensions with key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "dim_names": ["C", "H", "W"] }``
+
+  - Example with ``uniform_dimensions`` metadata for a set of color images
+    with variable width:

Review Comment:
   Is it legal to pass `uniform_dimensions` but not `uniform_shape`? 
   
   Come to think of it, perhaps we don't need two separate metadata members? For example, an absent `uniform_shape` would mean all dimensions are non-uniform. While a `null` entry in a present `uniform_shape` would point to a non-uniform dimension?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1343972176


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where

Review Comment:
   I don't think this was discussed and indeed `int64` would be more consistent, will change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347396052


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,109 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensors dimensions are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with 0s. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, 0, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "dim_names": ["C", "H", "W"] }``
+
+  - Example with ``uniform_shape`` metadata for a set of color images
+    with variable width:

Review Comment:
   ```suggestion
     - Example with ``uniform_shape`` metadata for a set of color images
       with fixed height, variable width and three color channels:
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1317182293


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,76 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.

Review Comment:
   Are you saying you get `ndim` at query time and expect `ndim=const` over the entire array or are you saying that every row will potentially have different `ndim`? If latter is the case we could use storage:
   ```cpp
   struct_({field("shape", list(uint32())), field("data", list(value_type))})
   ```
   That would work if we can assume consistent memory layout (e.g. always row major). However if we want to allow for other memory layouts then we also need to store per row permutation or strides.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1708438228

   Ah, well perhaps start with `VariableShapeTensor`? We don't need to do everything up front :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347591001


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensors dimensions are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with 0s. If ``uniform_shape`` is not provided it is assumed

Review Comment:
   Switched language to `null`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1748950569

   Thank for the suggestions @pitrou! If there's nothing further I'd tally the vote on ML and merge this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1348598707


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensor's dimensions which are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with null. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, null, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``

Review Comment:
   Sorry, one more small nitpick: the minimal metadata is actually _no_ metadata, which is typically represented as an empty string (I am actually not fully sure if in this case the metadata key could also just not be present in the field metadata), instead of an empty json dict (I don't think we should necessarily recommend using an empty dict)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1335858512


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,107 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape over dimensions that are guaranteed to stay
+    constant over of all tensors in the array if all their ragged dimension
+    sizes were replaced by 0.
+    An array containing tensor with shape (2, 3, 4) and uniform dimensions
+    (0, 2) would have uniform shape (2, 0, 4).
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object including number of
+  dimensions of the contained tensors as an integer with key **"ndim"**

Review Comment:
   Is this `ndim` needed? (you can infer it from the storage type, although it might be practical to include) 
   If we keep it, I would also give a concrete example of minimal metadata as the first example in the list just below.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1335850003


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,107 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.

Review Comment:
   ```suggestion
       Data type of the list elements is uniform across the entire column.
   ```
   
   I don't think it is provided in the metadata? (and that is also not needed, since it is part of the type already)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1343976607


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape of the dimensions that are guaranteed to stay
+    constant over all tensors in the array, with the shape of the ragged dimensions

Review Comment:
   Since pytorch uses "jagged" and "tensorflow" ragged we might as well use "non-uniform" :D.
   However the current text is being voted on, so if we change this we should probably communicate to the mailing list?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1762891291

   After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit a7fab04cdd45e59d7f3af11e1233bdcb835ff043.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/17702798242) has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1733835062

   Thanks for the review @jorisvandenbossche ! I've addressed your comments. Do you think the format proposal as is is ready to call a vote?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] AlenkaF commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1695336365

   How much extra complexity is added if we enable each tensor in an array to have it's own permutation defined?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1744792088

   > @pitrou do you think it'd be a good idea to split out the type definition into a separate PR or should we merge this in one piece?
   
   Yes, it would be nice to have a separate PR for the Arrow spec additions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1343974885


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are
+    guaranteed to remain constant. Indices are a subset of all possible
+    dimension indices ([0, 1, .., N-1]).
+    The uniform dimensions must still be represented in the `shape` field,
+    and must always be the same value for all tensors in the array -- this
+    allows code to interpret the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity. uniform_dimensions can be left out,
+    in which case it is assumed that all dimensions might be variable.
+
+  * **uniform_shape** = shape of the dimensions that are guaranteed to stay
+    constant over all tensors in the array, with the shape of the ragged dimensions
+    set to 0.
+    An array containing tensor with shape (2, 3, 4) and uniform dimensions
+    (0, 2) would have uniform shape (2, 0, 4).
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object, that optionally includes
+  dimension names with keys **"dim_names"**, ordering of
+  dimensions with key **"permutation"**, indices of dimensions whose sizes
+  are guaranteed to remain constant with key **"uniform_dimensions"** and
+  shape of those dimensions with key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "dim_names": ["C", "H", "W"] }``
+
+  - Example with ``uniform_dimensions`` metadata for a set of color images
+    with variable width:

Review Comment:
   Is it legal to pass `uniform_dimensions` but `uniform_shape`?
   
   Come to think of it, perhaps we don't need two separate metadata members? For example, an absent `uniform_shape` would mean all dimensions are non-uniform. While a `null` entry in a present `uniform_shape` would point to a non-uniform dimension?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1343973088


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_dimensions** = indices of dimensions whose sizes are

Review Comment:
   Here and below, clarify what happens if `uniform_dimensions` / `uniform_shape` is absent?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347522194


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensors dimensions are
+    guaranteed to stay constant in uniform dimensions and can vary in

Review Comment:
   ```suggestion
     * **uniform_shape** = sizes of individual tensor's dimensions which are
       guaranteed to stay constant in uniform dimensions and can vary in
   ```



##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensors dimensions are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with 0s. If ``uniform_shape`` is not provided it is assumed

Review Comment:
   Should we rather take "-1" istead of "0"? We have some other places where we use -1 for "unknown" (eg null counts)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347574587


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensors dimensions are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with 0s. If ``uniform_shape`` is not provided it is assumed

Review Comment:
   Or JSON supports `null` which maps quite naturally to what we are trying to express, IMHO.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1708356657

   @pitrou I imagine no, but perhaps something comes up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1749152135

   @jorisvandenbossche Can I let you give this the final review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1757796226

   Thanks a lot @rok !
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1352083555


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements (each list element is
+    a single tensor). The List's value type is the value type of the tensor,
+    such as an integer or floating-point type.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensor's dimensions which are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with null. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, null, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``

Review Comment:
   Ping @jorisvandenbossche 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mpelko commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "mpelko (via GitHub)" <gi...@apache.org>.
mpelko commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1728181658

   As part of a group working a lot with point clouds (radar and lidar data), this is something we are really looking for. Specifically, we are using arrow for the storage of the training data and so far dumping point cloud data as binary blob (pyarrow.binary type).
   
   All this to say `VariableShapeTensor` is *exactly* what we are looking for (in our case no need for `VariableDimensionTensor` so far).
   
   Thanks for the effort and looking forward to this landing and making it into the release!
   
   (happy to test or explain the use-case more if it helps in any way)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1344096057


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,112 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where

Review Comment:
   I think `int32` is also fine (just not `uint32`)? (we have other places where we used int32 by default for positive integers, eg the offsets)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347400600


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,109 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensors dimensions are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with 0s. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, 0, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:
+
+    ``{ "dim_names": ["C", "H", "W"] }``
+
+  - Example with ``uniform_shape`` metadata for a set of color images
+    with variable width:
+
+    ``{ "dim_names": ["H", "W", "C"], "uniform_shape": [400, 0, 3] }``
+
+  - Example of permuted 3-dimensional tensor:
+
+    ``{ "permutation": [2, 0, 1] }``
+
+    This is the physical layout shape and the shape of the logical
+    layout would given an individual tensor of shape [100, 200, 500]
+    be ``[500, 100, 200]``.
+
+.. note::
+
+  With the exception of ``permutation``, the parameters and storage
+  of VariableShapeTensor relate to the *physical* storage of the tensor.
+
+  For example, consider a tensor with:

Review Comment:
   Nit: markup for a code block
   ```suggestion
     For example, consider a tensor with::
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347395346


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,109 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.
+  * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+    the size of the list ``ndim`` is equal to the number of dimensions of the
+    tensor.
+
+* Extension type parameters:
+
+  * **value_type** = the Arrow data type of individual tensor elements.
+
+  Optional parameters describing the logical layout:
+
+  * **dim_names** = explicit names to tensor dimensions
+    as an array. The length of it should be equal to the shape
+    length and equal to the number of dimensions.
+
+    ``dim_names`` can be used if the dimensions have well-known
+    names and they map to the physical layout (row-major).
+
+  * **permutation**  = indices of the desired ordering of the
+    original dimensions, defined as an array.
+
+    The indices contain a permutation of the values [0, 1, .., N-1] where
+    N is the number of dimensions. The permutation indicates which
+    dimension of the logical layout corresponds to which dimension of the
+    physical tensor (the i-th dimension of the logical view corresponds
+    to the dimension with number ``permutations[i]`` of the physical tensor).
+
+    Permutation can be useful in case the logical order of
+    the tensor is a permutation of the physical order (row-major).
+
+    When logical and physical layout are equal, the permutation will always
+    be ([0, 1, .., N-1]) and can therefore be left out.
+
+  * **uniform_shape** = sizes of individual tensors dimensions are
+    guaranteed to stay constant in uniform dimensions and can vary in
+    non-uniform dimensions. This holds over all tensors in the array.
+    Sizes in uniform dimensions are represented with int32 values, while
+    sizes of the non-uniform dimensions are not known in advance and are
+    represented with 0s. If ``uniform_shape`` is not provided it is assumed
+    that all dimensions are non-uniform.
+    An array containing a tensor with shape (2, 3, 4) and whose first and
+    last dimensions are uniform would have ``uniform_shape`` (2, 0, 4).
+    This allows for interpreting the tensor correctly without accounting for
+    uniform dimensions while still permitting optional optimizations that
+    take advantage of the uniformity.
+
+* Description of the serialization:
+
+  The metadata must be a valid JSON object that optionally includes
+  dimension names with keys **"dim_names"** and  ordering of dimensions
+  with key **"permutation"**.
+  Shapes of tensors can be defined in a subset of dimensions by providing
+  key **"uniform_shape"**.
+  Minimal metadata is an empty JSON object.
+
+  - Example of minimal metadata is:
+
+    ``{}``
+
+  - Example with ``dim_names`` metadata for NCHW ordered data:

Review Comment:
   ```suggestion
     - Example with ``dim_names`` metadata for NCHW ordered data (note that the first
       logical dimension, ``N``, is mapped to the **data** List array: each element in the List
       is a CHW tensor and the List of tensors implicitly constitutes a single NCHW tensor):
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1347404051


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,109 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column.

Review Comment:
   ```suggestion
     * **data** is a ``List`` holding tensor elements (each list element is
       a single tensor). The List's value type is the value type of the tensor,
       such as an integer or floating-point type.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] lhoestq commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "lhoestq (via GitHub)" <gi...@apache.org>.
lhoestq commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1695286146

   No strong opinion regarding storing strides. Cc @thomasw21 I'm not even sure that nested tensors are used that often no ? I'm also pinging other experts to get more insights.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rok commented on a diff in pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1305701838


##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,60 @@ Fixed shape tensor
   by this specification. Instead, this extension type lets one use fixed shape tensors
   as elements in a field of a RecordBatch or a Table.
 
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+  is composed of **data** and **shape** fields describing a single
+  tensor per row:
+
+  * **data** is a ``List`` holding tensor elements of a single tensor.
+    Data type of the list elements is uniform across the entire column
+    and also provided in metadata.
+  * **shape** is a ``FixedSizeList`` of the tensor shape where
+    the size of the list is equal to the number of dimensions of the
+    tensor.

Review Comment:
   Indeed! Changed to:
   ```rst
     * **shape** is a ``FixedSizeList<uint32>[ndim]`` of the tensor shape where
       the size of the list ``ndim`` is equal to the number of dimensions of the
       tensor.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org