You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rok (via GitHub)" <gi...@apache.org> on 2023/10/04 09:55:30 UTC

[PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

rok opened a new pull request, #38008:
URL: https://github.com/apache/arrow/pull/38008

   ### Rationale for this change
   
   We want to add VariableShapeTensor extension type definition for arrays containing tensors with variable shapes.
   
   ### What changes are included in this PR?
   
   This adds a C++ implementation and a Python wrapper.
   
   ### Are these changes tested?
   
   Yes.
   
   ### Are there any user-facing changes?
   
   This adds a new extension type C++ and Python.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1521039481


##########
cpp/src/arrow/acero/CMakeLists.txt:
##########
@@ -117,7 +117,8 @@ endforeach()
 if(ARROW_TESTING)
   # test_nodes.cc isn't used by all tests but link to it for simple
   # CMakeLists.txt.
-  add_library(arrow_acero_testing OBJECT test_nodes.cc test_util_internal.cc)
+  add_library(arrow_acero_testing OBJECT test_nodes.cc test_util_internal.cc
+                                         ../extension/tensor_extension_array_test.cc)

Review Comment:
   Sorry if I missed the discussion on this but am wondering, why do we need to add the tensor tests here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1427126734


##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,100 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)

Review Comment:
   Good point! Added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409341931


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):

Review Comment:
   `to_numpy_ndarray_list`?
   I'm a bit uncertain how uses of this would look like.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1415802827


##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,114 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+    """
+
+    cdef void init(self, const shared_ptr[CDataType]& type) except *:
+        BaseExtensionType.init(self, type)
+        self.tensor_ext_type = <const CVariableShapeTensorType*> type.get()
+
+    @property
+    def value_type(self):
+        """
+        Data type of an individual tensor.
+        """
+        return pyarrow_wrap_data_type(self.tensor_ext_type.value_type())
+
+    @property
+    def ndim(self):
+        """
+        Number of dimensions of the tensors.
+        """
+        return self.tensor_ext_type.ndim()
+
+    @property
+    def dim_names(self):
+        """
+        Explicit names of the dimensions.
+        """
+        list_of_bytes = self.tensor_ext_type.dim_names()
+        if len(list_of_bytes) != 0:
+            return [frombytes(x) for x in list_of_bytes]
+        else:
+            return None
+
+    @property
+    def permutation(self):
+        """
+        Indices of the dimensions ordering.
+        """
+        indices = self.tensor_ext_type.permutation()
+        if len(indices) != 0:
+            return indices
+        else:
+            return None
+
+    @property
+    def uniform_shape(self):
+        """
+        Shape over dimensions that are guaranteed to be constant.
+        """
+        cdef:
+            vector[optional[int64_t]] c_uniform_shape = self.tensor_ext_type.uniform_shape()
+            length = c_uniform_shape.size()
+
+        if length == 0:
+            return None
+
+        uniform_shape = []
+        for i in range(length):
+            if c_uniform_shape[i].has_value():
+                uniform_shape.append(c_uniform_shape[i].value())
+            else:
+                uniform_shape.append(None)
+
+        return uniform_shape
+
+    def __arrow_ext_serialize__(self):
+        """
+        Serialized representation of metadata to reconstruct the type object.
+        """
+        return self.tensor_ext_type.Serialize()
+
+    @classmethod
+    def __arrow_ext_deserialize__(self, storage_type, serialized):
+        """
+        Return an VariableShapeTensor type instance from the storage type and serialized
+        metadata.
+        """
+        return self.tensor_ext_type.Deserialize(storage_type, serialized)
+
+    def __arrow_ext_class__(self):
+        return VariableShapeTensorArray
+
+    def __reduce__(self):
+        return variable_shape_tensor, (self.value_type, self.ndim,
+                                       self.dim_names, self.permutation,
+                                       self.uniform_shape)

Review Comment:
   Why not use `__arrow_ext_serialize__` and `__arrow_ext_deserialize__` generically instead of defining a specific `__reduce__` for each particular extension type?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1414521731


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):

Review Comment:
   Or maybe it's better to remove this for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542158156


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1543008837


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);

Review Comment:
   ```
   ARROW_CHECK(data_scalar->is_valid);
   ARROW_CHECK(shape_scalar->is_valid);
   ```
   Will fail fail if there are nulls in the data (e.g. in `ComputeStrides`).



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;

Review Comment:
   Same as above.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406527956


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')

Review Comment:
   If it incurs a copy, then the code should be corrected not to? :-)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2848506834


##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+#include <string>
+#include <vector>
+
+#include "arrow/extension_type.h"
+
+namespace arrow::extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type, int32_t ndim,
+                          std::vector<int64_t> permutation = {},
+                          std::vector<std::string> dim_names = {},
+                          std::vector<std::optional<int64_t>> uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("data", list(value_type)),
+                               ::arrow::field("shape", fixed_size_list(int32(), ndim))})),
+        value_type_(value_type),
+        ndim_(ndim),
+        permutation_(std::move(permutation)),
+        dim_names_(std::move(dim_names)),
+        uniform_shape_(std::move(uniform_shape)) {}
+
+  std::string extension_name() const override { return "arrow.variable_shape_tensor"; }
+  std::string ToString(bool show_metadata = false) const override;
+
+  /// Number of dimensions of tensor elements
+  int32_t ndim() const { return ndim_; }
+
+  /// Value type of tensor elements
+  const std::shared_ptr<DataType>& value_type() const { return value_type_; }
+
+  /// Permutation mapping from logical to physical memory layout of tensor elements
+  const std::vector<int64_t>& permutation() const { return permutation_; }
+
+  /// Dimension names of tensor elements. Dimensions are ordered physically.
+  const std::vector<std::string>& dim_names() const { return dim_names_; }
+
+  /// Shape of uniform dimensions.
+  const std::vector<std::optional<int64_t>>& uniform_shape() const {
+    return uniform_shape_;
+  }
+
+  bool ExtensionEquals(const ExtensionType& other) const override;
+
+  std::string Serialize() const override;
+
+  Result<std::shared_ptr<DataType>> Deserialize(
+      std::shared_ptr<DataType> storage_type,
+      const std::string& serialized_data) const override;
+
+  /// Create a VariableShapeTensorArray from ArrayData
+  std::shared_ptr<Array> MakeArray(std::shared_ptr<ArrayData> data) const override;
+
+  /// \brief Convert an ExtensionScalar to a Tensor
+  ///
+  /// This method will return a Tensor from ExtensionScalar with strides derived
+  /// from shape and permutation stored. Shape and dim_names will be permuted
+  /// according to permutation stored in the VariableShapeTensorType.
+  static Result<std::shared_ptr<Tensor>> MakeTensor(
+      const std::shared_ptr<ExtensionScalar>&);
+
+  /// \brief Create a VariableShapeTensorType instance
+  static Result<std::shared_ptr<DataType>> Make(
+      const std::shared_ptr<DataType>& value_type, int32_t ndim,
+      const std::vector<int64_t>& permutation = {},
+      const std::vector<std::string>& dim_names = {},
+      const std::vector<std::optional<int64_t>>& uniform_shape = {});
+
+ private:
+  std::shared_ptr<DataType> value_type_;
+  int32_t ndim_;
+  std::vector<int64_t> permutation_;
+  std::vector<std::string> dim_names_;
+  std::vector<std::optional<int64_t>> uniform_shape_;
+};
+
+/// \brief Return a VariableShapeTensorType instance.
+ARROW_EXPORT std::shared_ptr<DataType> variable_shape_tensor(
+    const std::shared_ptr<DataType>& value_type, int32_t ndim,
+    const std::vector<int64_t>& permutation = {},
+    const std::vector<std::string>& dim_names = {},
+    const std::vector<std::optional<int64_t>>& uniform_shape = {});

Review Comment:
   Same as above https://github.com/apache/arrow/pull/38008/commits/0de39f23c29d25f3da245ac9e297ff03c7d578c9



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+#include <string>
+#include <vector>
+
+#include "arrow/extension_type.h"
+
+namespace arrow::extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type, int32_t ndim,
+                          std::vector<int64_t> permutation = {},
+                          std::vector<std::string> dim_names = {},
+                          std::vector<std::optional<int64_t>> uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("data", list(value_type)),
+                               ::arrow::field("shape", fixed_size_list(int32(), ndim))})),
+        value_type_(value_type),
+        ndim_(ndim),
+        permutation_(std::move(permutation)),
+        dim_names_(std::move(dim_names)),
+        uniform_shape_(std::move(uniform_shape)) {}
+
+  std::string extension_name() const override { return "arrow.variable_shape_tensor"; }
+  std::string ToString(bool show_metadata = false) const override;
+
+  /// Number of dimensions of tensor elements
+  int32_t ndim() const { return ndim_; }
+
+  /// Value type of tensor elements
+  const std::shared_ptr<DataType>& value_type() const { return value_type_; }
+
+  /// Permutation mapping from logical to physical memory layout of tensor elements
+  const std::vector<int64_t>& permutation() const { return permutation_; }
+
+  /// Dimension names of tensor elements. Dimensions are ordered physically.
+  const std::vector<std::string>& dim_names() const { return dim_names_; }
+
+  /// Shape of uniform dimensions.
+  const std::vector<std::optional<int64_t>>& uniform_shape() const {
+    return uniform_shape_;
+  }
+
+  bool ExtensionEquals(const ExtensionType& other) const override;
+
+  std::string Serialize() const override;
+
+  Result<std::shared_ptr<DataType>> Deserialize(
+      std::shared_ptr<DataType> storage_type,
+      const std::string& serialized_data) const override;
+
+  /// Create a VariableShapeTensorArray from ArrayData
+  std::shared_ptr<Array> MakeArray(std::shared_ptr<ArrayData> data) const override;
+
+  /// \brief Convert an ExtensionScalar to a Tensor
+  ///
+  /// This method will return a Tensor from ExtensionScalar with strides derived
+  /// from shape and permutation stored. Shape and dim_names will be permuted
+  /// according to permutation stored in the VariableShapeTensorType.
+  static Result<std::shared_ptr<Tensor>> MakeTensor(
+      const std::shared_ptr<ExtensionScalar>&);
+
+  /// \brief Create a VariableShapeTensorType instance
+  static Result<std::shared_ptr<DataType>> Make(
+      const std::shared_ptr<DataType>& value_type, int32_t ndim,
+      const std::vector<int64_t>& permutation = {},
+      const std::vector<std::string>& dim_names = {},
+      const std::vector<std::optional<int64_t>>& uniform_shape = {});

Review Comment:
   Done, https://github.com/apache/arrow/pull/38008/commits/0de39f23c29d25f3da245ac9e297ff03c7d578c9



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2835230060


##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1712,7 +1723,7 @@ def test_tensor_type_is_picklable(pickle_module):
         'fixed_shape_tensor[value_type=int64, shape=[2,2,3], dim_names=[C,H,W]]'
     )
 ])
-def test_tensor_type_str(tensor_type, text):
+def test_tensor_type_str(tensor_type, text, pickle_module):

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542164378


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1543000177


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& permutation,
+                             std::vector<int64_t>* strides) {
+  const auto fixed_width_type =
+      internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);
+  }
+  const int byte_width = value_type->byte_width();
+
+  int64_t remaining = 0;
+  if (!shape.empty() && shape.front() > 0) {
+    remaining = byte_width;
+    for (auto i : permutation) {
+      if (i > 0) {
+        if (internal::MultiplyWithOverflow(remaining, shape[i], &remaining)) {
+          return Status::Invalid(
+              "Strides computed from shape would not fit in 64-bit integer");
+        }
+      }
+    }
+  }
+
+  if (remaining == 0) {
+    strides->assign(shape.size(), byte_width);
+    return Status::OK();
+  }
+
+  strides->push_back(remaining);
+  for (auto i : permutation) {
+    if (i > 0) {
+      remaining /= shape[i];

Review Comment:
   Good point. I'll make an issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542174266


##########
cpp/src/arrow/extension/tensor_extension_array_test.cc:
##########
@@ -165,47 +170,51 @@ TEST_F(TestExtensionType, CreateFromArray) {
   ASSERT_EQ(ext_arr->null_count(), 0);
 }
 
+template <typename T>

Review Comment:
   Indeed. Removing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542158037


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,

Review Comment:
   Moved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1484014373


##########
python/pyarrow/array.pxi:
##########
@@ -3609,7 +3609,7 @@ cdef class FixedShapeTensorArray(ExtensionArray):
         and the rest of the dimensions will match the permuted shape of the fixed
         shape tensor.
 
-        The conversion is zero-copy.
+        The conversion is zero-copy if data is primitive numeric and without nulls.

Review Comment:
   In this case it is always zero-copy (because tensors only support numeric data without nulls, so AFAIK it would raise an error instead of being non-zero-copy)
   
   Also, can you can the same note to the Scalar `to_numpy()` docstring?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1407882212


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   Do you think it would make sense to provide a `Scalar.GetSlicedBuffer()`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1829792012

   @pitrou with the slicing issue seemingly resolved I think this is ready for review. Could I ask you to take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409319643


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());

Review Comment:
   Changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409319185


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {

Review Comment:
   Indeed, spec now states `uniform_shape.size() == ndim`. Changed appropriately.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-2153475314

   @pitrou any time to spare for a review here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3893674529

   @pitrou I'd like to merge this next week, do you recall if there were somesort of considerations stopping it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-2343516805

   Sorry to go here immediately after #13901 @pitrou, but do you think we could review this now? :)
   It's already rebased.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-2036793922

   @pitrou ping


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406526981


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')

Review Comment:
   Yes, this is temporary. I'd like to have this just call `to_tensor().to_numpy()` to get a correctly strided Tensor. (although I'm not sure if that will incur a copy. If yes I'd go the way you're proposing.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1410029983


##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1472,10 +1618,24 @@ def test_tensor_type_is_picklable(pickle_module):
         'fixed_shape_tensor[value_type=int64, shape=[2,2,3], dim_names=[C,H,W]]'
     )
 ])
-def test_tensor_type_str(tensor_type, text):
+def test_tensor_type_str(tensor_type, text, pickle_module):

Review Comment:
   Done.



##########
python/pyarrow/types.pxi:
##########
@@ -4948,6 +5056,121 @@ def fixed_shape_tensor(DataType value_type, shape, dim_names=None, permutation=N
     return out
 
 
+def variable_shape_tensor(DataType value_type, ndim, dim_names=None, permutation=None,
+                          uniform_shape=None):
+    """
+    Create instance of variable shape tensor extension type with number of
+    dimensions and optional names of tensor dimensions and indices of the
+    desired logical ordering of dimensions.
+
+    Parameters
+    ----------
+    value_type : DataType
+        Data type of individual tensor elements.
+    ndim : integer
+        The number of dimensions of the contained tensors.
+    dim_names : tuple or list of strings, default None
+        Explicit names to tensor dimensions.
+    permutation : tuple or list integers, default None
+        Indices of the desired ordering of the original dimensions.
+        The indices contain a permutation of the values ``[0, 1, .., N-1]`` where
+        N is the number of dimensions. The permutation indicates which dimension
+        of the logical layout corresponds to which dimension of the physical tensor.
+        For more information on this parameter see
+        :ref:`fixed_shape_tensor_extension`.
+    uniform_shape : tuple or list of integers, default None
+        Shape of dimensions that are guaranteed to stay constant over all tensors
+        in the array if all their non-dimensions sizes were replaced by None.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+    >>> tensor_type
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Inspect the data type:
+
+    >>> tensor_type.value_type
+    DataType(int32)
+    >>> tensor_type.ndim
+    2
+
+    Create a table with variable shape tensor extension array:
+
+    >>> fields = [pa.field("shape", pa.list_(pa.uint32(), 2)), pa.field("data", pa.list_(pa.int32()))]
+    >>> storage = pa.array([([2, 3], [1, 2, 3, 4, 5, 6]), ([1, 2], [7, 8])], type=pa.struct(fields))
+    >>> tensor = pa.ExtensionArray.from_storage(tensor_type, storage)
+    >>> pa.table([tensor], names=["tensor_array"])
+    pyarrow.Table
+    tensor_array: extension<arrow.variable_shape_tensor>
+    ----
+    tensor_array: [  -- is_valid: all not null
+      -- child 0 type: fixed_size_list<item: uint32>[2]
+    [[2,3],[1,2]]
+      -- child 1 type: list<item: int32>
+    [[1,2,3,4,5,6],[7,8]]]
+
+    Create an instance of variable shape tensor extension type with names
+    of tensor dimensions:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     dim_names=['C', 'H', 'W'])
+    >>> tensor_type.dim_names
+    ['C', 'H', 'W']
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+
+    Returns
+    -------
+    type : VariableShapeTensorType
+    """
+
+    cdef:
+        uint32_t c_ndim
+        vector[int64_t] c_permutation
+        vector[c_string] c_dim_names
+        vector[optional[int64_t]] c_uniform_shape
+        shared_ptr[CDataType] c_tensor_ext_type
+
+    assert value_type is not None
+    assert ndim is not None
+
+    c_ndim = ndim
+
+    if permutation is not None:
+        for i in permutation:
+            c_permutation.push_back(i)
+
+    if dim_names is not None:
+        for x in dim_names:
+            c_dim_names.push_back(tobytes(x))
+
+    if uniform_shape is not None:
+        for x in uniform_shape:
+            if x is None:
+                c_uniform_shape.push_back(<optional[int64_t]>nullopt)
+            else:
+                c_uniform_shape.push_back(<optional[int64_t]>(<int64_t>x))
+
+    cdef VariableShapeTensorType out = VariableShapeTensorType.__new__(VariableShapeTensorType)
+
+    c_tensor_ext_type = GetResultValue(CVariableShapeTensorType.Make(
+        value_type.sp_type, c_ndim, c_permutation, c_dim_names, c_uniform_shape))

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409313923


##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,

Review Comment:
   Indeed, I missed this after the discussion on the spec PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1423714126


##########
cpp/src/arrow/extension/fixed_shape_tensor.h:
##########
@@ -15,9 +15,21 @@
 // specific language governing permissions and limitations
 // under the License.
 
+#pragma once
+
 #include "arrow/extension_type.h"
 
 namespace arrow {
+namespace internal {

Review Comment:
   Should then be removed here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1423722525


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):

Review Comment:
   Let's remove this `to_numpy_ndarray` method?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542162406


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {

Review Comment:
   Added length check, but we're currently allowing for null values, [see test](https://github.com/apache/arrow/pull/38008/files#diff-a3dbccdee5e48b964a02b9593b4c6de13858f513b866826b8303c7cba205611cR814-R816).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542160937


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }

Review Comment:
   Indeed. Moved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542172903


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {

Review Comment:
   Indeed, removing the redundant checks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1864866735

   @pitrou ping :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2835246091


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,314 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/util/print_internal.h"
+#include "arrow/util/sort_internal.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow::extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = internal::checked_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (internal::checked_cast<const FixedSizeListType&>(*storage_type->field(1)->type())
+          .value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(0)->type()->field(0)->type();
+  const uint32_t ndim =
+      internal::checked_cast<const FixedSizeListType&>(*storage_type->field(1)->type())
+          .list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    permutation.reserve(ndim);
+    for (const auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }

Review Comment:
   Added some checks.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,314 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/util/print_internal.h"
+#include "arrow/util/sort_internal.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow::extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = internal::checked_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {

Review Comment:
   Factored out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1427126351


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,160 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.float64(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.int32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.float64()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: int32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: double>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        If permutation is non-trivial a strided numpy arrays are produced.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)

Review Comment:
   Added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1426742968


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,160 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.float64(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.int32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.float64()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: int32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: double>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        If permutation is non-trivial a strided numpy arrays are produced.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)
+        <pyarrow.lib.VariableShapeTensorArray object at ...>
+        -- is_valid: all not null
+        -- child 0 type: fixed_size_list<item: int32>[2]
+          [
+            [
+              2,
+              3
+            ],
+            [
+              1,
+              2
+            ]
+          ]
+        -- child 1 type: list<item: float>
+          [
+            [
+              1,
+              2,
+              3,
+              4,
+              5,
+              6
+            ],
+            [
+              7,
+              8
+            ]
+          ]
+        """
+        assert isinstance(obj, list), 'obj must be a list of numpy arrays'
+        numpy_type = obj[0].dtype
+        ndim = obj[0].ndim
+        bw = numpy_type.itemsize
+
+        permutation = (-np.array(obj[0].strides)).argsort()
+
+        if not all([o.dtype == numpy_type for o in obj]):
+            raise TypeError('All numpy arrays need to have the same dtype.')
+
+        if not all([o.ndim == ndim for o in obj]):
+            raise ValueError('All numpy arrays need to have the same ndim.')
+
+        if not all([np.array_equal((-np.array(o.strides)).argsort(), permutation) for o in obj]):
+            raise ValueError('All numpy arrays need to have the same permutation.')
+
+        arrow_type=from_numpy_dtype(numpy_type)
+        values=array([np.lib.stride_tricks.as_strided(
+            o, shape=(np.prod(o.shape),), strides=(bw,)) for o in obj], list_(arrow_type))
+        shapes=array([o.shape for o in obj], list_(int32(), list_size=ndim))
+        struct_arr=StructArray.from_arrays([shapes, values], names=["shape", "data"])

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1426742620


##########
python/pyarrow/types.pxi:
##########
@@ -4948,6 +5028,122 @@ def fixed_shape_tensor(DataType value_type, shape, dim_names=None, permutation=N
     return out
 
 
+def variable_shape_tensor(DataType value_type, ndim, dim_names=None, permutation=None,
+                          uniform_shape=None):
+    """
+    Create instance of variable shape tensor extension type with number of
+    dimensions and optional names of tensor dimensions and indices of the
+    desired logical ordering of dimensions.
+
+    Parameters
+    ----------
+    value_type : DataType
+        Data type of individual tensor elements.
+    ndim : integer
+        The number of dimensions of the contained tensors.
+    dim_names : tuple or list of strings, default None
+        Explicit names to tensor dimensions.
+    permutation : tuple or list integers, default None
+        Indices of the desired ordering of the original dimensions.
+        The indices contain a permutation of the values ``[0, 1, .., N-1]`` where
+        N is the number of dimensions. The permutation indicates which dimension
+        of the logical layout corresponds to which dimension of the physical tensor.
+        For more information on this parameter see
+        :ref:`fixed_shape_tensor_extension`.

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1435415050


##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1317,6 +1317,54 @@ def test_tensor_type():
     assert tensor_type.dim_names == ['C', 'H', 'W']
     assert tensor_type.permutation is None
 
+    tensor_type = pa.variable_shape_tensor(pa.int8(), 2)

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1527275970


##########
cpp/src/arrow/acero/CMakeLists.txt:
##########
@@ -117,7 +117,8 @@ endforeach()
 if(ARROW_TESTING)
   # test_nodes.cc isn't used by all tests but link to it for simple
   # CMakeLists.txt.
-  add_library(arrow_acero_testing OBJECT test_nodes.cc test_util_internal.cc)
+  add_library(arrow_acero_testing OBJECT test_nodes.cc test_util_internal.cc
+                                         ../extension/tensor_extension_array_test.cc)

Review Comment:
   I'm don't remember why this was added but it seems redundant at the moment :). Removing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-2025227651

   Thanks for the extensive review @pitrou ! I've resolved some points and replied to others. Could you please do another pass?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1814644769

   Hi @rok! It would be great to have this in 15.0.0 (added the milestone for visibility). Do you think you have enough bandwidth to work on it at the moment?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406550632


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   But how would it be applied in Python? Getting a scalar in Python just returns an `ExtensionScalar` that is unaware of it's offset and hence can't correctly slice from array's underlying buffer. Maybe I'm missing something :D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1410028483


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();

Review Comment:
   I see. Changed to return `Status::Invalid`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409338993


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(this->value_type(), shape, this->permutation(), &strides));
+
+  const auto array = std::static_pointer_cast<FixedSizeListScalar>(data)->value;
+  const auto byte_width = this->value_type()->byte_width();
+  const auto start_position = array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  // Create a slice of the buffer
+  std::shared_ptr<arrow::Buffer> buffer =
+      arrow::SliceBuffer(array->data()->buffers[1], start_position, size * byte_width);
+
+  return Tensor::Make(this->value_type(), buffer, shape, strides, this->dim_names());
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Make(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  if (!permutation.empty() && permutation.size() != ndim) {
+    return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
+                           " Got: ", permutation.size());
+  }
+  if (!dim_names.empty() && dim_names.size() != ndim) {
+    return Status::Invalid("dim_names size must match ndim. Expected: ", ndim,
+                           " Got: ", dim_names.size());
+  }
+  if (uniform_shape.size() > ndim) {
+    return Status::Invalid("uniform_shape size must be less or equal ndim.");
+  }

Review Comment:
   Added a permutation check:
   ```cpp
     if (!permutation.empty()) {
       if (permutation.size() != static_cast<size_t>(ndim)) {
         return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
                                " Got: ", permutation.size());
       }
       const std::set<int> permutation_set(permutation.begin(), permutation.end());
       if (permutation_set.size() != permutation.size()) {
         return Status::Invalid("permutation must be a valid permutation vector");
       }
       for (auto p : permutation) {
         if (p < 0 || ndim <= p) {
           return Status::Invalid("permutation must be a valid permutation vector");
         }
       }
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3954683816

   Thanks for pushing this through @rok ! It's a good thing we finally have an implementation :tada: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "pimdh (via GitHub)" <gi...@apache.org>.
pimdh commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3522691312

   > @pimdh Yes, that seems right.
   > Now that some time has passed would you say there's still need for `VariableShapeTensorArray` with LargeList for `data`? We would need to amend the extension type or introduce `LargeVariableShapeTensorArray`.
   
   No, as I didn't yet run into any limits of the int32 arrays, I have no need for the large version.
   Appreciate your work on this! Would love to replace our internal version with an official one.
   Also, then others, like the Ray library, can use the standardised version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1814711175

   @AlenkaF yes I'd very much like to continue working on this, I was waiting for the release before chasing reviewers :D
   I'll rebase tonight and would very much appreciate reviews afterwards.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406555540


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   The `ExtensionScalar` has an underlying storage scalar, no?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1407877772


##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,
+                          const std::vector<int64_t>& permutation = {},
+                          const std::vector<std::string>& dim_names = {},
+                          const std::vector<std::optional<int64_t>>& uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(uint32(), ndim)),
+                               ::arrow::field("data", list(value_type))})),
+        value_type_(value_type),
+        permutation_(permutation),
+        dim_names_(dim_names),
+        uniform_shape_(uniform_shape) {}

Review Comment:
   Consider passing by value where preferable and moving when forwarded along.



##########
cpp/src/arrow/extension/fixed_shape_tensor.h:
##########
@@ -15,9 +15,21 @@
 // specific language governing permissions and limitations
 // under the License.
 
+#pragma once
+
 #include "arrow/extension_type.h"
 
 namespace arrow {
+namespace internal {

Review Comment:
   It would be better for readability to put this into its own header, for example `tensor_internal.h`.



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,

Review Comment:
   I would expect `int32_t` here.



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,
+                          const std::vector<int64_t>& permutation = {},
+                          const std::vector<std::string>& dim_names = {},
+                          const std::vector<std::optional<int64_t>>& uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(uint32(), ndim)),
+                               ::arrow::field("data", list(value_type))})),
+        value_type_(value_type),
+        permutation_(permutation),
+        dim_names_(dim_names),
+        uniform_shape_(uniform_shape) {}
+
+  std::string extension_name() const override { return "arrow.variable_shape_tensor"; }
+
+  /// Number of dimensions of tensor elements
+  uint32_t ndim() const {
+    std::shared_ptr<DataType> storage_type = this->storage_type()->field(0)->type();
+    return std::static_pointer_cast<FixedSizeListType>(storage_type)->list_size();

Review Comment:
   It may help to store `ndim` directly on this type instead of recomputing it like this.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();

Review Comment:
   Can take const references here
   ```suggestion
     const auto& tensor_scalar = internal::checked_cast<const StructScalar&>(*scalar->value);
     const auto& fields = this->storage_type()->fields();
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;

Review Comment:
   Here as well, you can probably take const refs?
   ```suggestion
     const auto& shape_array =
         checked_cast<const Int32Array&>(*checked_cast<const FixedSizeListScalar&>(*shape_scalar)->value);
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(this->value_type(), shape, this->permutation(), &strides));
+
+  const auto array = std::static_pointer_cast<FixedSizeListScalar>(data)->value;
+  const auto byte_width = this->value_type()->byte_width();
+  const auto start_position = array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  // Create a slice of the buffer
+  std::shared_ptr<arrow::Buffer> buffer =
+      arrow::SliceBuffer(array->data()->buffers[1], start_position, size * byte_width);
+
+  return Tensor::Make(this->value_type(), buffer, shape, strides, this->dim_names());
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Make(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  if (!permutation.empty() && permutation.size() != ndim) {
+    return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
+                           " Got: ", permutation.size());
+  }
+  if (!dim_names.empty() && dim_names.size() != ndim) {
+    return Status::Invalid("dim_names size must match ndim. Expected: ", ndim,
+                           " Got: ", dim_names.size());
+  }
+  if (uniform_shape.size() > ndim) {
+    return Status::Invalid("uniform_shape size must be less or equal ndim.");
+  }

Review Comment:
   Can you also type-check value type and check the shape size?
   (might even check that permutation has valid values...)



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):

Review Comment:
   The name seems a bit of a misnomer? It returns a list, not a `ndarray`...
   
   I also don't know if this API is useful. @jorisvandenbossche What do you think?



##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1353,6 +1386,70 @@ def test_tensor_class_methods():
         arr.to_numpy_ndarray()
 
 
+@pytest.mark.parametrize("value_type", (np.int8, np.int32, np.int64, np.float64))
+def test_variable_shape_tensor_class_methods(value_type):
+    ndim = 2
+    shape_type = pa.list_(pa.uint32(), ndim)
+    arrow_type = pa.from_numpy_dtype(value_type)
+    tensor_type = pa.variable_shape_tensor(
+        arrow_type,
+        ndim,
+        dim_names=["H", "W"],
+        permutation=[0, 1],
+        uniform_shape=[None, None],
+    )
+    fields = [pa.field("shape", shape_type), pa.field("data", pa.list_(arrow_type))]
+
+    shapes = pa.array([[2, 3], [2, 1]], shape_type)
+    values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(arrow_type))
+    struct_arr = pa.StructArray.from_arrays([shapes, values], fields=fields)
+    arr = pa.ExtensionArray.from_storage(tensor_type, struct_arr)
+    basic_arr = pa.ExtensionArray.from_storage(
+        pa.variable_shape_tensor(arrow_type, ndim), struct_arr
+    )
+
+    storage = pa.array(
+        [([2, 3], [1, 2, 3, 4, 5, 6]), ([2, 1], [7, 8])], type=pa.struct(fields)
+    )
+    assert pa.ExtensionArray.from_storage(tensor_type, storage).equals(arr)
+
+    assert arr.type == tensor_type
+
+    ndarray_list = [
+        np.array([[1, 2, 3], [4, 5, 6]], dtype=value_type),
+        np.array([[7], [8]], dtype=value_type),
+    ]
+    assert all(zip(x == y for x, y in zip(arr.to_numpy_ndarray(), ndarray_list)))

Review Comment:
   This is... complicated. Why not something like
   ```suggestion
       assert arr.to_numpy_ndarray() == ndarray_list
   ```



##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1317,6 +1317,39 @@ def test_tensor_type():
     assert tensor_type.dim_names == ['C', 'H', 'W']
     assert tensor_type.permutation is None
 
+    tensor_type = pa.variable_shape_tensor(pa.int8(), 2)
+    expected_storage_type = pa.struct([
+        pa.field("shape", pa.list_(pa.uint32(), 2)),
+        pa.field("data", pa.list_(pa.int8()))
+    ])
+    assert tensor_type.extension_name == "arrow.variable_shape_tensor"
+    assert tensor_type.storage_type == expected_storage_type
+    assert tensor_type.ndim == 2
+    assert tensor_type.dim_names is None
+    assert tensor_type.permutation is None
+
+    tensor_type = pa.variable_shape_tensor(pa.int64(), 3, dim_names=['C', 'H', 'W'])
+    expected_storage_type = pa.struct([
+        pa.field("shape", pa.list_(pa.uint32(), 3)),
+        pa.field("data", pa.list_(pa.int64()))
+    ])
+    assert tensor_type.extension_name == "arrow.variable_shape_tensor"
+    assert tensor_type.storage_type == expected_storage_type
+    assert tensor_type.ndim == 3
+    assert tensor_type.dim_names == ['C', 'H', 'W']
+    assert tensor_type.permutation is None
+
+    tensor_type = pa.variable_shape_tensor(pa.bool_(), 2, permutation=[1, 0])
+    expected_storage_type = pa.struct([
+        pa.field("shape", pa.list_(pa.uint32(), 2)),
+        pa.field("data", pa.list_(pa.bool_()))
+    ])
+    assert tensor_type.extension_name == "arrow.variable_shape_tensor"
+    assert tensor_type.storage_type == expected_storage_type
+    assert tensor_type.ndim == 2
+    assert tensor_type.dim_names is None
+    assert tensor_type.permutation == [1, 0]

Review Comment:
   Is `uniform_shape` supported? If so, add tests for it?



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))

Review Comment:
   Can you use a different value type than the shape type? For example `float64`.



##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1353,6 +1386,70 @@ def test_tensor_class_methods():
         arr.to_numpy_ndarray()
 
 
+@pytest.mark.parametrize("value_type", (np.int8, np.int32, np.int64, np.float64))
+def test_variable_shape_tensor_class_methods(value_type):
+    ndim = 2
+    shape_type = pa.list_(pa.uint32(), ndim)
+    arrow_type = pa.from_numpy_dtype(value_type)
+    tensor_type = pa.variable_shape_tensor(
+        arrow_type,
+        ndim,
+        dim_names=["H", "W"],
+        permutation=[0, 1],
+        uniform_shape=[None, None],
+    )
+    fields = [pa.field("shape", shape_type), pa.field("data", pa.list_(arrow_type))]
+
+    shapes = pa.array([[2, 3], [2, 1]], shape_type)
+    values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(arrow_type))

Review Comment:
   What if I want a null tensor somewhere in this list?



##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,114 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+    """
+
+    cdef void init(self, const shared_ptr[CDataType]& type) except *:
+        BaseExtensionType.init(self, type)
+        self.tensor_ext_type = <const CVariableShapeTensorType*> type.get()
+
+    @property
+    def value_type(self):
+        """
+        Data type of an individual tensor.
+        """
+        return pyarrow_wrap_data_type(self.tensor_ext_type.value_type())
+
+    @property
+    def ndim(self):
+        """
+        Number of dimensions of the tensors.
+        """
+        return self.tensor_ext_type.ndim()
+
+    @property
+    def dim_names(self):
+        """
+        Explicit names of the dimensions.
+        """
+        list_of_bytes = self.tensor_ext_type.dim_names()
+        if len(list_of_bytes) != 0:
+            return [frombytes(x) for x in list_of_bytes]
+        else:
+            return None
+
+    @property
+    def permutation(self):
+        """
+        Indices of the dimensions ordering.
+        """
+        indices = self.tensor_ext_type.permutation()
+        if len(indices) != 0:
+            return indices
+        else:
+            return None
+
+    @property
+    def uniform_shape(self):
+        """
+        Shape over dimensions that are guaranteed to be constant.
+        """
+        cdef:
+            vector[optional[int64_t]] c_uniform_shape = self.tensor_ext_type.uniform_shape()
+            length = c_uniform_shape.size()
+
+        if length == 0:
+            return None
+
+        uniform_shape = []
+        for i in range(length):
+            if c_uniform_shape[i].has_value():
+                uniform_shape.append(c_uniform_shape[i].value())
+            else:
+                uniform_shape.append(None)
+
+        return uniform_shape
+
+    def __arrow_ext_serialize__(self):
+        """
+        Serialized representation of metadata to reconstruct the type object.
+        """
+        return self.tensor_ext_type.Serialize()
+
+    @classmethod
+    def __arrow_ext_deserialize__(self, storage_type, serialized):
+        """
+        Return an VariableShapeTensor type instance from the storage type and serialized
+        metadata.
+        """
+        return self.tensor_ext_type.Deserialize(storage_type, serialized)

Review Comment:
   I don't think it makes sense to define these methods. When would they be called?



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {

Review Comment:
   I'm not sure it should be allowed to have `uniform_shape` smaller than the number of dimensions.



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))

Review Comment:
   Should be `int32`.



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,
+                          const std::vector<int64_t>& permutation = {},
+                          const std::vector<std::string>& dim_names = {},
+                          const std::vector<std::optional<int64_t>>& uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(uint32(), ndim)),

Review Comment:
   Shape should be `int32` by the spec, not `uint32`.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());

Review Comment:
   Use `checked_cast` here?



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();

Review Comment:
   Can you do a bit more type checking and also ensure the right number of fields are present?



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,
+                          const std::vector<int64_t>& permutation = {},
+                          const std::vector<std::string>& dim_names = {},
+                          const std::vector<std::optional<int64_t>>& uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(uint32(), ndim)),
+                               ::arrow::field("data", list(value_type))})),
+        value_type_(value_type),
+        permutation_(permutation),
+        dim_names_(dim_names),
+        uniform_shape_(uniform_shape) {}
+
+  std::string extension_name() const override { return "arrow.variable_shape_tensor"; }
+
+  /// Number of dimensions of tensor elements
+  uint32_t ndim() const {
+    std::shared_ptr<DataType> storage_type = this->storage_type()->field(0)->type();
+    return std::static_pointer_cast<FixedSizeListType>(storage_type)->list_size();
+  }
+
+  /// Value type of tensor elements
+  const std::shared_ptr<DataType> value_type() const { return value_type_; }

Review Comment:
   ```suggestion
     const std::shared_ptr<DataType>& value_type() const { return value_type_; }
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,
+                          const std::vector<int64_t>& permutation = {},
+                          const std::vector<std::string>& dim_names = {},
+                          const std::vector<std::optional<int64_t>>& uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(uint32(), ndim)),
+                               ::arrow::field("data", list(value_type))})),
+        value_type_(value_type),
+        permutation_(permutation),
+        dim_names_(dim_names),
+        uniform_shape_(uniform_shape) {}
+
+  std::string extension_name() const override { return "arrow.variable_shape_tensor"; }
+
+  /// Number of dimensions of tensor elements
+  uint32_t ndim() const {
+    std::shared_ptr<DataType> storage_type = this->storage_type()->field(0)->type();
+    return std::static_pointer_cast<FixedSizeListType>(storage_type)->list_size();
+  }
+
+  /// Value type of tensor elements
+  const std::shared_ptr<DataType> value_type() const { return value_type_; }
+
+  /// Permutation mapping from logical to physical memory layout of tensor elements
+  const std::vector<int64_t>& permutation() const { return permutation_; }
+
+  /// Dimension names of tensor elements. Dimensions are ordered physically.
+  const std::vector<std::string>& dim_names() const { return dim_names_; }
+
+  /// Shape of uniform dimensions.
+  const std::vector<std::optional<int64_t>>& uniform_shape() const {
+    return uniform_shape_;
+  }
+
+  bool ExtensionEquals(const ExtensionType& other) const override;
+
+  std::string Serialize() const override;
+
+  Result<std::shared_ptr<DataType>> Deserialize(
+      std::shared_ptr<DataType> storage_type,
+      const std::string& serialized_data) const override;
+
+  /// Create a VariableShapeTensorArray from ArrayData
+  std::shared_ptr<Array> MakeArray(std::shared_ptr<ArrayData> data) const override;
+
+  /// Convert an ExtensionScalar to a Tensor
+  Result<std::shared_ptr<Tensor>> GetTensor(
+      const std::shared_ptr<ExtensionScalar>&) const;
+
+  /// \brief Create a VariableShapeTensorType instance
+  static Result<std::shared_ptr<DataType>> Make(
+      const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,

Review Comment:
   Here as well, can pass some of these by value.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));

Review Comment:
   This is not checking that shape values are valid...



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(this->value_type(), shape, this->permutation(), &strides));
+
+  const auto array = std::static_pointer_cast<FixedSizeListScalar>(data)->value;

Review Comment:
   It's not a fixed-size list, is it? :-)
   Also, can use a const ref.
   ```suggestion
     const auto& array = checked_cast<const BaseListScalar&>(*data)->value;
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(this->value_type(), shape, this->permutation(), &strides));
+
+  const auto array = std::static_pointer_cast<FixedSizeListScalar>(data)->value;
+  const auto byte_width = this->value_type()->byte_width();
+  const auto start_position = array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  // Create a slice of the buffer
+  std::shared_ptr<arrow::Buffer> buffer =
+      arrow::SliceBuffer(array->data()->buffers[1], start_position, size * byte_width);
+
+  return Tensor::Make(this->value_type(), buffer, shape, strides, this->dim_names());
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Make(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  if (!permutation.empty() && permutation.size() != ndim) {
+    return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
+                           " Got: ", permutation.size());
+  }
+  if (!dim_names.empty() && dim_names.size() != ndim) {
+    return Status::Invalid("dim_names size must match ndim. Expected: ", ndim,
+                           " Got: ", dim_names.size());
+  }
+  if (uniform_shape.size() > ndim) {

Review Comment:
   Should it just be equal to `ndim`?



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(this->value_type(), shape, this->permutation(), &strides));
+
+  const auto array = std::static_pointer_cast<FixedSizeListScalar>(data)->value;
+  const auto byte_width = this->value_type()->byte_width();
+  const auto start_position = array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  // Create a slice of the buffer
+  std::shared_ptr<arrow::Buffer> buffer =
+      arrow::SliceBuffer(array->data()->buffers[1], start_position, size * byte_width);
+
+  return Tensor::Make(this->value_type(), buffer, shape, strides, this->dim_names());
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Make(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  if (!permutation.empty() && permutation.size() != ndim) {
+    return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
+                           " Got: ", permutation.size());
+  }
+  if (!dim_names.empty() && dim_names.size() != ndim) {
+    return Status::Invalid("dim_names size must match ndim. Expected: ", ndim,
+                           " Got: ", dim_names.size());
+  }
+  if (uniform_shape.size() > ndim) {
+    return Status::Invalid("uniform_shape size must be less or equal ndim.");
+  }
+  return std::make_shared<VariableShapeTensorType>(value_type, ndim, permutation,
+                                                   dim_names, uniform_shape);
+}
+
+std::shared_ptr<DataType> variable_shape_tensor(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  auto maybe_type = VariableShapeTensorType::Make(value_type, ndim, permutation,
+                                                  dim_names, uniform_shape);

Review Comment:
   Here as well, can take most of these by value and move them.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(this->value_type(), shape, this->permutation(), &strides));
+
+  const auto array = std::static_pointer_cast<FixedSizeListScalar>(data)->value;
+  const auto byte_width = this->value_type()->byte_width();
+  const auto start_position = array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  // Create a slice of the buffer
+  std::shared_ptr<arrow::Buffer> buffer =
+      arrow::SliceBuffer(array->data()->buffers[1], start_position, size * byte_width);
+
+  return Tensor::Make(this->value_type(), buffer, shape, strides, this->dim_names());

Review Comment:
   ```suggestion
     return Tensor::Make(this->value_type(), std::move(buffer), std::move(shape), std::move(strides), this->dim_names());
   ```



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)

Review Comment:
   What if the i-th value is null?



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Numpy arrays needs to be C-contiguous in memory (``obj.flags["C_CONTIGUOUS"]==True``).
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)
+        <pyarrow.lib.VariableShapeTensorArray object at ...>
+        -- is_valid: all not null
+        -- child 0 type: fixed_size_list<item: uint32>[2]
+          [
+            [
+              2,
+              3
+            ],
+            [
+              1,
+              2
+            ]
+          ]
+        -- child 1 type: list<item: float>
+          [
+            [
+              1,
+              2,
+              3,
+              4,
+              5,
+              6
+            ],
+            [
+              7,
+              8
+            ]
+          ]
+        """
+        if not all([o.flags["C_CONTIGUOUS"] for o in obj]):
+            raise ValueError('The data in the numpy arrays need to be in a single, '
+                             'C-style contiguous segment.')

Review Comment:
   Is this a temporary limitation? Ideally we should be able to ingest all kinds of Numpy arrays...



##########
python/pyarrow/lib.pxd:
##########
@@ -453,6 +458,8 @@ cdef class DictionaryArray(Array):
 cdef class ExtensionArray(Array):
     pass
 
+cdef class VariableShapeTensorArray(ExtensionArray):

Review Comment:
   Why isn't there a corresponding definition for FixedShapeTensorArray here?



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Numpy arrays needs to be C-contiguous in memory (``obj.flags["C_CONTIGUOUS"]==True``).
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)
+        <pyarrow.lib.VariableShapeTensorArray object at ...>
+        -- is_valid: all not null
+        -- child 0 type: fixed_size_list<item: uint32>[2]
+          [
+            [
+              2,
+              3
+            ],
+            [
+              1,
+              2
+            ]
+          ]
+        -- child 1 type: list<item: float>
+          [
+            [
+              1,
+              2,
+              3,
+              4,
+              5,
+              6
+            ],
+            [
+              7,
+              8
+            ]
+          ]
+        """
+        if not all([o.flags["C_CONTIGUOUS"] for o in obj]):
+            raise ValueError('The data in the numpy arrays need to be in a single, '
+                             'C-style contiguous segment.')
+        numpy_type = obj[0].dtype
+        ndim = obj[0].ndim
+
+        if not all([o.dtype == numpy_type for o in obj]):
+            raise ValueError('All numpy arrays need to have the same dtype.')
+
+        if not all([o.ndim == ndim for o in obj]):
+            raise ValueError('All numpy arrays need to have the same ndim.')
+
+        arrow_type = from_numpy_dtype(numpy_type)
+        values = array([np.ravel(o, order='C') for o in obj], list_(arrow_type))
+        shapes = array([o.shape for o in obj], list_(uint32(), list_size=ndim))

Review Comment:
   `int32`



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(this->value_type(), shape, this->permutation(), &strides));
+
+  const auto array = std::static_pointer_cast<FixedSizeListScalar>(data)->value;
+  const auto byte_width = this->value_type()->byte_width();
+  const auto start_position = array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  // Create a slice of the buffer
+  std::shared_ptr<arrow::Buffer> buffer =
+      arrow::SliceBuffer(array->data()->buffers[1], start_position, size * byte_width);
+
+  return Tensor::Make(this->value_type(), buffer, shape, strides, this->dim_names());
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Make(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  if (!permutation.empty() && permutation.size() != ndim) {
+    return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
+                           " Got: ", permutation.size());
+  }
+  if (!dim_names.empty() && dim_names.size() != ndim) {
+    return Status::Invalid("dim_names size must match ndim. Expected: ", ndim,
+                           " Got: ", dim_names.size());
+  }
+  if (uniform_shape.size() > ndim) {
+    return Status::Invalid("uniform_shape size must be less or equal ndim.");
+  }
+  return std::make_shared<VariableShapeTensorType>(value_type, ndim, permutation,
+                                                   dim_names, uniform_shape);
+}
+
+std::shared_ptr<DataType> variable_shape_tensor(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  auto maybe_type = VariableShapeTensorType::Make(value_type, ndim, permutation,
+                                                  dim_names, uniform_shape);
+  ARROW_DCHECK_OK(maybe_type.status());

Review Comment:
   I don't think we gain anything by making this check debug-only.
   ```suggestion
     ARROW_CHECK_OK(maybe_type.status());
   ```



##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -2644,6 +2649,35 @@ cdef extern from "arrow/extension_type.h" namespace "arrow":
         shared_ptr[CArray] storage()
 
 
+cdef extern from "arrow/extension/variable_shape_tensor.h" namespace "arrow::extension":
+    cdef cppclass CVariableShapeTensorType \
+            " arrow::extension::VariableShapeTensorType"(CExtensionType):
+
+        CResult[shared_ptr[CTensor]] GetTensor(const shared_ptr[CExtensionScalar]& scalar) const
+
+        @staticmethod
+        CResult[shared_ptr[CDataType]] Make(const shared_ptr[CDataType]& value_type,
+                                            const uint32_t ndim,
+                                            const vector[int64_t]& permutation,
+                                            const vector[c_string]& dim_names,
+                                            const vector[optional[int64_t]]& uniform_shape)
+
+        CResult[shared_ptr[CDataType]] Deserialize(const shared_ptr[CDataType] storage_type,
+                                                   const c_string& serialized_data) const
+
+        c_string Serialize() const

Review Comment:
   These are already defined in `CExtensionType`, no? You shouldn't have to duplicate these definitions here...



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Numpy arrays needs to be C-contiguous in memory (``obj.flags["C_CONTIGUOUS"]==True``).
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)
+        <pyarrow.lib.VariableShapeTensorArray object at ...>
+        -- is_valid: all not null
+        -- child 0 type: fixed_size_list<item: uint32>[2]
+          [
+            [
+              2,
+              3
+            ],
+            [
+              1,
+              2
+            ]
+          ]
+        -- child 1 type: list<item: float>
+          [
+            [
+              1,
+              2,
+              3,
+              4,
+              5,
+              6
+            ],
+            [
+              7,
+              8
+            ]
+          ]
+        """
+        if not all([o.flags["C_CONTIGUOUS"] for o in obj]):
+            raise ValueError('The data in the numpy arrays need to be in a single, '
+                             'C-style contiguous segment.')
+        numpy_type = obj[0].dtype
+        ndim = obj[0].ndim
+
+        if not all([o.dtype == numpy_type for o in obj]):
+            raise ValueError('All numpy arrays need to have the same dtype.')

Review Comment:
   `TypeError` perhaps?



##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1353,6 +1386,70 @@ def test_tensor_class_methods():
         arr.to_numpy_ndarray()
 
 
+@pytest.mark.parametrize("value_type", (np.int8, np.int32, np.int64, np.float64))
+def test_variable_shape_tensor_class_methods(value_type):
+    ndim = 2
+    shape_type = pa.list_(pa.uint32(), ndim)
+    arrow_type = pa.from_numpy_dtype(value_type)
+    tensor_type = pa.variable_shape_tensor(
+        arrow_type,
+        ndim,
+        dim_names=["H", "W"],
+        permutation=[0, 1],
+        uniform_shape=[None, None],
+    )
+    fields = [pa.field("shape", shape_type), pa.field("data", pa.list_(arrow_type))]
+
+    shapes = pa.array([[2, 3], [2, 1]], shape_type)
+    values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(arrow_type))
+    struct_arr = pa.StructArray.from_arrays([shapes, values], fields=fields)
+    arr = pa.ExtensionArray.from_storage(tensor_type, struct_arr)
+    basic_arr = pa.ExtensionArray.from_storage(
+        pa.variable_shape_tensor(arrow_type, ndim), struct_arr
+    )
+
+    storage = pa.array(
+        [([2, 3], [1, 2, 3, 4, 5, 6]), ([2, 1], [7, 8])], type=pa.struct(fields)
+    )
+    assert pa.ExtensionArray.from_storage(tensor_type, storage).equals(arr)
+
+    assert arr.type == tensor_type
+
+    ndarray_list = [
+        np.array([[1, 2, 3], [4, 5, 6]], dtype=value_type),
+        np.array([[7], [8]], dtype=value_type),
+    ]
+    assert all(zip(x == y for x, y in zip(arr.to_numpy_ndarray(), ndarray_list)))
+
+    assert pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list).equals(
+        basic_arr
+    )
+    assert pa.VariableShapeTensorArray.from_numpy_ndarray(
+        arr.to_numpy_ndarray()
+    ).equals(basic_arr)
+
+    assert arr.to_pylist() == [
+        {"data": [1, 2, 3, 4, 5, 6], "shape": [2, 3]},
+        {"data": [7, 8], "shape": [2, 1]},
+    ]
+
+    expected_0 = np.array([[1, 2, 3], [4, 5, 6]], dtype=value_type)
+    expected_1 = np.array([[7], [8]], dtype=value_type)
+
+    np.testing.assert_array_equal(arr[0].to_tensor().to_numpy(), expected_0)
+    np.testing.assert_array_equal(arr[1].to_tensor().to_numpy(), expected_1)
+
+    np.testing.assert_array_equal(arr[0].to_numpy_ndarray(), expected_0)
+    np.testing.assert_array_equal(arr[1].to_numpy_ndarray(), expected_1)
+
+    assert arr[0].to_tensor().equals(
+        pa.Tensor.from_numpy(expected_0, dim_names=["H", "W"]))
+
+    # TODO: due to wrong offset this would return [[1], [2]] instead of [[7], [8]]

Review Comment:
   Does it?



##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,31 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        """
+        return self.to_tensor().to_numpy()
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            CVariableShapeTensorType* c_type = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type).get()
+            shared_ptr[CExtensionScalar] scalar = static_pointer_cast[CExtensionScalar, CScalar](self.wrapped)
+            shared_ptr[CTensor] ctensor
+
+        ctensor = GetResultValue(c_type.GetTensor(scalar))

Review Comment:
   Perhaps release the GIL here.



##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -2664,6 +2698,11 @@ cdef extern from "arrow/extension/fixed_shape_tensor.h" namespace "arrow::extens
         const vector[int64_t] permutation()
         const vector[c_string] dim_names()
 
+cdef extern from "arrow/extension/fixed_shape_tensor.h" namespace "arrow::internal" nogil:
+    cdef CStatus ComputeStrides(const shared_ptr[CDataType]& value_type,

Review Comment:
   Is this used in PyArrow?



##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,114 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+    """
+
+    cdef void init(self, const shared_ptr[CDataType]& type) except *:
+        BaseExtensionType.init(self, type)
+        self.tensor_ext_type = <const CVariableShapeTensorType*> type.get()
+
+    @property
+    def value_type(self):
+        """
+        Data type of an individual tensor.
+        """
+        return pyarrow_wrap_data_type(self.tensor_ext_type.value_type())
+
+    @property
+    def ndim(self):
+        """
+        Number of dimensions of the tensors.
+        """
+        return self.tensor_ext_type.ndim()
+
+    @property
+    def dim_names(self):
+        """
+        Explicit names of the dimensions.
+        """
+        list_of_bytes = self.tensor_ext_type.dim_names()
+        if len(list_of_bytes) != 0:
+            return [frombytes(x) for x in list_of_bytes]
+        else:
+            return None
+
+    @property
+    def permutation(self):
+        """
+        Indices of the dimensions ordering.
+        """
+        indices = self.tensor_ext_type.permutation()
+        if len(indices) != 0:
+            return indices
+        else:
+            return None
+
+    @property
+    def uniform_shape(self):
+        """
+        Shape over dimensions that are guaranteed to be constant.
+        """
+        cdef:
+            vector[optional[int64_t]] c_uniform_shape = self.tensor_ext_type.uniform_shape()
+            length = c_uniform_shape.size()
+
+        if length == 0:
+            return None
+
+        uniform_shape = []
+        for i in range(length):
+            if c_uniform_shape[i].has_value():
+                uniform_shape.append(c_uniform_shape[i].value())
+            else:
+                uniform_shape.append(None)
+
+        return uniform_shape
+
+    def __arrow_ext_serialize__(self):
+        """
+        Serialized representation of metadata to reconstruct the type object.
+        """
+        return self.tensor_ext_type.Serialize()
+
+    @classmethod
+    def __arrow_ext_deserialize__(self, storage_type, serialized):
+        """
+        Return an VariableShapeTensor type instance from the storage type and serialized
+        metadata.
+        """
+        return self.tensor_ext_type.Deserialize(storage_type, serialized)
+
+    def __arrow_ext_class__(self):
+        return VariableShapeTensorArray
+
+    def __reduce__(self):
+        return variable_shape_tensor, (self.value_type, self.ndim,
+                                       self.dim_names, self.permutation,
+                                       self.uniform_shape)

Review Comment:
   Same here: when would this be called?



##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1472,10 +1618,24 @@ def test_tensor_type_is_picklable(pickle_module):
         'fixed_shape_tensor[value_type=int64, shape=[2,2,3], dim_names=[C,H,W]]'
     )
 ])
-def test_tensor_type_str(tensor_type, text):
+def test_tensor_type_str(tensor_type, text, pickle_module):

Review Comment:
   Can you write a separate test for pickling instead of adding to this function?



##########
python/pyarrow/types.pxi:
##########
@@ -4948,6 +5056,121 @@ def fixed_shape_tensor(DataType value_type, shape, dim_names=None, permutation=N
     return out
 
 
+def variable_shape_tensor(DataType value_type, ndim, dim_names=None, permutation=None,
+                          uniform_shape=None):
+    """
+    Create instance of variable shape tensor extension type with number of
+    dimensions and optional names of tensor dimensions and indices of the
+    desired logical ordering of dimensions.
+
+    Parameters
+    ----------
+    value_type : DataType
+        Data type of individual tensor elements.
+    ndim : integer
+        The number of dimensions of the contained tensors.
+    dim_names : tuple or list of strings, default None
+        Explicit names to tensor dimensions.
+    permutation : tuple or list integers, default None
+        Indices of the desired ordering of the original dimensions.
+        The indices contain a permutation of the values ``[0, 1, .., N-1]`` where
+        N is the number of dimensions. The permutation indicates which dimension
+        of the logical layout corresponds to which dimension of the physical tensor.
+        For more information on this parameter see
+        :ref:`fixed_shape_tensor_extension`.
+    uniform_shape : tuple or list of integers, default None
+        Shape of dimensions that are guaranteed to stay constant over all tensors
+        in the array if all their non-dimensions sizes were replaced by None.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+    >>> tensor_type
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Inspect the data type:
+
+    >>> tensor_type.value_type
+    DataType(int32)
+    >>> tensor_type.ndim
+    2
+
+    Create a table with variable shape tensor extension array:
+
+    >>> fields = [pa.field("shape", pa.list_(pa.uint32(), 2)), pa.field("data", pa.list_(pa.int32()))]
+    >>> storage = pa.array([([2, 3], [1, 2, 3, 4, 5, 6]), ([1, 2], [7, 8])], type=pa.struct(fields))
+    >>> tensor = pa.ExtensionArray.from_storage(tensor_type, storage)
+    >>> pa.table([tensor], names=["tensor_array"])
+    pyarrow.Table
+    tensor_array: extension<arrow.variable_shape_tensor>
+    ----
+    tensor_array: [  -- is_valid: all not null
+      -- child 0 type: fixed_size_list<item: uint32>[2]
+    [[2,3],[1,2]]
+      -- child 1 type: list<item: int32>
+    [[1,2,3,4,5,6],[7,8]]]
+
+    Create an instance of variable shape tensor extension type with names
+    of tensor dimensions:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     dim_names=['C', 'H', 'W'])
+    >>> tensor_type.dim_names
+    ['C', 'H', 'W']
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+
+    Returns
+    -------
+    type : VariableShapeTensorType
+    """
+
+    cdef:
+        uint32_t c_ndim
+        vector[int64_t] c_permutation
+        vector[c_string] c_dim_names
+        vector[optional[int64_t]] c_uniform_shape
+        shared_ptr[CDataType] c_tensor_ext_type
+
+    assert value_type is not None
+    assert ndim is not None
+
+    c_ndim = ndim
+
+    if permutation is not None:
+        for i in permutation:
+            c_permutation.push_back(i)
+
+    if dim_names is not None:
+        for x in dim_names:
+            c_dim_names.push_back(tobytes(x))
+
+    if uniform_shape is not None:
+        for x in uniform_shape:
+            if x is None:
+                c_uniform_shape.push_back(<optional[int64_t]>nullopt)
+            else:
+                c_uniform_shape.push_back(<optional[int64_t]>(<int64_t>x))
+
+    cdef VariableShapeTensorType out = VariableShapeTensorType.__new__(VariableShapeTensorType)
+
+    c_tensor_ext_type = GetResultValue(CVariableShapeTensorType.Make(
+        value_type.sp_type, c_ndim, c_permutation, c_dim_names, c_uniform_shape))

Review Comment:
   `with nogil` perhaps?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409321555


##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,
+                          const std::vector<int64_t>& permutation = {},
+                          const std::vector<std::string>& dim_names = {},
+                          const std::vector<std::optional<int64_t>>& uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(uint32(), ndim)),
+                               ::arrow::field("data", list(value_type))})),
+        value_type_(value_type),
+        permutation_(permutation),
+        dim_names_(dim_names),
+        uniform_shape_(uniform_shape) {}
+
+  std::string extension_name() const override { return "arrow.variable_shape_tensor"; }
+
+  /// Number of dimensions of tensor elements
+  uint32_t ndim() const {
+    std::shared_ptr<DataType> storage_type = this->storage_type()->field(0)->type();
+    return std::static_pointer_cast<FixedSizeListType>(storage_type)->list_size();
+  }
+
+  /// Value type of tensor elements
+  const std::shared_ptr<DataType> value_type() const { return value_type_; }
+
+  /// Permutation mapping from logical to physical memory layout of tensor elements
+  const std::vector<int64_t>& permutation() const { return permutation_; }
+
+  /// Dimension names of tensor elements. Dimensions are ordered physically.
+  const std::vector<std::string>& dim_names() const { return dim_names_; }
+
+  /// Shape of uniform dimensions.
+  const std::vector<std::optional<int64_t>>& uniform_shape() const {
+    return uniform_shape_;
+  }
+
+  bool ExtensionEquals(const ExtensionType& other) const override;
+
+  std::string Serialize() const override;
+
+  Result<std::shared_ptr<DataType>> Deserialize(
+      std::shared_ptr<DataType> storage_type,
+      const std::string& serialized_data) const override;
+
+  /// Create a VariableShapeTensorArray from ArrayData
+  std::shared_ptr<Array> MakeArray(std::shared_ptr<ArrayData> data) const override;
+
+  /// Convert an ExtensionScalar to a Tensor
+  Result<std::shared_ptr<Tensor>> GetTensor(
+      const std::shared_ptr<ExtensionScalar>&) const;
+
+  /// \brief Create a VariableShapeTensorType instance
+  static Result<std::shared_ptr<DataType>> Make(
+      const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,

Review Comment:
   Changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1410027049


##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,114 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+    """
+
+    cdef void init(self, const shared_ptr[CDataType]& type) except *:
+        BaseExtensionType.init(self, type)
+        self.tensor_ext_type = <const CVariableShapeTensorType*> type.get()
+
+    @property
+    def value_type(self):
+        """
+        Data type of an individual tensor.
+        """
+        return pyarrow_wrap_data_type(self.tensor_ext_type.value_type())
+
+    @property
+    def ndim(self):
+        """
+        Number of dimensions of the tensors.
+        """
+        return self.tensor_ext_type.ndim()
+
+    @property
+    def dim_names(self):
+        """
+        Explicit names of the dimensions.
+        """
+        list_of_bytes = self.tensor_ext_type.dim_names()
+        if len(list_of_bytes) != 0:
+            return [frombytes(x) for x in list_of_bytes]
+        else:
+            return None
+
+    @property
+    def permutation(self):
+        """
+        Indices of the dimensions ordering.
+        """
+        indices = self.tensor_ext_type.permutation()
+        if len(indices) != 0:
+            return indices
+        else:
+            return None
+
+    @property
+    def uniform_shape(self):
+        """
+        Shape over dimensions that are guaranteed to be constant.
+        """
+        cdef:
+            vector[optional[int64_t]] c_uniform_shape = self.tensor_ext_type.uniform_shape()
+            length = c_uniform_shape.size()
+
+        if length == 0:
+            return None
+
+        uniform_shape = []
+        for i in range(length):
+            if c_uniform_shape[i].has_value():
+                uniform_shape.append(c_uniform_shape[i].value())
+            else:
+                uniform_shape.append(None)
+
+        return uniform_shape
+
+    def __arrow_ext_serialize__(self):
+        """
+        Serialized representation of metadata to reconstruct the type object.
+        """
+        return self.tensor_ext_type.Serialize()
+
+    @classmethod
+    def __arrow_ext_deserialize__(self, storage_type, serialized):
+        """
+        Return an VariableShapeTensor type instance from the storage type and serialized
+        metadata.
+        """
+        return self.tensor_ext_type.Deserialize(storage_type, serialized)
+
+    def __arrow_ext_class__(self):
+        return VariableShapeTensorArray
+
+    def __reduce__(self):
+        return variable_shape_tensor, (self.value_type, self.ndim,
+                                       self.dim_names, self.permutation,
+                                       self.uniform_shape)

Review Comment:
   When pickling apparently. See: https://github.com/apache/arrow/pull/35933/files



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409998926


##########
python/pyarrow/lib.pxd:
##########
@@ -453,6 +458,8 @@ cdef class DictionaryArray(Array):
 cdef class ExtensionArray(Array):
     pass
 
+cdef class VariableShapeTensorArray(ExtensionArray):

Review Comment:
   I forget why this was needed. Removing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409339734


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(this->value_type(), shape, this->permutation(), &strides));
+
+  const auto array = std::static_pointer_cast<FixedSizeListScalar>(data)->value;
+  const auto byte_width = this->value_type()->byte_width();
+  const auto start_position = array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  // Create a slice of the buffer
+  std::shared_ptr<arrow::Buffer> buffer =
+      arrow::SliceBuffer(array->data()->buffers[1], start_position, size * byte_width);
+
+  return Tensor::Make(this->value_type(), buffer, shape, strides, this->dim_names());
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Make(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  if (!permutation.empty() && permutation.size() != ndim) {
+    return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
+                           " Got: ", permutation.size());
+  }
+  if (!dim_names.empty() && dim_names.size() != ndim) {
+    return Status::Invalid("dim_names size must match ndim. Expected: ", ndim,
+                           " Got: ", dim_names.size());
+  }
+  if (uniform_shape.size() > ndim) {
+    return Status::Invalid("uniform_shape size must be less or equal ndim.");
+  }
+  return std::make_shared<VariableShapeTensorType>(value_type, ndim, permutation,
+                                                   dim_names, uniform_shape);
+}
+
+std::shared_ptr<DataType> variable_shape_tensor(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  auto maybe_type = VariableShapeTensorType::Make(value_type, ndim, permutation,
+                                                  dim_names, uniform_shape);

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1977020572

   @pitrou Sure, I've split Python out into #40354.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1414513505


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):

Review Comment:
   Oh, I assumed array of arrays was not techincally possible. Will change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542163349


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+    ARROW_ASSIGN_OR_RAISE(const auto size, shape_array->GetScalar(j));
+    auto size_value = internal::checked_pointer_cast<Int32Scalar>(size)->value;
+    if (size_value < 0) {
+      return Status::Invalid("shape must have non-negative values");
+    }
+    shape.push_back(std::move(size_value));
+  }
+  internal::Permute<int64_t>(permutation, &shape);
+
+  std::vector<std::string> dim_names = ext_type->dim_names();
+  if (!dim_names.empty()) {
+    internal::Permute<std::string>(permutation, &dim_names);
+  }
+
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(internal::ComputeStrides(value_type, shape, permutation, &strides));
+
+  const auto byte_width = value_type->byte_width();
+  const auto start_position = data_array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  ARROW_ASSIGN_OR_RAISE(
+      const auto buffer,
+      SliceBufferSafe(data_array->data()->buffers[1], start_position, size * byte_width));

Review Comment:
   Added a check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1544948342


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& permutation,
+                             std::vector<int64_t>* strides) {
+  const auto fixed_width_type =
+      internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);
+  }
+  const int byte_width = value_type->byte_width();
+
+  int64_t remaining = 0;
+  if (!shape.empty() && shape.front() > 0) {
+    remaining = byte_width;
+    for (auto i : permutation) {
+      if (i > 0) {
+        if (internal::MultiplyWithOverflow(remaining, shape[i], &remaining)) {
+          return Status::Invalid(
+              "Strides computed from shape would not fit in 64-bit integer");
+        }
+      }
+    }
+  }
+
+  if (remaining == 0) {
+    strides->assign(shape.size(), byte_width);
+    return Status::OK();
+  }
+
+  strides->push_back(remaining);
+  for (auto i : permutation) {
+    if (i > 0) {
+      remaining /= shape[i];

Review Comment:
   Spec states:
   
     * **permutation**  = indices of the desired ordering of the
       original dimensions, defined as an array.
   
       The indices contain a permutation of the values [0, 1, .., N-1] where
       N is the number of dimensions. The permutation indicates which
       dimension of the logical layout corresponds to which dimension of the
       physical tensor (the i-th dimension of the logical view corresponds
       to the dimension with number ``permutations[i]`` of the physical tensor).
   
       Permutation can be useful in case the logical order of
       the tensor is a permutation of the physical order (row-major).
   
       When logical and physical layout are equal, the permutation will always
       be ([0, 1, .., N-1]) and can therefore be left out.
   
   I understand this as that the logical layout will have to have different strides than the physical layout not just a permutation of physical strides.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1544935676


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& permutation,
+                             std::vector<int64_t>* strides) {
+  const auto fixed_width_type =
+      internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);
+  }
+  const int byte_width = value_type->byte_width();
+
+  int64_t remaining = 0;
+  if (!shape.empty() && shape.front() > 0) {
+    remaining = byte_width;
+    for (auto i : permutation) {
+      if (i > 0) {
+        if (internal::MultiplyWithOverflow(remaining, shape[i], &remaining)) {
+          return Status::Invalid(
+              "Strides computed from shape would not fit in 64-bit integer");
+        }
+      }
+    }
+  }
+
+  if (remaining == 0) {
+    strides->assign(shape.size(), byte_width);
+    return Status::OK();
+  }
+
+  strides->push_back(remaining);
+  for (auto i : permutation) {
+    if (i > 0) {
+      remaining /= shape[i];

Review Comment:
   How about this: https://github.com/apache/arrow/pull/38008/commits/8e6e99093f493bbe517cdec102d2dca01d2fcac5
   Somewhat disturbi
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2835145568


##########
cpp/src/arrow/compute/kernels/CMakeLists.txt:
##########
@@ -22,7 +22,8 @@ arrow_install_all_headers("arrow/compute/kernels")
 
 # Define arrow_compute_kernels_testing object library for common test files
 if(ARROW_TESTING)
-  add_library(arrow_compute_kernels_testing OBJECT test_util_internal.cc)
+  add_library(arrow_compute_kernels_testing OBJECT
+              test_util_internal.cc ../../extension/tensor_extension_array_test.cc)

Review Comment:
   Removing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2848507812


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -20,25 +20,27 @@
 #include <cstdint>
 #include <vector>
 
-#include "arrow/status.h"
-#include "arrow/util/print_internal.h"
+#include <span>

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3953538261

   > By the way, do you want to expose it in Python soon after? That would allow validating against NumPy.
   
   Yes, I'm jumping on https://github.com/apache/arrow/pull/40354 right after we merge here :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2848314201


##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+#include <string>
+#include <vector>
+
+#include "arrow/extension_type.h"
+
+namespace arrow::extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type, int32_t ndim,
+                          std::vector<int64_t> permutation = {},
+                          std::vector<std::string> dim_names = {},
+                          std::vector<std::optional<int64_t>> uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("data", list(value_type)),
+                               ::arrow::field("shape", fixed_size_list(int32(), ndim))})),
+        value_type_(value_type),
+        ndim_(ndim),
+        permutation_(std::move(permutation)),
+        dim_names_(std::move(dim_names)),
+        uniform_shape_(std::move(uniform_shape)) {}
+
+  std::string extension_name() const override { return "arrow.variable_shape_tensor"; }
+  std::string ToString(bool show_metadata = false) const override;
+
+  /// Number of dimensions of tensor elements
+  int32_t ndim() const { return ndim_; }
+
+  /// Value type of tensor elements
+  const std::shared_ptr<DataType>& value_type() const { return value_type_; }
+
+  /// Permutation mapping from logical to physical memory layout of tensor elements
+  const std::vector<int64_t>& permutation() const { return permutation_; }
+
+  /// Dimension names of tensor elements. Dimensions are ordered physically.
+  const std::vector<std::string>& dim_names() const { return dim_names_; }
+
+  /// Shape of uniform dimensions.
+  const std::vector<std::optional<int64_t>>& uniform_shape() const {
+    return uniform_shape_;
+  }
+
+  bool ExtensionEquals(const ExtensionType& other) const override;
+
+  std::string Serialize() const override;
+
+  Result<std::shared_ptr<DataType>> Deserialize(
+      std::shared_ptr<DataType> storage_type,
+      const std::string& serialized_data) const override;
+
+  /// Create a VariableShapeTensorArray from ArrayData
+  std::shared_ptr<Array> MakeArray(std::shared_ptr<ArrayData> data) const override;
+
+  /// \brief Convert an ExtensionScalar to a Tensor
+  ///
+  /// This method will return a Tensor from ExtensionScalar with strides derived
+  /// from shape and permutation stored. Shape and dim_names will be permuted
+  /// according to permutation stored in the VariableShapeTensorType.
+  static Result<std::shared_ptr<Tensor>> MakeTensor(
+      const std::shared_ptr<ExtensionScalar>&);
+
+  /// \brief Create a VariableShapeTensorType instance
+  static Result<std::shared_ptr<DataType>> Make(
+      const std::shared_ptr<DataType>& value_type, int32_t ndim,
+      const std::vector<int64_t>& permutation = {},
+      const std::vector<std::string>& dim_names = {},
+      const std::vector<std::optional<int64_t>>& uniform_shape = {});
+
+ private:
+  std::shared_ptr<DataType> value_type_;
+  int32_t ndim_;
+  std::vector<int64_t> permutation_;
+  std::vector<std::string> dim_names_;
+  std::vector<std::optional<int64_t>> uniform_shape_;
+};
+
+/// \brief Return a VariableShapeTensorType instance.
+ARROW_EXPORT std::shared_ptr<DataType> variable_shape_tensor(
+    const std::shared_ptr<DataType>& value_type, int32_t ndim,
+    const std::vector<int64_t>& permutation = {},
+    const std::vector<std::string>& dim_names = {},
+    const std::vector<std::optional<int64_t>>& uniform_shape = {});

Review Comment:
   Same here



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+#include <string>
+#include <vector>
+
+#include "arrow/extension_type.h"
+
+namespace arrow::extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type, int32_t ndim,
+                          std::vector<int64_t> permutation = {},
+                          std::vector<std::string> dim_names = {},
+                          std::vector<std::optional<int64_t>> uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("data", list(value_type)),
+                               ::arrow::field("shape", fixed_size_list(int32(), ndim))})),
+        value_type_(value_type),
+        ndim_(ndim),
+        permutation_(std::move(permutation)),
+        dim_names_(std::move(dim_names)),
+        uniform_shape_(std::move(uniform_shape)) {}
+
+  std::string extension_name() const override { return "arrow.variable_shape_tensor"; }
+  std::string ToString(bool show_metadata = false) const override;
+
+  /// Number of dimensions of tensor elements
+  int32_t ndim() const { return ndim_; }
+
+  /// Value type of tensor elements
+  const std::shared_ptr<DataType>& value_type() const { return value_type_; }
+
+  /// Permutation mapping from logical to physical memory layout of tensor elements
+  const std::vector<int64_t>& permutation() const { return permutation_; }
+
+  /// Dimension names of tensor elements. Dimensions are ordered physically.
+  const std::vector<std::string>& dim_names() const { return dim_names_; }
+
+  /// Shape of uniform dimensions.
+  const std::vector<std::optional<int64_t>>& uniform_shape() const {
+    return uniform_shape_;
+  }
+
+  bool ExtensionEquals(const ExtensionType& other) const override;
+
+  std::string Serialize() const override;
+
+  Result<std::shared_ptr<DataType>> Deserialize(
+      std::shared_ptr<DataType> storage_type,
+      const std::string& serialized_data) const override;
+
+  /// Create a VariableShapeTensorArray from ArrayData
+  std::shared_ptr<Array> MakeArray(std::shared_ptr<ArrayData> data) const override;
+
+  /// \brief Convert an ExtensionScalar to a Tensor
+  ///
+  /// This method will return a Tensor from ExtensionScalar with strides derived
+  /// from shape and permutation stored. Shape and dim_names will be permuted
+  /// according to permutation stored in the VariableShapeTensorType.
+  static Result<std::shared_ptr<Tensor>> MakeTensor(
+      const std::shared_ptr<ExtensionScalar>&);
+
+  /// \brief Create a VariableShapeTensorType instance
+  static Result<std::shared_ptr<DataType>> Make(
+      const std::shared_ptr<DataType>& value_type, int32_t ndim,
+      const std::vector<int64_t>& permutation = {},
+      const std::vector<std::string>& dim_names = {},
+      const std::vector<std::optional<int64_t>>& uniform_shape = {});

Review Comment:
   Why not pass those vectors by value just like in the constructor?



##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -20,25 +20,27 @@
 #include <cstdint>
 #include <vector>
 
-#include "arrow/status.h"
-#include "arrow/util/print_internal.h"
+#include <span>

Review Comment:
   Please move these with the other stdlib includes above?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2835316124


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -20,25 +20,17 @@
 #include <cstdint>
 #include <vector>
 
-#include "arrow/status.h"
-#include "arrow/util/print_internal.h"
+#include "arrow/array/array_nested.h"
 
 namespace arrow::internal {
 
-inline Status IsPermutationValid(const std::vector<int64_t>& permutation) {
-  const auto size = static_cast<int64_t>(permutation.size());
-  std::vector<uint8_t> dim_seen(size, 0);
+ARROW_EXPORT
+Status IsPermutationValid(const std::vector<int64_t>& permutation);

Review Comment:
   Done. I hope `std::span` won't cause CRAN issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3953630980

   Oh wait, rebases are cheap now https://github.com/apache/arrow/pull/40354


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "pimdh (via GitHub)" <gi...@apache.org>.
pimdh commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-2792225268

   Could you please consider using LargeList for the values, or at least providing an option for that? Tensor arrays with >4.2 billion items is a use-case for us. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542164240


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+    ARROW_ASSIGN_OR_RAISE(const auto size, shape_array->GetScalar(j));
+    auto size_value = internal::checked_pointer_cast<Int32Scalar>(size)->value;
+    if (size_value < 0) {
+      return Status::Invalid("shape must have non-negative values");
+    }
+    shape.push_back(std::move(size_value));
+  }
+  internal::Permute<int64_t>(permutation, &shape);
+
+  std::vector<std::string> dim_names = ext_type->dim_names();
+  if (!dim_names.empty()) {
+    internal::Permute<std::string>(permutation, &dim_names);
+  }
+
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(internal::ComputeStrides(value_type, shape, permutation, &strides));

Review Comment:
   Switched to `ARROW_ASSIGN_OR_RAISE`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1423729406


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,160 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.float64(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.int32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.float64()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: int32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: double>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        If permutation is non-trivial a strided numpy arrays are produced.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)
+        <pyarrow.lib.VariableShapeTensorArray object at ...>
+        -- is_valid: all not null
+        -- child 0 type: fixed_size_list<item: int32>[2]
+          [
+            [
+              2,
+              3
+            ],
+            [
+              1,
+              2
+            ]
+          ]
+        -- child 1 type: list<item: float>
+          [
+            [
+              1,
+              2,
+              3,
+              4,
+              5,
+              6
+            ],
+            [
+              7,
+              8
+            ]
+          ]
+        """
+        assert isinstance(obj, list), 'obj must be a list of numpy arrays'
+        numpy_type = obj[0].dtype
+        ndim = obj[0].ndim
+        bw = numpy_type.itemsize
+
+        permutation = (-np.array(obj[0].strides)).argsort()
+
+        if not all([o.dtype == numpy_type for o in obj]):
+            raise TypeError('All numpy arrays need to have the same dtype.')
+
+        if not all([o.ndim == ndim for o in obj]):
+            raise ValueError('All numpy arrays need to have the same ndim.')
+
+        if not all([np.array_equal((-np.array(o.strides)).argsort(), permutation) for o in obj]):
+            raise ValueError('All numpy arrays need to have the same permutation.')
+
+        arrow_type=from_numpy_dtype(numpy_type)
+        values=array([np.lib.stride_tricks.as_strided(
+            o, shape=(np.prod(o.shape),), strides=(bw,)) for o in obj], list_(arrow_type))
+        shapes=array([o.shape for o in obj], list_(int32(), list_size=ndim))
+        struct_arr=StructArray.from_arrays([shapes, values], names=["shape", "data"])

Review Comment:
   Can you apply some PEP8 formatting here?



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,160 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.float64(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.int32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.float64()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: int32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: double>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        If permutation is non-trivial a strided numpy arrays are produced.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)

Review Comment:
   ```suggestion
           obj : list of numpy.ndarray
   ```



##########
python/pyarrow/types.pxi:
##########
@@ -4948,6 +5028,122 @@ def fixed_shape_tensor(DataType value_type, shape, dim_names=None, permutation=N
     return out
 
 
+def variable_shape_tensor(DataType value_type, ndim, dim_names=None, permutation=None,
+                          uniform_shape=None):
+    """
+    Create instance of variable shape tensor extension type with number of
+    dimensions and optional names of tensor dimensions and indices of the
+    desired logical ordering of dimensions.
+
+    Parameters
+    ----------
+    value_type : DataType
+        Data type of individual tensor elements.
+    ndim : integer
+        The number of dimensions of the contained tensors.
+    dim_names : tuple or list of strings, default None
+        Explicit names to tensor dimensions.
+    permutation : tuple or list integers, default None
+        Indices of the desired ordering of the original dimensions.
+        The indices contain a permutation of the values ``[0, 1, .., N-1]`` where
+        N is the number of dimensions. The permutation indicates which dimension
+        of the logical layout corresponds to which dimension of the physical tensor.
+        For more information on this parameter see
+        :ref:`fixed_shape_tensor_extension`.
+    uniform_shape : tuple or list of integers, default None
+        Shape of dimensions that are guaranteed to stay constant over all tensors
+        in the array if all their non-dimensions sizes were replaced by None.

Review Comment:
   ```suggestion
           in the array if all their non-uniform sizes were replaced by None.
   ```



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)

Review Comment:
   > > What if the i-th value is null?
   >
   > It returns a tensor with all dimension sizes == 0. Added a test.
   
   Should it return None instead on the Python side? (an dim=0 tensor could in theory be an actual value)



##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1317,6 +1317,54 @@ def test_tensor_type():
     assert tensor_type.dim_names == ['C', 'H', 'W']
     assert tensor_type.permutation is None
 
+    tensor_type = pa.variable_shape_tensor(pa.int8(), 2)

Review Comment:
   Can you make this its own test (and rename the one above to be specific about fixed shape tensor)



##########
python/pyarrow/types.pxi:
##########
@@ -4948,6 +5028,122 @@ def fixed_shape_tensor(DataType value_type, shape, dim_names=None, permutation=N
     return out
 
 
+def variable_shape_tensor(DataType value_type, ndim, dim_names=None, permutation=None,
+                          uniform_shape=None):
+    """
+    Create instance of variable shape tensor extension type with number of
+    dimensions and optional names of tensor dimensions and indices of the
+    desired logical ordering of dimensions.
+
+    Parameters
+    ----------
+    value_type : DataType
+        Data type of individual tensor elements.
+    ndim : integer
+        The number of dimensions of the contained tensors.
+    dim_names : tuple or list of strings, default None
+        Explicit names to tensor dimensions.
+    permutation : tuple or list integers, default None
+        Indices of the desired ordering of the original dimensions.
+        The indices contain a permutation of the values ``[0, 1, .., N-1]`` where
+        N is the number of dimensions. The permutation indicates which dimension
+        of the logical layout corresponds to which dimension of the physical tensor.
+        For more information on this parameter see
+        :ref:`fixed_shape_tensor_extension`.
+    uniform_shape : tuple or list of integers, default None
+        Shape of dimensions that are guaranteed to stay constant over all tensors
+        in the array if all their non-dimensions sizes were replaced by None.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+    >>> tensor_type
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Inspect the data type:
+
+    >>> tensor_type.value_type
+    DataType(int32)
+    >>> tensor_type.ndim
+    2
+
+    Create a table with variable shape tensor extension array:
+
+    >>> fields = [pa.field("shape", pa.list_(pa.int32(), 2)), pa.field("data", pa.list_(pa.int32()))]
+    >>> storage = pa.array([([2, 3], [1, 2, 3, 4, 5, 6]), ([1, 2], [7, 8])], type=pa.struct(fields))
+    >>> tensor = pa.ExtensionArray.from_storage(tensor_type, storage)
+    >>> pa.table([tensor], names=["tensor_array"])
+    pyarrow.Table
+    tensor_array: extension<arrow.variable_shape_tensor>
+    ----
+    tensor_array: [  -- is_valid: all not null
+      -- child 0 type: fixed_size_list<item: int32>[2]
+    [[2,3],[1,2]]
+      -- child 1 type: list<item: int32>
+    [[1,2,3,4,5,6],[7,8]]]
+
+    Create an instance of variable shape tensor extension type with names
+    of tensor dimensions:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     dim_names=['C', 'H', 'W'])

Review Comment:
   ```suggestion
       ...                                        dim_names=['C', 'H', 'W'])
   ```



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,160 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.float64(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.int32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.float64()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: int32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: double>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        If permutation is non-trivial a strided numpy arrays are produced.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)

Review Comment:
   Can you maybe also show the type of the resulting array? (which shows the inferred dimension)



##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,100 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)

Review Comment:
   Can you add a repr similar as we have for FixedShapeTensorType (that includes the parametrization)



##########
python/pyarrow/types.pxi:
##########
@@ -4948,6 +5028,122 @@ def fixed_shape_tensor(DataType value_type, shape, dim_names=None, permutation=N
     return out
 
 
+def variable_shape_tensor(DataType value_type, ndim, dim_names=None, permutation=None,
+                          uniform_shape=None):
+    """
+    Create instance of variable shape tensor extension type with number of
+    dimensions and optional names of tensor dimensions and indices of the
+    desired logical ordering of dimensions.
+
+    Parameters
+    ----------
+    value_type : DataType
+        Data type of individual tensor elements.
+    ndim : integer
+        The number of dimensions of the contained tensors.
+    dim_names : tuple or list of strings, default None
+        Explicit names to tensor dimensions.
+    permutation : tuple or list integers, default None
+        Indices of the desired ordering of the original dimensions.
+        The indices contain a permutation of the values ``[0, 1, .., N-1]`` where
+        N is the number of dimensions. The permutation indicates which dimension
+        of the logical layout corresponds to which dimension of the physical tensor.
+        For more information on this parameter see
+        :ref:`fixed_shape_tensor_extension`.

Review Comment:
   Update link to variable shape tensor



##########
python/pyarrow/types.pxi:
##########
@@ -4948,6 +5028,122 @@ def fixed_shape_tensor(DataType value_type, shape, dim_names=None, permutation=N
     return out
 
 
+def variable_shape_tensor(DataType value_type, ndim, dim_names=None, permutation=None,
+                          uniform_shape=None):
+    """
+    Create instance of variable shape tensor extension type with number of
+    dimensions and optional names of tensor dimensions and indices of the
+    desired logical ordering of dimensions.
+
+    Parameters
+    ----------
+    value_type : DataType
+        Data type of individual tensor elements.
+    ndim : integer
+        The number of dimensions of the contained tensors.
+    dim_names : tuple or list of strings, default None
+        Explicit names to tensor dimensions.
+    permutation : tuple or list integers, default None
+        Indices of the desired ordering of the original dimensions.
+        The indices contain a permutation of the values ``[0, 1, .., N-1]`` where
+        N is the number of dimensions. The permutation indicates which dimension
+        of the logical layout corresponds to which dimension of the physical tensor.
+        For more information on this parameter see
+        :ref:`fixed_shape_tensor_extension`.
+    uniform_shape : tuple or list of integers, default None
+        Shape of dimensions that are guaranteed to stay constant over all tensors
+        in the array if all their non-dimensions sizes were replaced by None.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+    >>> tensor_type
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Inspect the data type:
+
+    >>> tensor_type.value_type
+    DataType(int32)
+    >>> tensor_type.ndim
+    2
+
+    Create a table with variable shape tensor extension array:
+
+    >>> fields = [pa.field("shape", pa.list_(pa.int32(), 2)), pa.field("data", pa.list_(pa.int32()))]
+    >>> storage = pa.array([([2, 3], [1, 2, 3, 4, 5, 6]), ([1, 2], [7, 8])], type=pa.struct(fields))
+    >>> tensor = pa.ExtensionArray.from_storage(tensor_type, storage)
+    >>> pa.table([tensor], names=["tensor_array"])
+    pyarrow.Table
+    tensor_array: extension<arrow.variable_shape_tensor>
+    ----
+    tensor_array: [  -- is_valid: all not null
+      -- child 0 type: fixed_size_list<item: int32>[2]
+    [[2,3],[1,2]]
+      -- child 1 type: list<item: int32>
+    [[1,2,3,4,5,6],[7,8]]]
+
+    Create an instance of variable shape tensor extension type with names
+    of tensor dimensions:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     dim_names=['C', 'H', 'W'])
+    >>> tensor_type.dim_names
+    ['C', 'H', 'W']
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])

Review Comment:
   ```suggestion
       ...                                        permutation=[0, 2, 1])
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406523487


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   This does add a `GetTensor(i)` method, but it can't be called "from within" Python's `ExtensionScalar` as far as I know.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409359468


##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1317,6 +1317,39 @@ def test_tensor_type():
     assert tensor_type.dim_names == ['C', 'H', 'W']
     assert tensor_type.permutation is None
 
+    tensor_type = pa.variable_shape_tensor(pa.int8(), 2)
+    expected_storage_type = pa.struct([
+        pa.field("shape", pa.list_(pa.uint32(), 2)),
+        pa.field("data", pa.list_(pa.int8()))
+    ])
+    assert tensor_type.extension_name == "arrow.variable_shape_tensor"
+    assert tensor_type.storage_type == expected_storage_type
+    assert tensor_type.ndim == 2
+    assert tensor_type.dim_names is None
+    assert tensor_type.permutation is None
+
+    tensor_type = pa.variable_shape_tensor(pa.int64(), 3, dim_names=['C', 'H', 'W'])
+    expected_storage_type = pa.struct([
+        pa.field("shape", pa.list_(pa.uint32(), 3)),
+        pa.field("data", pa.list_(pa.int64()))
+    ])
+    assert tensor_type.extension_name == "arrow.variable_shape_tensor"
+    assert tensor_type.storage_type == expected_storage_type
+    assert tensor_type.ndim == 3
+    assert tensor_type.dim_names == ['C', 'H', 'W']
+    assert tensor_type.permutation is None
+
+    tensor_type = pa.variable_shape_tensor(pa.bool_(), 2, permutation=[1, 0])
+    expected_storage_type = pa.struct([
+        pa.field("shape", pa.list_(pa.uint32(), 2)),
+        pa.field("data", pa.list_(pa.bool_()))
+    ])
+    assert tensor_type.extension_name == "arrow.variable_shape_tensor"
+    assert tensor_type.storage_type == expected_storage_type
+    assert tensor_type.ndim == 2
+    assert tensor_type.dim_names is None
+    assert tensor_type.permutation == [1, 0]

Review Comment:
   Added a test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409315452


##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,
+                          const std::vector<int64_t>& permutation = {},
+                          const std::vector<std::string>& dim_names = {},
+                          const std::vector<std::optional<int64_t>>& uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(uint32(), ndim)),
+                               ::arrow::field("data", list(value_type))})),
+        value_type_(value_type),
+        permutation_(permutation),
+        dim_names_(dim_names),
+        uniform_shape_(uniform_shape) {}
+
+  std::string extension_name() const override { return "arrow.variable_shape_tensor"; }
+
+  /// Number of dimensions of tensor elements
+  uint32_t ndim() const {
+    std::shared_ptr<DataType> storage_type = this->storage_type()->field(0)->type();
+    return std::static_pointer_cast<FixedSizeListType>(storage_type)->list_size();

Review Comment:
   Changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409313923


##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,

Review Comment:
   Indeed, I missed this after the discussion on the spec PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409505068


##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1353,6 +1386,70 @@ def test_tensor_class_methods():
         arr.to_numpy_ndarray()
 
 
+@pytest.mark.parametrize("value_type", (np.int8, np.int32, np.int64, np.float64))
+def test_variable_shape_tensor_class_methods(value_type):
+    ndim = 2
+    shape_type = pa.list_(pa.uint32(), ndim)
+    arrow_type = pa.from_numpy_dtype(value_type)
+    tensor_type = pa.variable_shape_tensor(
+        arrow_type,
+        ndim,
+        dim_names=["H", "W"],
+        permutation=[0, 1],
+        uniform_shape=[None, None],
+    )
+    fields = [pa.field("shape", shape_type), pa.field("data", pa.list_(arrow_type))]
+
+    shapes = pa.array([[2, 3], [2, 1]], shape_type)
+    values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(arrow_type))
+    struct_arr = pa.StructArray.from_arrays([shapes, values], fields=fields)
+    arr = pa.ExtensionArray.from_storage(tensor_type, struct_arr)
+    basic_arr = pa.ExtensionArray.from_storage(
+        pa.variable_shape_tensor(arrow_type, ndim), struct_arr
+    )
+
+    storage = pa.array(
+        [([2, 3], [1, 2, 3, 4, 5, 6]), ([2, 1], [7, 8])], type=pa.struct(fields)
+    )
+    assert pa.ExtensionArray.from_storage(tensor_type, storage).equals(arr)
+
+    assert arr.type == tensor_type
+
+    ndarray_list = [
+        np.array([[1, 2, 3], [4, 5, 6]], dtype=value_type),
+        np.array([[7], [8]], dtype=value_type),
+    ]
+    assert all(zip(x == y for x, y in zip(arr.to_numpy_ndarray(), ndarray_list)))

Review Comment:
   Ah, right. It would be more idiomatic (and more helpful in case of failures) to use something like https://numpy.org/doc/stable/reference/generated/numpy.testing.assert_array_equal.html , though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1407043145


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   Sorry :)
   
   I've moved the python `get_tensor` logic to C++ (https://github.com/apache/arrow/pull/38008/commits/7e054f7a29243d372fa28f7430f6e31d191a6162).
   Here's the `VariableShapeTensorType.GetTensor(ExtensionScalar)` method. Unfortunately it also always gets values from the beginning of the buffer instead offset of the Scalar. I imagine there's a way to somehow grab the correct buffer slice that I'm missing. See line 220.
   
   https://github.com/apache/arrow/blob/7e054f7a29243d372fa28f7430f6e31d191a6162/cpp/src/arrow/extension/variable_shape_tensor.cc#L198-L224



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409320856


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));

Review Comment:
   Changed to:
   ```cpp
     std::vector<int64_t> shape;
     for (int32_t j = 0; j < this->ndim(); ++j) {
       ARROW_ASSIGN_OR_RAISE(const auto size, shape_array.GetScalar(j));
       const auto size_value = internal::checked_pointer_cast<Int32Scalar>(size)->value;
       if (size_value < 0) {
         return Status::Invalid("shape must have non-negative values");
       }
       shape.push_back(size_value);
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406633917


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   I think we're talking past each other here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1415802827


##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,114 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+    """
+
+    cdef void init(self, const shared_ptr[CDataType]& type) except *:
+        BaseExtensionType.init(self, type)
+        self.tensor_ext_type = <const CVariableShapeTensorType*> type.get()
+
+    @property
+    def value_type(self):
+        """
+        Data type of an individual tensor.
+        """
+        return pyarrow_wrap_data_type(self.tensor_ext_type.value_type())
+
+    @property
+    def ndim(self):
+        """
+        Number of dimensions of the tensors.
+        """
+        return self.tensor_ext_type.ndim()
+
+    @property
+    def dim_names(self):
+        """
+        Explicit names of the dimensions.
+        """
+        list_of_bytes = self.tensor_ext_type.dim_names()
+        if len(list_of_bytes) != 0:
+            return [frombytes(x) for x in list_of_bytes]
+        else:
+            return None
+
+    @property
+    def permutation(self):
+        """
+        Indices of the dimensions ordering.
+        """
+        indices = self.tensor_ext_type.permutation()
+        if len(indices) != 0:
+            return indices
+        else:
+            return None
+
+    @property
+    def uniform_shape(self):
+        """
+        Shape over dimensions that are guaranteed to be constant.
+        """
+        cdef:
+            vector[optional[int64_t]] c_uniform_shape = self.tensor_ext_type.uniform_shape()
+            length = c_uniform_shape.size()
+
+        if length == 0:
+            return None
+
+        uniform_shape = []
+        for i in range(length):
+            if c_uniform_shape[i].has_value():
+                uniform_shape.append(c_uniform_shape[i].value())
+            else:
+                uniform_shape.append(None)
+
+        return uniform_shape
+
+    def __arrow_ext_serialize__(self):
+        """
+        Serialized representation of metadata to reconstruct the type object.
+        """
+        return self.tensor_ext_type.Serialize()
+
+    @classmethod
+    def __arrow_ext_deserialize__(self, storage_type, serialized):
+        """
+        Return an VariableShapeTensor type instance from the storage type and serialized
+        metadata.
+        """
+        return self.tensor_ext_type.Deserialize(storage_type, serialized)
+
+    def __arrow_ext_class__(self):
+        return VariableShapeTensorArray
+
+    def __reduce__(self):
+        return variable_shape_tensor, (self.value_type, self.ndim,
+                                       self.dim_names, self.permutation,
+                                       self.uniform_shape)

Review Comment:
   Why not use `__arrow_ext_serialize__` and `__arrow_ext_deserialize__` generically instead of defining a specific `__reduce__` for each particular extension type?
   
   cc @jorisvandenbossche 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1403489583


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   @jorisvandenbossche is it expected behavior that this returns the full array underlying buffer instead of the scalar underlying slice? Is there a way to get access to the slice or the offset here? Currently this always returns the first scalar in the array which is not terribly useful.
   cc @AlenkaF 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1908377111

   @AlenkaF I do believe it is!
   There are a couple of open questions ([1](https://github.com/apache/arrow/pull/38008#discussion_r1407984112), [2](https://github.com/apache/arrow/pull/38008#discussion_r1427367604)) but I think they don't block review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542162612


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+    ARROW_ASSIGN_OR_RAISE(const auto size, shape_array->GetScalar(j));

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1543000177


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& permutation,
+                             std::vector<int64_t>* strides) {
+  const auto fixed_width_type =
+      internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);
+  }
+  const int byte_width = value_type->byte_width();
+
+  int64_t remaining = 0;
+  if (!shape.empty() && shape.front() > 0) {
+    remaining = byte_width;
+    for (auto i : permutation) {
+      if (i > 0) {
+        if (internal::MultiplyWithOverflow(remaining, shape[i], &remaining)) {
+          return Status::Invalid(
+              "Strides computed from shape would not fit in 64-bit integer");
+        }
+      }
+    }
+  }
+
+  if (remaining == 0) {
+    strides->assign(shape.size(), byte_width);
+    return Status::OK();
+  }
+
+  strides->push_back(remaining);
+  for (auto i : permutation) {
+    if (i > 0) {
+      remaining /= shape[i];

Review Comment:
   Good point. I'll make an issu.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409321257


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(this->value_type(), shape, this->permutation(), &strides));
+
+  const auto array = std::static_pointer_cast<FixedSizeListScalar>(data)->value;
+  const auto byte_width = this->value_type()->byte_width();
+  const auto start_position = array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  // Create a slice of the buffer
+  std::shared_ptr<arrow::Buffer> buffer =
+      arrow::SliceBuffer(array->data()->buffers[1], start_position, size * byte_width);
+
+  return Tensor::Make(this->value_type(), buffer, shape, strides, this->dim_names());
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Make(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  if (!permutation.empty() && permutation.size() != ndim) {
+    return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
+                           " Got: ", permutation.size());
+  }
+  if (!dim_names.empty() && dim_names.size() != ndim) {
+    return Status::Invalid("dim_names size must match ndim. Expected: ", ndim,
+                           " Got: ", dim_names.size());
+  }
+  if (uniform_shape.size() > ndim) {

Review Comment:
   Changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409505068


##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1353,6 +1386,70 @@ def test_tensor_class_methods():
         arr.to_numpy_ndarray()
 
 
+@pytest.mark.parametrize("value_type", (np.int8, np.int32, np.int64, np.float64))
+def test_variable_shape_tensor_class_methods(value_type):
+    ndim = 2
+    shape_type = pa.list_(pa.uint32(), ndim)
+    arrow_type = pa.from_numpy_dtype(value_type)
+    tensor_type = pa.variable_shape_tensor(
+        arrow_type,
+        ndim,
+        dim_names=["H", "W"],
+        permutation=[0, 1],
+        uniform_shape=[None, None],
+    )
+    fields = [pa.field("shape", shape_type), pa.field("data", pa.list_(arrow_type))]
+
+    shapes = pa.array([[2, 3], [2, 1]], shape_type)
+    values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(arrow_type))
+    struct_arr = pa.StructArray.from_arrays([shapes, values], fields=fields)
+    arr = pa.ExtensionArray.from_storage(tensor_type, struct_arr)
+    basic_arr = pa.ExtensionArray.from_storage(
+        pa.variable_shape_tensor(arrow_type, ndim), struct_arr
+    )
+
+    storage = pa.array(
+        [([2, 3], [1, 2, 3, 4, 5, 6]), ([2, 1], [7, 8])], type=pa.struct(fields)
+    )
+    assert pa.ExtensionArray.from_storage(tensor_type, storage).equals(arr)
+
+    assert arr.type == tensor_type
+
+    ndarray_list = [
+        np.array([[1, 2, 3], [4, 5, 6]], dtype=value_type),
+        np.array([[7], [8]], dtype=value_type),
+    ]
+    assert all(zip(x == y for x, y in zip(arr.to_numpy_ndarray(), ndarray_list)))

Review Comment:
   Ah, right. It would be more idiomatic (and more helpful in case of failures) to use https://numpy.org/doc/stable/reference/generated/numpy.testing.assert_array_equal.html , though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406515217


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   Why are you not doing this in C++ instead? This would probably allow doing what you want and would also be useful to non-Python libraries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409320856


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));

Review Comment:
   Changed to:
   ```cpp
     std::vector<int64_t> shape;
     for (int32_t j = 0; j < this->ndim(); ++j) {
       ARROW_ASSIGN_OR_RAISE(const auto size, shape_array.GetScalar(j));
       const auto size_value = internal::checked_pointer_cast<Int32Scalar>(size)->value;
       ARROW_DCHECK_GE(size_value, 0);
       shape.push_back(size_value);
     }```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));

Review Comment:
   Changed to:
   ```cpp
     std::vector<int64_t> shape;
     for (int32_t j = 0; j < this->ndim(); ++j) {
       ARROW_ASSIGN_OR_RAISE(const auto size, shape_array.GetScalar(j));
       const auto size_value = internal::checked_pointer_cast<Int32Scalar>(size)->value;
       ARROW_DCHECK_GE(size_value, 0);
       shape.push_back(size_value);
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1407406322


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   You simply have to apply the array offset to the buffer pointer, unless I'm missing something. If you're ignoring the offset, you'll certainly get the wrong data area indeed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1435411936


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)

Review Comment:
   We've decided to remove `VariableShapeTensorArray.to_tensor(i)` so this is moot.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1934620027

   @pitrou - now that #37533 was merged I've rebased this and it's again ready for review. :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1484612155


##########
python/pyarrow/array.pxi:
##########
@@ -3609,7 +3609,7 @@ cdef class FixedShapeTensorArray(ExtensionArray):
         and the rest of the dimensions will match the permuted shape of the fixed
         shape tensor.
 
-        The conversion is zero-copy.
+        The conversion is zero-copy if data is primitive numeric and without nulls.

Review Comment:
   Indeed. Changed, Scalar as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-2795560350

   That makes sense @pimdh! We will need to adjust [canonical extension spec](https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor) first or add LargeVariableShapeTensorArray.
   I imagine the same would then hold for [FixedShapeTensorArray](https://arrow.apache.org/docs/python/generated/pyarrow.FixedShapeTensorArray.html)?
   I'll have some time for this in a couple of weeks.
   
   @pimdh I (and probably @AlenkaF too) am curious about how you're reading and passing data around (e.g. DLpack) or what you are currently missing. We'd like to make this as useful as possible. Would you mind sharing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-2504300498

   @rok Can you ping me? Self-ping doesn't seem to generate any e-mail notifications :D


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3955756667

   After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 2fcc3ec9f04cfe73facef5155e831dac19ead853.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/64770403837) has more details. It also includes information about 9 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3952360543

   @pitrou I addressed the review comments and would like to proceed and merge this tomorrow. Any further comments?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-2504285743

   Self-ping @pitrou


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1547016173


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& permutation,
+                             std::vector<int64_t>* strides) {
+  const auto fixed_width_type =
+      internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);
+  }
+  const int byte_width = value_type->byte_width();
+
+  int64_t remaining = 0;
+  if (!shape.empty() && shape.front() > 0) {
+    remaining = byte_width;
+    for (auto i : permutation) {
+      if (i > 0) {
+        if (internal::MultiplyWithOverflow(remaining, shape[i], &remaining)) {
+          return Status::Invalid(
+              "Strides computed from shape would not fit in 64-bit integer");
+        }
+      }
+    }
+  }
+
+  if (remaining == 0) {
+    strides->assign(shape.size(), byte_width);
+    return Status::OK();
+  }
+
+  strides->push_back(remaining);
+  for (auto i : permutation) {
+    if (i > 0) {
+      remaining /= shape[i];

Review Comment:
   False alarm. Most `FixedShapeTensor` tests don't need to be refactored. Except two, which is still alarming and we should verify wheter this is introducing a new bug or fixing an old one ot both.
   On the positivne note `ComputeStrides` is now easier to reason about. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3521914095

   > As I understand it, the FixedShapeTensorArray has less need for a Large version, because the FixedSizeListType does not hold an offset int32 array like the ListType does. So 1 array can hold many tensors, as long as each is less than 2^31 elements. The VariableShapeTensorArray as it's currently documented can hold at most 2^31 elements in all tensors in the array combined, so that limit is hit earlier. Also, there doesn't seem to be a LargeFixedSizeListType, which has an int64 size?
   
   @pimdh Yes, that seems right.
   Now that some time has passed would you say there's still need for `VariableShapeTensorArray` with LargeList for `data`? We would need to amend the extension type or introduce `LargeVariableShapeTensorArray`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3954026339

   Thank you for all the reviews @pitrou & others!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2817423624


##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1482,6 +1482,17 @@ def test_tensor_class_methods(np_type_str):
     assert result.to_tensor().shape == (1, 3, 2, 2)
     assert result.to_tensor().strides == (12 * bw, 1 * bw, 6 * bw, 2 * bw)
 
+    tensor_type = pa.fixed_shape_tensor(arrow_type, [2, 2, 3], permutation=[2, 1, 0])

Review Comment:
   I suppose it's unrelated but it's testing something useful anyway?



##########
cpp/src/arrow/compute/kernels/CMakeLists.txt:
##########
@@ -22,7 +22,8 @@ arrow_install_all_headers("arrow/compute/kernels")
 
 # Define arrow_compute_kernels_testing object library for common test files
 if(ARROW_TESTING)
-  add_library(arrow_compute_kernels_testing OBJECT test_util_internal.cc)
+  add_library(arrow_compute_kernels_testing OBJECT
+              test_util_internal.cc ../../extension/tensor_extension_array_test.cc)

Review Comment:
   Why do we need this? This doesn't sound logical.



##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1712,7 +1723,7 @@ def test_tensor_type_is_picklable(pickle_module):
         'fixed_shape_tensor[value_type=int64, shape=[2,2,3], dim_names=[C,H,W]]'
     )
 ])
-def test_tensor_type_str(tensor_type, text):
+def test_tensor_type_str(tensor_type, text, pickle_module):

Review Comment:
   This change doesn't seem useful?



##########
cpp/src/arrow/extension/tensor_internal.cc:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/extension/tensor_internal.h"
+
+#include "arrow/tensor.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/sort_internal.h"
+
+#include "arrow/status.h"
+#include "arrow/util/print_internal.h"
+
+namespace arrow::internal {
+
+Status IsPermutationValid(const std::vector<int64_t>& permutation) {
+  const auto size = static_cast<int64_t>(permutation.size());
+  std::vector<uint8_t> dim_seen(size, 0);
+
+  for (const auto p : permutation) {
+    if (p < 0 || p >= size || dim_seen[p] != 0) {
+      return Status::Invalid(
+          "Permutation indices for ", size,
+          " dimensional tensors must be unique and within [0, ", size - 1,
+          "] range. Got: ", ::arrow::internal::PrintVector{permutation, ","});
+    }
+    dim_seen[p] = 1;
+  }
+  return Status::OK();
+}
+
+Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                      const std::vector<int64_t>& shape,
+                      const std::vector<int64_t>& permutation,
+                      std::vector<int64_t>* strides) {
+  auto fixed_width_type = internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);

Review Comment:
   This one is doing the exact same thing except no permutation, please reconcile the two functions instead of duplicating code.



##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -20,25 +20,17 @@
 #include <cstdint>
 #include <vector>
 
-#include "arrow/status.h"
-#include "arrow/util/print_internal.h"
+#include "arrow/array/array_nested.h"
 
 namespace arrow::internal {
 
-inline Status IsPermutationValid(const std::vector<int64_t>& permutation) {
-  const auto size = static_cast<int64_t>(permutation.size());
-  std::vector<uint8_t> dim_seen(size, 0);
+ARROW_EXPORT
+Status IsPermutationValid(const std::vector<int64_t>& permutation);
 
-  for (const auto p : permutation) {
-    if (p < 0 || p >= size || dim_seen[p] != 0) {
-      return Status::Invalid(
-          "Permutation indices for ", size,
-          " dimensional tensors must be unique and within [0, ", size - 1,
-          "] range. Got: ", ::arrow::internal::PrintVector{permutation, ","});
-    }
-    dim_seen[p] = 1;
-  }
-  return Status::OK();
-}
+ARROW_EXPORT
+Status ComputeStrides(const std::shared_ptr<DataType>& value_type,

Review Comment:
   Could return `Result<std::vector<int64_t>>` for better ergonomics



##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -20,25 +20,17 @@
 #include <cstdint>
 #include <vector>
 
-#include "arrow/status.h"
-#include "arrow/util/print_internal.h"
+#include "arrow/array/array_nested.h"
 
 namespace arrow::internal {
 
-inline Status IsPermutationValid(const std::vector<int64_t>& permutation) {
-  const auto size = static_cast<int64_t>(permutation.size());
-  std::vector<uint8_t> dim_seen(size, 0);
+ARROW_EXPORT
+Status IsPermutationValid(const std::vector<int64_t>& permutation);

Review Comment:
   These helper functions all could take `std::span<const int64_t>` as inputs, though not important either.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,314 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/util/print_internal.h"
+#include "arrow/util/sort_internal.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow::extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = internal::checked_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (internal::checked_cast<const FixedSizeListType&>(*storage_type->field(1)->type())
+          .value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(0)->type()->field(0)->type();
+  const uint32_t ndim =
+      internal::checked_cast<const FixedSizeListType&>(*storage_type->field(1)->type())
+          .list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    permutation.reserve(ndim);
+    for (const auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    dim_names.reserve(ndim);
+    for (const auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    uniform_shape.reserve(ndim);
+    for (const auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+  }
+
+  return VariableShapeTensorType::Make(value_type, ndim, permutation, dim_names,
+                                       uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto& tensor_scalar = internal::checked_cast<const StructScalar&>(*scalar->value);
+  const auto& ext_type =
+      internal::checked_cast<const VariableShapeTensorType&>(*scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar.field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar.field(1));
+  ARROW_CHECK(tensor_scalar.is_valid);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+
+  const auto& value_type =
+      internal::checked_cast<const FixedWidthType&>(*ext_type.value_type());
+
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls to Tensor.");
+  }
+
+  auto permutation = ext_type.permutation();
+  if (permutation.empty()) {
+    permutation.resize(ext_type.ndim());
+    std::iota(permutation.begin(), permutation.end(), 0);
+  }
+
+  ARROW_CHECK_EQ(shape_array->length(), ext_type.ndim());
+  std::vector<int64_t> shape;
+  shape.reserve(ext_type.ndim());
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type.ndim()); ++j) {
+    const auto size_value = shape_array->Value(j);
+    if (size_value < 0) {
+      return Status::Invalid("shape must have non-negative values");
+    }
+    shape.push_back(std::move(size_value));
+  }
+
+  std::vector<std::string> dim_names = ext_type.dim_names();
+  if (!dim_names.empty()) {
+    internal::Permute<std::string>(permutation, &dim_names);
+  }
+
+  std::vector<int64_t> strides;
+  ARROW_RETURN_NOT_OK(
+      internal::ComputeStrides(ext_type.value_type(), shape, permutation, &strides));
+  internal::Permute<int64_t>(permutation, &shape);
+
+  const auto byte_width = value_type.byte_width();
+  const auto start_position = data_array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+  ARROW_CHECK_EQ(size * byte_width, data_array->length() * byte_width);

Review Comment:
   This can be evidently simplified.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,314 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/util/print_internal.h"
+#include "arrow/util/sort_internal.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow::extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = internal::checked_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {

Review Comment:
   Again, please don't duplicate code like this, factor out into common helpers.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,314 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/util/print_internal.h"
+#include "arrow/util/sort_internal.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow::extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = internal::checked_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (internal::checked_cast<const FixedSizeListType&>(*storage_type->field(1)->type())
+          .value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(0)->type()->field(0)->type();
+  const uint32_t ndim =
+      internal::checked_cast<const FixedSizeListType&>(*storage_type->field(1)->type())
+          .list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    permutation.reserve(ndim);
+    for (const auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }

Review Comment:
   What happens if "permutation" is not an array of ints?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2835241761


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,314 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/util/print_internal.h"
+#include "arrow/util/sort_internal.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow::extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = internal::checked_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (internal::checked_cast<const FixedSizeListType&>(*storage_type->field(1)->type())
+          .value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(0)->type()->field(0)->type();
+  const uint32_t ndim =
+      internal::checked_cast<const FixedSizeListType&>(*storage_type->field(1)->type())
+          .list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    permutation.reserve(ndim);
+    for (const auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    dim_names.reserve(ndim);
+    for (const auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    uniform_shape.reserve(ndim);
+    for (const auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+  }
+
+  return VariableShapeTensorType::Make(value_type, ndim, permutation, dim_names,
+                                       uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto& tensor_scalar = internal::checked_cast<const StructScalar&>(*scalar->value);
+  const auto& ext_type =
+      internal::checked_cast<const VariableShapeTensorType&>(*scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar.field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar.field(1));
+  ARROW_CHECK(tensor_scalar.is_valid);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+
+  const auto& value_type =
+      internal::checked_cast<const FixedWidthType&>(*ext_type.value_type());
+
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls to Tensor.");
+  }
+
+  auto permutation = ext_type.permutation();
+  if (permutation.empty()) {
+    permutation.resize(ext_type.ndim());
+    std::iota(permutation.begin(), permutation.end(), 0);
+  }
+
+  ARROW_CHECK_EQ(shape_array->length(), ext_type.ndim());
+  std::vector<int64_t> shape;
+  shape.reserve(ext_type.ndim());
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type.ndim()); ++j) {
+    const auto size_value = shape_array->Value(j);
+    if (size_value < 0) {
+      return Status::Invalid("shape must have non-negative values");
+    }
+    shape.push_back(std::move(size_value));
+  }
+
+  std::vector<std::string> dim_names = ext_type.dim_names();
+  if (!dim_names.empty()) {
+    internal::Permute<std::string>(permutation, &dim_names);
+  }
+
+  std::vector<int64_t> strides;
+  ARROW_RETURN_NOT_OK(
+      internal::ComputeStrides(ext_type.value_type(), shape, permutation, &strides));
+  internal::Permute<int64_t>(permutation, &shape);
+
+  const auto byte_width = value_type.byte_width();
+  const auto start_position = data_array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+  ARROW_CHECK_EQ(size * byte_width, data_array->length() * byte_width);

Review Comment:
   Indeed! Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406595306


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   If I do this: `shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])` I get the whole buffer of the underlying array instead of a slice that I'd expect to be the scalar. Is there a way to gust get the buffer slice?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406508534


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   cc @pitrou 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409314825


##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,
+                          const std::vector<int64_t>& permutation = {},
+                          const std::vector<std::string>& dim_names = {},
+                          const std::vector<std::optional<int64_t>>& uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(uint32(), ndim)),

Review Comment:
   Indeed! Changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1410129161


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)

Review Comment:
   It returns a tensor with all dimension sizes == 0. Added a test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409322590


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(this->value_type(), shape, this->permutation(), &strides));
+
+  const auto array = std::static_pointer_cast<FixedSizeListScalar>(data)->value;
+  const auto byte_width = this->value_type()->byte_width();
+  const auto start_position = array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  // Create a slice of the buffer
+  std::shared_ptr<arrow::Buffer> buffer =
+      arrow::SliceBuffer(array->data()->buffers[1], start_position, size * byte_width);
+
+  return Tensor::Make(this->value_type(), buffer, shape, strides, this->dim_names());
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Make(
+    const std::shared_ptr<DataType>& value_type, const uint32_t& ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  if (!permutation.empty() && permutation.size() != ndim) {
+    return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
+                           " Got: ", permutation.size());
+  }
+  if (!dim_names.empty() && dim_names.size() != ndim) {
+    return Status::Invalid("dim_names size must match ndim. Expected: ", ndim,
+                           " Got: ", dim_names.size());
+  }
+  if (uniform_shape.size() > ndim) {
+    return Status::Invalid("uniform_shape size must be less or equal ndim.");
+  }

Review Comment:
   Changed to:
   ```cpp
     if (!uniform_shape.empty() && uniform_shape.size() != static_cast<size_t>(ndim)) {
       return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
                              " Got: ", uniform_shape.size());
     }
     if (!uniform_shape.empty()) {
       for (const auto& v : uniform_shape) {
         if (v.has_value() && v.value() < 0) {
           return Status::Invalid("uniform_shape must have non-negative values");
         }
       }
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409328480


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();

Review Comment:
   Thank you. However, these should return an error instead of aborting. The idea is that deserialization can be passed arbitrary data (perhaps malicious), so we should ensure we validate the data and always return an error without crashing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409314825


##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,
+                          const std::vector<int64_t>& permutation = {},
+                          const std::vector<std::string>& dim_names = {},
+                          const std::vector<std::optional<int64_t>>& uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(uint32(), ndim)),

Review Comment:
   Indeed!



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,

Review Comment:
   Changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409341931


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):

Review Comment:
   `to_numpy_ndarray_list`?
   I'm a bit uncertain how usage of this would look like.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1414528595


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):

Review Comment:
   BTW this method also has a different return type than the `FixedShapeTensorArray.to_numpy_ndarray` we added, so that's another reason to use a different name (or just remove it)
   
   For FixedShapeTensorArray the idea was that we wanted to be able to return a single nD array, which is possible for fixed shape, but not for variable shape.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409358717


##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -2664,6 +2698,11 @@ cdef extern from "arrow/extension/fixed_shape_tensor.h" namespace "arrow::extens
         const vector[int64_t] permutation()
         const vector[c_string] dim_names()
 
+cdef extern from "arrow/extension/fixed_shape_tensor.h" namespace "arrow::internal" nogil:
+    cdef CStatus ComputeStrides(const shared_ptr[CDataType]& value_type,

Review Comment:
   Not anymore. Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409365260


##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,114 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+    """
+
+    cdef void init(self, const shared_ptr[CDataType]& type) except *:
+        BaseExtensionType.init(self, type)
+        self.tensor_ext_type = <const CVariableShapeTensorType*> type.get()
+
+    @property
+    def value_type(self):
+        """
+        Data type of an individual tensor.
+        """
+        return pyarrow_wrap_data_type(self.tensor_ext_type.value_type())
+
+    @property
+    def ndim(self):
+        """
+        Number of dimensions of the tensors.
+        """
+        return self.tensor_ext_type.ndim()
+
+    @property
+    def dim_names(self):
+        """
+        Explicit names of the dimensions.
+        """
+        list_of_bytes = self.tensor_ext_type.dim_names()
+        if len(list_of_bytes) != 0:
+            return [frombytes(x) for x in list_of_bytes]
+        else:
+            return None
+
+    @property
+    def permutation(self):
+        """
+        Indices of the dimensions ordering.
+        """
+        indices = self.tensor_ext_type.permutation()
+        if len(indices) != 0:
+            return indices
+        else:
+            return None
+
+    @property
+    def uniform_shape(self):
+        """
+        Shape over dimensions that are guaranteed to be constant.
+        """
+        cdef:
+            vector[optional[int64_t]] c_uniform_shape = self.tensor_ext_type.uniform_shape()
+            length = c_uniform_shape.size()
+
+        if length == 0:
+            return None
+
+        uniform_shape = []
+        for i in range(length):
+            if c_uniform_shape[i].has_value():
+                uniform_shape.append(c_uniform_shape[i].value())
+            else:
+                uniform_shape.append(None)
+
+        return uniform_shape
+
+    def __arrow_ext_serialize__(self):
+        """
+        Serialized representation of metadata to reconstruct the type object.
+        """
+        return self.tensor_ext_type.Serialize()
+
+    @classmethod
+    def __arrow_ext_deserialize__(self, storage_type, serialized):
+        """
+        Return an VariableShapeTensor type instance from the storage type and serialized
+        metadata.
+        """
+        return self.tensor_ext_type.Deserialize(storage_type, serialized)

Review Comment:
   Removed also for FixedSizeShapeTensor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1837045096

   Thank you for the thorough review @pitrou ! I addressed your comments (except for the `from_numpy` allowing for general strides which I'll do asap).
   Feel free to do another pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1410029838


##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1353,6 +1386,70 @@ def test_tensor_class_methods():
         arr.to_numpy_ndarray()
 
 
+@pytest.mark.parametrize("value_type", (np.int8, np.int32, np.int64, np.float64))
+def test_variable_shape_tensor_class_methods(value_type):
+    ndim = 2
+    shape_type = pa.list_(pa.uint32(), ndim)
+    arrow_type = pa.from_numpy_dtype(value_type)
+    tensor_type = pa.variable_shape_tensor(
+        arrow_type,
+        ndim,
+        dim_names=["H", "W"],
+        permutation=[0, 1],
+        uniform_shape=[None, None],
+    )
+    fields = [pa.field("shape", shape_type), pa.field("data", pa.list_(arrow_type))]
+
+    shapes = pa.array([[2, 3], [2, 1]], shape_type)
+    values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(arrow_type))
+    struct_arr = pa.StructArray.from_arrays([shapes, values], fields=fields)
+    arr = pa.ExtensionArray.from_storage(tensor_type, struct_arr)
+    basic_arr = pa.ExtensionArray.from_storage(
+        pa.variable_shape_tensor(arrow_type, ndim), struct_arr
+    )
+
+    storage = pa.array(
+        [([2, 3], [1, 2, 3, 4, 5, 6]), ([2, 1], [7, 8])], type=pa.struct(fields)
+    )
+    assert pa.ExtensionArray.from_storage(tensor_type, storage).equals(arr)
+
+    assert arr.type == tensor_type
+
+    ndarray_list = [
+        np.array([[1, 2, 3], [4, 5, 6]], dtype=value_type),
+        np.array([[7], [8]], dtype=value_type),
+    ]
+    assert all(zip(x == y for x, y in zip(arr.to_numpy_ndarray(), ndarray_list)))

Review Comment:
   Hm, changed to:
   ```python
       list(np.testing.assert_array_equal(x, y) for x, y in zip(arr.to_numpy_ndarray(), ndarray_list))
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1410029133


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,31 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        """
+        return self.to_tensor().to_numpy()
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            CVariableShapeTensorType* c_type = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type).get()
+            shared_ptr[CExtensionScalar] scalar = static_pointer_cast[CExtensionScalar, CScalar](self.wrapped)
+            shared_ptr[CTensor] ctensor
+
+        ctensor = GetResultValue(c_type.GetTensor(scalar))

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409344344


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Numpy arrays needs to be C-contiguous in memory (``obj.flags["C_CONTIGUOUS"]==True``).
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)
+        <pyarrow.lib.VariableShapeTensorArray object at ...>
+        -- is_valid: all not null
+        -- child 0 type: fixed_size_list<item: uint32>[2]
+          [
+            [
+              2,
+              3
+            ],
+            [
+              1,
+              2
+            ]
+          ]
+        -- child 1 type: list<item: float>
+          [
+            [
+              1,
+              2,
+              3,
+              4,
+              5,
+              6
+            ],
+            [
+              7,
+              8
+            ]
+          ]
+        """
+        if not all([o.flags["C_CONTIGUOUS"] for o in obj]):
+            raise ValueError('The data in the numpy arrays need to be in a single, '
+                             'C-style contiguous segment.')
+        numpy_type = obj[0].dtype
+        ndim = obj[0].ndim
+
+        if not all([o.dtype == numpy_type for o in obj]):
+            raise ValueError('All numpy arrays need to have the same dtype.')
+
+        if not all([o.ndim == ndim for o in obj]):
+            raise ValueError('All numpy arrays need to have the same ndim.')
+
+        arrow_type = from_numpy_dtype(numpy_type)
+        values = array([np.ravel(o, order='C') for o in obj], list_(arrow_type))
+        shapes = array([o.shape for o in obj], list_(uint32(), list_size=ndim))

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409346282


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Numpy arrays needs to be C-contiguous in memory (``obj.flags["C_CONTIGUOUS"]==True``).
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)
+        <pyarrow.lib.VariableShapeTensorArray object at ...>
+        -- is_valid: all not null
+        -- child 0 type: fixed_size_list<item: uint32>[2]
+          [
+            [
+              2,
+              3
+            ],
+            [
+              1,
+              2
+            ]
+          ]
+        -- child 1 type: list<item: float>
+          [
+            [
+              1,
+              2,
+              3,
+              4,
+              5,
+              6
+            ],
+            [
+              7,
+              8
+            ]
+          ]
+        """
+        if not all([o.flags["C_CONTIGUOUS"] for o in obj]):
+            raise ValueError('The data in the numpy arrays need to be in a single, '
+                             'C-style contiguous segment.')
+        numpy_type = obj[0].dtype
+        ndim = obj[0].ndim
+
+        if not all([o.dtype == numpy_type for o in obj]):
+            raise ValueError('All numpy arrays need to have the same dtype.')

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409318263


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();

Review Comment:
   Added DCHECKs below. Do you think anything is missing?
   ```cpp
     ARROW_DCHECK_EQ(storage_type->num_fields(), 2);
     ARROW_DCHECK_EQ(storage_type->field(0)->type()->id(), Type::FIXED_SIZE_LIST);
     ARROW_DCHECK_EQ(
         std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
             ->value_type(),
         int32());
     ARROW_DCHECK_EQ(storage_type->field(1)->type()->id(), Type::LIST);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409316165


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))

Review Comment:
   Changed.



##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1544935676


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& permutation,
+                             std::vector<int64_t>* strides) {
+  const auto fixed_width_type =
+      internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);
+  }
+  const int byte_width = value_type->byte_width();
+
+  int64_t remaining = 0;
+  if (!shape.empty() && shape.front() > 0) {
+    remaining = byte_width;
+    for (auto i : permutation) {
+      if (i > 0) {
+        if (internal::MultiplyWithOverflow(remaining, shape[i], &remaining)) {
+          return Status::Invalid(
+              "Strides computed from shape would not fit in 64-bit integer");
+        }
+      }
+    }
+  }
+
+  if (remaining == 0) {
+    strides->assign(shape.size(), byte_width);
+    return Status::OK();
+  }
+
+  strides->push_back(remaining);
+  for (auto i : permutation) {
+    if (i > 0) {
+      remaining /= shape[i];

Review Comment:
   How about this: https://github.com/apache/arrow/pull/38008/commits/8e6e99093f493bbe517cdec102d2dca01d2fcac5
   Somewhat disturbingly this required changing strides in tests, implying either this is wrong or the old implementation  was.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1407051615


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   [Example error](https://github.com/apache/arrow/actions/runs/7013790354/job/19080485614?pr=38008#step:6:6320). Any suggestions how to access the correct slice?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406547047


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')

Review Comment:
   Indeed!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1407717457


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   Done!
   I was surprised that we default to full buffer there or at least that slice is not easily available.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1414216962


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Numpy arrays needs to be C-contiguous in memory (``obj.flags["C_CONTIGUOUS"]==True``).
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)
+        <pyarrow.lib.VariableShapeTensorArray object at ...>
+        -- is_valid: all not null
+        -- child 0 type: fixed_size_list<item: uint32>[2]
+          [
+            [
+              2,
+              3
+            ],
+            [
+              1,
+              2
+            ]
+          ]
+        -- child 1 type: list<item: float>
+          [
+            [
+              1,
+              2,
+              3,
+              4,
+              5,
+              6
+            ],
+            [
+              7,
+              8
+            ]
+          ]
+        """
+        if not all([o.flags["C_CONTIGUOUS"] for o in obj]):
+            raise ValueError('The data in the numpy arrays need to be in a single, '
+                             'C-style contiguous segment.')

Review Comment:
   Ok, this should work now. We should test it well to be certain.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1976951007

   @rok I think this would be easier to review if you first submitted a purely C++ PR, can you do that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1427125194


##########
cpp/src/arrow/extension/fixed_shape_tensor.h:
##########
@@ -15,9 +15,21 @@
 // specific language governing permissions and limitations
 // under the License.
 
+#pragma once
+
 #include "arrow/extension_type.h"
 
 namespace arrow {
+namespace internal {

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1427141819


##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,114 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+    """
+
+    cdef void init(self, const shared_ptr[CDataType]& type) except *:
+        BaseExtensionType.init(self, type)
+        self.tensor_ext_type = <const CVariableShapeTensorType*> type.get()
+
+    @property
+    def value_type(self):
+        """
+        Data type of an individual tensor.
+        """
+        return pyarrow_wrap_data_type(self.tensor_ext_type.value_type())
+
+    @property
+    def ndim(self):
+        """
+        Number of dimensions of the tensors.
+        """
+        return self.tensor_ext_type.ndim()
+
+    @property
+    def dim_names(self):
+        """
+        Explicit names of the dimensions.
+        """
+        list_of_bytes = self.tensor_ext_type.dim_names()
+        if len(list_of_bytes) != 0:
+            return [frombytes(x) for x in list_of_bytes]
+        else:
+            return None
+
+    @property
+    def permutation(self):
+        """
+        Indices of the dimensions ordering.
+        """
+        indices = self.tensor_ext_type.permutation()
+        if len(indices) != 0:
+            return indices
+        else:
+            return None
+
+    @property
+    def uniform_shape(self):
+        """
+        Shape over dimensions that are guaranteed to be constant.
+        """
+        cdef:
+            vector[optional[int64_t]] c_uniform_shape = self.tensor_ext_type.uniform_shape()
+            length = c_uniform_shape.size()
+
+        if length == 0:
+            return None
+
+        uniform_shape = []
+        for i in range(length):
+            if c_uniform_shape[i].has_value():
+                uniform_shape.append(c_uniform_shape[i].value())
+            else:
+                uniform_shape.append(None)
+
+        return uniform_shape
+
+    def __arrow_ext_serialize__(self):
+        """
+        Serialized representation of metadata to reconstruct the type object.
+        """
+        return self.tensor_ext_type.Serialize()
+
+    @classmethod
+    def __arrow_ext_deserialize__(self, storage_type, serialized):
+        """
+        Return an VariableShapeTensor type instance from the storage type and serialized
+        metadata.
+        """
+        return self.tensor_ext_type.Deserialize(storage_type, serialized)
+
+    def __arrow_ext_class__(self):
+        return VariableShapeTensorArray
+
+    def __reduce__(self):
+        return variable_shape_tensor, (self.value_type, self.ndim,
+                                       self.dim_names, self.permutation,
+                                       self.uniform_shape)

Review Comment:
   Is this the proposed solution? https://github.com/apache/arrow/issues/39094#issuecomment-1845619410



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1427367604


##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,114 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+    """
+
+    cdef void init(self, const shared_ptr[CDataType]& type) except *:
+        BaseExtensionType.init(self, type)
+        self.tensor_ext_type = <const CVariableShapeTensorType*> type.get()
+
+    @property
+    def value_type(self):
+        """
+        Data type of an individual tensor.
+        """
+        return pyarrow_wrap_data_type(self.tensor_ext_type.value_type())
+
+    @property
+    def ndim(self):
+        """
+        Number of dimensions of the tensors.
+        """
+        return self.tensor_ext_type.ndim()
+
+    @property
+    def dim_names(self):
+        """
+        Explicit names of the dimensions.
+        """
+        list_of_bytes = self.tensor_ext_type.dim_names()
+        if len(list_of_bytes) != 0:
+            return [frombytes(x) for x in list_of_bytes]
+        else:
+            return None
+
+    @property
+    def permutation(self):
+        """
+        Indices of the dimensions ordering.
+        """
+        indices = self.tensor_ext_type.permutation()
+        if len(indices) != 0:
+            return indices
+        else:
+            return None
+
+    @property
+    def uniform_shape(self):
+        """
+        Shape over dimensions that are guaranteed to be constant.
+        """
+        cdef:
+            vector[optional[int64_t]] c_uniform_shape = self.tensor_ext_type.uniform_shape()
+            length = c_uniform_shape.size()
+
+        if length == 0:
+            return None
+
+        uniform_shape = []
+        for i in range(length):
+            if c_uniform_shape[i].has_value():
+                uniform_shape.append(c_uniform_shape[i].value())
+            else:
+                uniform_shape.append(None)
+
+        return uniform_shape
+
+    def __arrow_ext_serialize__(self):
+        """
+        Serialized representation of metadata to reconstruct the type object.
+        """
+        return self.tensor_ext_type.Serialize()
+
+    @classmethod
+    def __arrow_ext_deserialize__(self, storage_type, serialized):
+        """
+        Return an VariableShapeTensor type instance from the storage type and serialized
+        metadata.
+        """
+        return self.tensor_ext_type.Deserialize(storage_type, serialized)
+
+    def __arrow_ext_class__(self):
+        return VariableShapeTensorArray
+
+    def __reduce__(self):
+        return variable_shape_tensor, (self.value_type, self.ndim,
+                                       self.dim_names, self.permutation,
+                                       self.uniform_shape)

Review Comment:
   Naively changing:
   ```diff
   --- a/python/pyarrow/includes/libarrow.pxd
   +++ b/python/pyarrow/includes/libarrow.pxd
   @@ -2672,6 +2672,11 @@ cdef extern from "arrow/extension/variable_shape_tensor.h" namespace "arrow::ext
            const vector[c_string] dim_names()
            const vector[optional[int64_t]] uniform_shape()
    
   +        CResult[shared_ptr[CBuffer]] Serialize() const
   +
   +        @staticmethod
   +        CResult[unique_ptr[CExtensionType]] Deserialize(
   +            const c_string& type_name, const CBuffer& buffer)
    
   diff --git a/python/pyarrow/types.pxi b/python/pyarrow/types.pxi
   index 5e2200ceb..e7baeb889 100644
   --- a/python/pyarrow/types.pxi
   +++ b/python/pyarrow/types.pxi
   @@ -1677,13 +1677,11 @@ cdef class VariableShapeTensorType(BaseExtensionType):
            return VariableShapeTensorArray
    
   +     def __arrow_ext_serialize__(self):
   +        return self.Serialize()
   +
   +     @classmethod
   +    def __arrow_ext_deserialize__(cls, storage_type, serialized):
   +        return cls.Deserialize(storage_type, serialized)
   ```
   
   Fails with:
   ```pytest
   _____________________ test_variable_shape_tensor_type_is_picklable[builtin_pickle] ______________________
   
   >   alias = _type_aliases[name]
   E   KeyError: 'extension<arrow.variable_shape_tensor[value_type=int32, ndim=2]>'
   
   pyarrow/types.pxi:5221: KeyError
   
   During handling of the above exception, another exception occurred:
   
   pickle_module = <module 'pickle' from '/usr/lib/python3.10/pickle.py'>
   
       def test_variable_shape_tensor_type_is_picklable(pickle_module):
           expected_type = pa.variable_shape_tensor(pa.int32(), 2)
   >       result = pickle_module.loads(pickle_module.dumps(expected_type))
   
   pyarrow/tests/test_extension_type.py:1667: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   
   >   raise ValueError('No type alias for {0}'.format(name))
   E   ValueError: No type alias for extension<arrow.variable_shape_tensor[value_type=int32, ndim=2]>
   
   pyarrow/types.pxi:5223: ValueError
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2835324321


##########
cpp/src/arrow/extension/tensor_internal.cc:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/extension/tensor_internal.h"
+
+#include "arrow/tensor.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/sort_internal.h"
+
+#include "arrow/status.h"
+#include "arrow/util/print_internal.h"
+
+namespace arrow::internal {
+
+Status IsPermutationValid(const std::vector<int64_t>& permutation) {
+  const auto size = static_cast<int64_t>(permutation.size());
+  std::vector<uint8_t> dim_seen(size, 0);
+
+  for (const auto p : permutation) {
+    if (p < 0 || p >= size || dim_seen[p] != 0) {
+      return Status::Invalid(
+          "Permutation indices for ", size,
+          " dimensional tensors must be unique and within [0, ", size - 1,
+          "] range. Got: ", ::arrow::internal::PrintVector{permutation, ","});
+    }
+    dim_seen[p] = 1;
+  }
+  return Status::OK();
+}
+
+Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                      const std::vector<int64_t>& shape,
+                      const std::vector<int64_t>& permutation,
+                      std::vector<int64_t>* strides) {
+  auto fixed_width_type = internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);

Review Comment:
   I think I addressed this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "pimdh (via GitHub)" <gi...@apache.org>.
pimdh commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-2800814128

   Thanks a lot! I'm currently using a combination of Arrow with Ray (Data) for data transformation pipelines, and Lance for storage. Now I'm using a custom VariableShapeTensor, awaiting this official type to land.
   This is all on host memory. I copy things to numpy first, and then to GPU. I plan to experiment with Arrow+DLPack in the future.
   
   As I understand it, the FixedShapeTensorArray has less need for a Large version, because the FixedSizeListType does not hold an offset int32 array like the ListType does. So 1 array can hold many tensors, as long as each is less than 2^31 elements. The VariableShapeTensorArray as it's currently documented can hold at most 2^31 elements in all tensors in the array combined, so that limit is hit earlier. Also, there doesn't seem to be a LargeFixedSizeListType, which has an int64 size?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3522208644

   Given that there is some interest in using [VariableShapeTensor](https://github.com/apache/arrow/discussions/48099) and that it was already [implemented in rust](https://arrow.apache.org/rust/arrow_schema/extension/struct.VariableShapeTensor.html) it might make sense to merge this PR. @pitrou would you happen to have time to take another look here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1907667047

   @rok is this ready for another round of review?
   If it is close to ready, I will tag the issue with 15.0.1 and 16.0.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542161085


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542160044


##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type, const int32_t ndim,
+                          const std::vector<int64_t> permutation = {},
+                          const std::vector<std::string> dim_names = {},
+                          const std::vector<std::optional<int64_t>> uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(int32(), ndim)),
+                               ::arrow::field("data", list(value_type))})),

Review Comment:
   Nice catch. Changed to match spec.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542159505


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& permutation,
+                             std::vector<int64_t>* strides) {
+  const auto fixed_width_type =
+      internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);
+  }
+  const int byte_width = value_type->byte_width();
+
+  int64_t remaining = 0;
+  if (!shape.empty() && shape.front() > 0) {

Review Comment:
   Generally no. If we're calculating strides for an element this is not needed. If we're transforming a full `FixedShapeTensorArray` to `Tensor` then yes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542157796


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -19,14 +19,19 @@
 
 #include <cstdint>
 #include <vector>
+#include "arrow/array/array_nested.h"
+#include "arrow/tensor.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/sort.h"
 
 #include "arrow/status.h"
 #include "arrow/util/print.h"
 
 namespace arrow::internal {
 
 ARROW_EXPORT
-Status IsPermutationValid(const std::vector<int64_t>& permutation) {
+inline Status IsPermutationValid(const std::vector<int64_t>& permutation) {

Review Comment:
   Seems fair, moved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1542174125


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+    ARROW_ASSIGN_OR_RAISE(const auto size, shape_array->GetScalar(j));
+    auto size_value = internal::checked_pointer_cast<Int32Scalar>(size)->value;
+    if (size_value < 0) {
+      return Status::Invalid("shape must have non-negative values");
+    }
+    shape.push_back(std::move(size_value));
+  }
+  internal::Permute<int64_t>(permutation, &shape);
+
+  std::vector<std::string> dim_names = ext_type->dim_names();
+  if (!dim_names.empty()) {
+    internal::Permute<std::string>(permutation, &dim_names);
+  }
+
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(internal::ComputeStrides(value_type, shape, permutation, &strides));
+
+  const auto byte_width = value_type->byte_width();
+  const auto start_position = data_array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  ARROW_ASSIGN_OR_RAISE(
+      const auto buffer,
+      SliceBufferSafe(data_array->data()->buffers[1], start_position, size * byte_width));
+
+  return Tensor::Make(value_type, std::move(buffer), std::move(shape), std::move(strides),
+                      ext_type->dim_names());
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Make(
+    const std::shared_ptr<DataType>& value_type, const int32_t ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  if (!dim_names.empty() && dim_names.size() != static_cast<size_t>(ndim)) {
+    return Status::Invalid("dim_names size must match ndim. Expected: ", ndim,
+                           " Got: ", dim_names.size());
+  }
+  if (!uniform_shape.empty() && uniform_shape.size() != static_cast<size_t>(ndim)) {
+    return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                           " Got: ", uniform_shape.size());
+  }
+  if (!uniform_shape.empty()) {
+    for (const auto& v : uniform_shape) {
+      if (v.has_value() && v.value() < 0) {
+        return Status::Invalid("uniform_shape must have non-negative values");
+      }
+    }
+  }
+  if (!permutation.empty()) {
+    if (permutation.size() != static_cast<size_t>(ndim)) {
+      return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
+                             " Got: ", permutation.size());
+    }
+
+    std::vector<int64_t> sorted_permutation = permutation;

Review Comment:
   Indeed, using it now. Still need to check `permutation.size() == ndim` as `IsPermutationValid` doesn't check for it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1539404320


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,

Review Comment:
   Same here: can move definition to `tensor_internal.cc`.



##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -19,14 +19,19 @@
 
 #include <cstdint>
 #include <vector>
+#include "arrow/array/array_nested.h"

Review Comment:
   Nit: add newline.
   ```suggestion
   
   #include "arrow/array/array_nested.h"
   ```



##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& permutation,
+                             std::vector<int64_t>* strides) {
+  const auto fixed_width_type =
+      internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);
+  }
+  const int byte_width = value_type->byte_width();
+
+  int64_t remaining = 0;
+  if (!shape.empty() && shape.front() > 0) {
+    remaining = byte_width;
+    for (auto i : permutation) {
+      if (i > 0) {
+        if (internal::MultiplyWithOverflow(remaining, shape[i], &remaining)) {
+          return Status::Invalid(
+              "Strides computed from shape would not fit in 64-bit integer");
+        }
+      }
+    }
+  }
+
+  if (remaining == 0) {
+    strides->assign(shape.size(), byte_width);
+    return Status::OK();
+  }
+
+  strides->push_back(remaining);
+  for (auto i : permutation) {
+    if (i > 0) {
+      remaining /= shape[i];
+      strides->push_back(remaining);
+    }
+  }

Review Comment:
   Perhaps a sanity check here:
   ```suggestion
     }
     DCHECK_EQ(strides.back(), byte_width);
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type, const int32_t ndim,
+                          const std::vector<int64_t> permutation = {},
+                          const std::vector<std::string> dim_names = {},
+                          const std::vector<std::optional<int64_t>> uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(int32(), ndim)),
+                               ::arrow::field("data", list(value_type))})),

Review Comment:
   The spec puts the `data` field before the `shape` field:
   https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor
   



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);

Review Comment:
   Let's check the type properly in debug builds
   ```suggestion
     const auto& other_ext = checked_cast<const VariableShapeTensorType&>(other);
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())

Review Comment:
   ```suggestion
     if (checked_cast<const FixedSizeListType&>(*storage_type->field(0)->type())
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {

Review Comment:
   ```suggestion
       uniform_shape.reserve(ndim);
       for (const auto& x : document["uniform_shape"].GetArray()) {
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {

Review Comment:
   Should first check that the shape array length and null_count.



##########
cpp/src/arrow/extension/tensor_extension_array_test.cc:
##########
@@ -165,47 +170,51 @@ TEST_F(TestExtensionType, CreateFromArray) {
   ASSERT_EQ(ext_arr->null_count(), 0);
 }
 
+template <typename T>

Review Comment:
   No need for this if we just cast to `ExtensionType` below.



##########
cpp/src/arrow/extension/tensor_extension_array_test.cc:
##########
@@ -640,4 +649,287 @@ TEST_F(TestExtensionType, GetTensor) {
       exact_ext_type->MakeTensor(ext_scalar));
 }
 
+class TestVariableShapeTensorType : public ::testing::Test {
+ public:
+  void SetUp() override {
+    ndim_ = 3;
+    value_type_ = int64();
+    data_type_ = list(value_type_);
+    shape_type_ = fixed_size_list(int32(), ndim_);
+    permutation_ = {0, 1, 2};
+    dim_names_ = {"x", "y", "z"};
+    uniform_shape_ = {std::nullopt, std::optional<int64_t>(1), std::nullopt};
+    ext_type_ = internal::checked_pointer_cast<ExtensionType>(variable_shape_tensor(
+        value_type_, ndim_, permutation_, dim_names_, uniform_shape_));
+    values_ = {0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11, 12, 13, 14, 15, 16, 17,
+               18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35};
+    shapes_ = ArrayFromJSON(fixed_size_list(int32(), ndim_), "[[2,1,3],[2,1,2],[3,1,3]]");
+    data_ = ArrayFromJSON(list(value_type_),
+                          "[[0,1,2,3,4,5],[6,7,8,9],[10,11,12,13,14,15,16,17,18]]");
+    serialized_ =
+        R"({"permutation":[0,1,2],"dim_names":["x","y","z"],"uniform_shape":[null,1,null]})";
+    storage_arr_ = ArrayFromJSON(
+        ext_type_->storage_type(),
+        R"([[[2,3,1],[0,1,2,3,4,5]],[[1,2,2],[6,7,8,9]],[[3,1,3],[10,11,12,13,14,15,16,17,18]]])");
+    ext_arr_ = internal::checked_pointer_cast<ExtensionArray>(
+        ExtensionType::WrapArray(ext_type_, storage_arr_));
+  }
+
+ protected:
+  int32_t ndim_;
+  std::shared_ptr<DataType> value_type_;
+  std::shared_ptr<DataType> data_type_;
+  std::shared_ptr<DataType> shape_type_;
+  std::vector<int64_t> permutation_;
+  std::vector<std::optional<int64_t>> uniform_shape_;
+  std::vector<std::string> dim_names_;
+  std::shared_ptr<ExtensionType> ext_type_;
+  std::vector<int64_t> values_;
+  std::shared_ptr<Array> shapes_;
+  std::shared_ptr<Array> data_;
+  std::string serialized_;
+  std::shared_ptr<Array> storage_arr_;
+  std::shared_ptr<ExtensionArray> ext_arr_;
+};
+
+TEST_F(TestVariableShapeTensorType, CheckDummyRegistration) {
+  // We need a registered dummy type at runtime to allow for IPC deserialization
+  auto registered_type = GetExtensionType("arrow.variable_shape_tensor");
+  ASSERT_TRUE(registered_type->type_id == Type::EXTENSION);

Review Comment:
   ```suggestion
     ASSERT_EQ(registered_type->type_id, Type::EXTENSION);
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {

Review Comment:
   Nit: ensure constness
   ```suggestion
       for (const auto& x : document["permutation"].GetArray()) {
   ```



##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& permutation,
+                             std::vector<int64_t>* strides) {
+  const auto fixed_width_type =
+      internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);
+  }
+  const int byte_width = value_type->byte_width();
+
+  int64_t remaining = 0;
+  if (!shape.empty() && shape.front() > 0) {

Review Comment:
   By the way, this is assuming that `permutation[0] == 0`, right? Can we add a DCHECK for this?



##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& permutation,
+                             std::vector<int64_t>* strides) {
+  const auto fixed_width_type =
+      internal::checked_pointer_cast<FixedWidthType>(value_type);
+  if (permutation.empty()) {
+    return internal::ComputeRowMajorStrides(*fixed_width_type.get(), shape, strides);
+  }
+  const int byte_width = value_type->byte_width();
+
+  int64_t remaining = 0;
+  if (!shape.empty() && shape.front() > 0) {
+    remaining = byte_width;
+    for (auto i : permutation) {
+      if (i > 0) {
+        if (internal::MultiplyWithOverflow(remaining, shape[i], &remaining)) {
+          return Status::Invalid(
+              "Strides computed from shape would not fit in 64-bit integer");
+        }
+      }
+    }
+  }
+
+  if (remaining == 0) {
+    strides->assign(shape.size(), byte_width);
+    return Status::OK();
+  }
+
+  strides->push_back(remaining);
+  for (auto i : permutation) {
+    if (i > 0) {
+      remaining /= shape[i];

Review Comment:
   This is correct and is also existing code that is being moved around, but it seems that maybe this could be simplified by walking the permutation from back to front.
   
   e.g. start with `strides[n-1] = byte_width`, then `strides[n-2] = byte_width * shape[permutation[n-1]]`, etc.
   
   If this reasoning is correct, perhaps create an issue for it as a minor improvement?



##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -19,14 +19,19 @@
 
 #include <cstdint>
 #include <vector>
+#include "arrow/array/array_nested.h"
+#include "arrow/tensor.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/sort.h"
 
 #include "arrow/status.h"
 #include "arrow/util/print.h"
 
 namespace arrow::internal {
 
 ARROW_EXPORT
-Status IsPermutationValid(const std::vector<int64_t>& permutation) {
+inline Status IsPermutationValid(const std::vector<int64_t>& permutation) {

Review Comment:
   Is it so performance-critical that it deserves to remain defined in this `.h`? I think we could move the definition to a corresponding `tensor_internal.cc` file instead.



##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -42,4 +47,46 @@ Status IsPermutationValid(const std::vector<int64_t>& permutation) {
   return Status::OK();
 }
 
+ARROW_EXPORT
+inline Status ComputeStrides(const std::shared_ptr<DataType>& value_type,

Review Comment:
   Also, can return `Result<std::vector<int64_t>>` instead of taking an additional pointer argument for the output value.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+    ARROW_ASSIGN_OR_RAISE(const auto size, shape_array->GetScalar(j));
+    auto size_value = internal::checked_pointer_cast<Int32Scalar>(size)->value;
+    if (size_value < 0) {
+      return Status::Invalid("shape must have non-negative values");
+    }
+    shape.push_back(std::move(size_value));
+  }
+  internal::Permute<int64_t>(permutation, &shape);
+
+  std::vector<std::string> dim_names = ext_type->dim_names();
+  if (!dim_names.empty()) {
+    internal::Permute<std::string>(permutation, &dim_names);
+  }
+
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(internal::ComputeStrides(value_type, shape, permutation, &strides));
+
+  const auto byte_width = value_type->byte_width();
+  const auto start_position = data_array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  ARROW_ASSIGN_OR_RAISE(
+      const auto buffer,
+      SliceBufferSafe(data_array->data()->buffers[1], start_position, size * byte_width));
+
+  return Tensor::Make(value_type, std::move(buffer), std::move(shape), std::move(strides),
+                      ext_type->dim_names());
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Make(
+    const std::shared_ptr<DataType>& value_type, const int32_t ndim,
+    const std::vector<int64_t>& permutation, const std::vector<std::string>& dim_names,
+    const std::vector<std::optional<int64_t>>& uniform_shape) {
+  if (!dim_names.empty() && dim_names.size() != static_cast<size_t>(ndim)) {
+    return Status::Invalid("dim_names size must match ndim. Expected: ", ndim,
+                           " Got: ", dim_names.size());
+  }
+  if (!uniform_shape.empty() && uniform_shape.size() != static_cast<size_t>(ndim)) {
+    return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                           " Got: ", uniform_shape.size());
+  }
+  if (!uniform_shape.empty()) {
+    for (const auto& v : uniform_shape) {
+      if (v.has_value() && v.value() < 0) {
+        return Status::Invalid("uniform_shape must have non-negative values");
+      }
+    }
+  }
+  if (!permutation.empty()) {
+    if (permutation.size() != static_cast<size_t>(ndim)) {
+      return Status::Invalid("permutation size must match ndim. Expected: ", ndim,
+                             " Got: ", permutation.size());
+    }
+
+    std::vector<int64_t> sorted_permutation = permutation;

Review Comment:
   Isn't there a utility function that checks a permutation's validity already? Why rewrite it here?



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type, const int32_t ndim,
+                          const std::vector<int64_t> permutation = {},
+                          const std::vector<std::string> dim_names = {},
+                          const std::vector<std::optional<int64_t>> uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(int32(), ndim)),
+                               ::arrow::field("data", list(value_type))})),
+        value_type_(value_type),
+        ndim_(std::move(ndim)),
+        permutation_(std::move(permutation)),
+        dim_names_(std::move(dim_names)),
+        uniform_shape_(std::move(uniform_shape)) {}
+
+  std::string extension_name() const override { return "arrow.variable_shape_tensor"; }
+  std::string ToString(bool show_metadata = false) const override;
+
+  /// Number of dimensions of tensor elements
+  int32_t ndim() const { return ndim_; }
+
+  /// Value type of tensor elements
+  const std::shared_ptr<DataType>& value_type() const { return value_type_; }
+
+  /// Permutation mapping from logical to physical memory layout of tensor elements
+  const std::vector<int64_t>& permutation() const { return permutation_; }
+
+  /// Dimension names of tensor elements. Dimensions are ordered physically.
+  const std::vector<std::string>& dim_names() const { return dim_names_; }
+
+  /// Shape of uniform dimensions.
+  const std::vector<std::optional<int64_t>>& uniform_shape() const {
+    return uniform_shape_;
+  }
+
+  bool ExtensionEquals(const ExtensionType& other) const override;
+
+  std::string Serialize() const override;
+
+  Result<std::shared_ptr<DataType>> Deserialize(
+      std::shared_ptr<DataType> storage_type,
+      const std::string& serialized_data) const override;
+
+  /// Create a VariableShapeTensorArray from ArrayData
+  std::shared_ptr<Array> MakeArray(std::shared_ptr<ArrayData> data) const override;
+
+  /// \brief Convert an ExtensionScalar to a Tensor
+  ///
+  /// This method will return a Tensor from ExtensionScalar with strides derived
+  /// from shape and permutation stored. Shape and dim_names will be permuted
+  /// according to permutation stored in the VariableShapeTensorType.
+  static Result<std::shared_ptr<Tensor>> MakeTensor(
+      const std::shared_ptr<ExtensionScalar>&);
+
+  /// \brief Create a VariableShapeTensorType instance
+  static Result<std::shared_ptr<DataType>> Make(
+      const std::shared_ptr<DataType>& value_type, const int32_t ndim,
+      const std::vector<int64_t>& permutation = {},
+      const std::vector<std::string>& dim_names = {},
+      const std::vector<std::optional<int64_t>>& uniform_shape = {});

Review Comment:
   The parameters passed by value in the constructor can probably be passed by value here as well?
   For example:
   ```suggestion
     static Result<std::shared_ptr<DataType>> Make(
         const std::shared_ptr<DataType>& value_type, const int32_t ndim,
         const std::vector<int64_t> permutation = {},
         const std::vector<std::string> dim_names = {},
         const std::vector<std::optional<int64_t>> uniform_shape = {});
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"

Review Comment:
   You should also include the required standard includes, such as:
   ```suggestion
   #include <cstdint>
   #include <memory>
   #include <optional>
   #include <string>
   #include <vector>
   
   #include "arrow/extension_type.h"
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();

Review Comment:
   ```suggestion
     const int32_t ndim =
         checked_cast<const FixedSizeListType&>(*storage_type->field(0)->type()).list_size();
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {

Review Comment:
   See comment wrt. struct field order in the spec.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {

Review Comment:
   Can presize here.
   ```suggestion
       permutation.reserve(ndim);
       for (auto& x : document["permutation"].GetArray()) {
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {

Review Comment:
   ```suggestion
       dim_names.reserve(ndim);
       for (const auto& x : document["dim_names"].GetArray()) {
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);

Review Comment:
   Should first check scalar validity.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);

Review Comment:
   ```suggestion
     return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), std::move(permutation),
                                  std::move(dim_names), std::move(uniform_shape));
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));

Review Comment:
   See comment wrt. struct field order in the spec.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }

Review Comment:
   We already know that the Scalar's type is VariableShapeTensorType, so we shouldn't need to do any more type checks here?
   Please move extension type validation inside `VariableShapeTensorType::Make`.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }

Review Comment:
   ```suggestion
       permutation.resize(ndim);
       std::iota(permutation.begin(), permutation.end(), 0);
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");

Review Comment:
   Nit
   ```suggestion
       return Status::Invalid("Cannot convert data with nulls to Tensor.");
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);

Review Comment:
   This is not used below?



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;

Review Comment:
   Should first check scalar validity.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+    ARROW_ASSIGN_OR_RAISE(const auto size, shape_array->GetScalar(j));

Review Comment:
   Scalar wrapping through `GetScalar` is expensive and not useful here. You should just call `shape_array->Value(i)` instead.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+    ARROW_ASSIGN_OR_RAISE(const auto size, shape_array->GetScalar(j));
+    auto size_value = internal::checked_pointer_cast<Int32Scalar>(size)->value;
+    if (size_value < 0) {
+      return Status::Invalid("shape must have non-negative values");
+    }
+    shape.push_back(std::move(size_value));
+  }
+  internal::Permute<int64_t>(permutation, &shape);
+
+  std::vector<std::string> dim_names = ext_type->dim_names();
+  if (!dim_names.empty()) {
+    internal::Permute<std::string>(permutation, &dim_names);
+  }
+
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(internal::ComputeStrides(value_type, shape, permutation, &strides));

Review Comment:
   Why abort instead of propagating the error here?



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;

Review Comment:
   ```suggestion
     std::vector<int64_t> shape;
     shape.reserve(ndim);
   ```



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,

Review Comment:
   Please use `VariableShapeTensorType::Make` so that we can get actual error propagation instead of abort.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() != ndim) {
+      return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
+                             " Got: ", uniform_shape.size());
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<int32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            internal::checked_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::MakeTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) {
+  const auto ext_scalar = internal::checked_pointer_cast<ExtensionScalar>(scalar);
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto ext_type =
+      internal::checked_pointer_cast<VariableShapeTensorType>(scalar->type);
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data_scalar, tensor_scalar->field(1));
+  const auto shape_array = internal::checked_pointer_cast<Int32Array>(
+      internal::checked_pointer_cast<FixedSizeListScalar>(shape_scalar)->value);
+  const auto data_array =
+      internal::checked_pointer_cast<BaseListScalar>(data_scalar)->value;
+
+  const auto value_type =
+      internal::checked_pointer_cast<FixedWidthType>(ext_type->value_type());
+
+  if (!is_fixed_width(*value_type)) {
+    return Status::Invalid("Cannot convert non-fixed-width values to Tensor.");
+  }
+  if (data_array->null_count() > 0) {
+    return Status::Invalid("Cannot convert data with nulls values to Tensor.");
+  }
+
+  auto permutation = ext_type->permutation();
+  if (permutation.empty()) {
+    for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+      permutation.emplace_back(j);
+    }
+  }
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < static_cast<int64_t>(ext_type->ndim()); ++j) {
+    ARROW_ASSIGN_OR_RAISE(const auto size, shape_array->GetScalar(j));
+    auto size_value = internal::checked_pointer_cast<Int32Scalar>(size)->value;
+    if (size_value < 0) {
+      return Status::Invalid("shape must have non-negative values");
+    }
+    shape.push_back(std::move(size_value));
+  }
+  internal::Permute<int64_t>(permutation, &shape);
+
+  std::vector<std::string> dim_names = ext_type->dim_names();
+  if (!dim_names.empty()) {
+    internal::Permute<std::string>(permutation, &dim_names);
+  }
+
+  std::vector<int64_t> strides;
+  ARROW_CHECK_OK(internal::ComputeStrides(value_type, shape, permutation, &strides));
+
+  const auto byte_width = value_type->byte_width();
+  const auto start_position = data_array->offset() * byte_width;
+  const auto size = std::accumulate(shape.begin(), shape.end(), static_cast<int64_t>(1),
+                                    std::multiplies<>());
+
+  ARROW_ASSIGN_OR_RAISE(
+      const auto buffer,
+      SliceBufferSafe(data_array->data()->buffers[1], start_position, size * byte_width));

Review Comment:
   Should first check that data_array's length matches the shape.



##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,341 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <set>
+#include <sstream>
+
+#include "arrow/extension/tensor_internal.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/print.h"
+#include "arrow/util/sort.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::ToString(bool show_metadata) const {
+  std::stringstream ss;
+  ss << "extension<" << this->extension_name()
+     << "[value_type=" << value_type_->ToString() << ", ndim=" << ndim_;
+
+  if (!permutation_.empty()) {
+    ss << ", permutation=" << ::arrow::internal::PrintVector{permutation_, ","};
+  }
+  if (!dim_names_.empty()) {
+    ss << ", dim_names=[" << internal::JoinStrings(dim_names_, ",") << "]";
+  }
+  if (!uniform_shape_.empty()) {
+    std::vector<std::string> uniform_shape;
+    for (const auto& v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.emplace_back(std::to_string(v.value()));
+      } else {
+        uniform_shape.emplace_back("null");
+      }
+    }
+    ss << ", uniform_shape=[" << internal::JoinStrings(uniform_shape, ",") << "]";
+  }
+  ss << "]>";
+  return ss.str();
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+
+  if (storage_type->num_fields() != 2) {
+    return Status::Invalid("Expected Struct storage type with 2 fields, got ",
+                           storage_type->num_fields());
+  }
+  if (storage_type->field(0)->type()->id() != Type::FIXED_SIZE_LIST) {
+    return Status::Invalid("Expected FixedSizeList storage type, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+  if (storage_type->field(1)->type()->id() != Type::LIST) {
+    return Status::Invalid("Expected List storage type, got ",
+                           storage_type->field(1)->type()->ToString());
+  }
+  if (std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->value_type() != int32()) {
+    return Status::Invalid("Expected FixedSizeList value type int32, got ",
+                           storage_type->field(0)->type()->ToString());
+  }
+
+  const auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {

Review Comment:
   It's nice that you're doing these validity checks, but it seems you're also doing them in `VariableShapeTensorType::Make`, so perhaps just rely on `VariableShapeTensorType::Make`?



##########
cpp/src/arrow/extension/tensor_extension_array_test.cc:
##########
@@ -165,47 +170,51 @@ TEST_F(TestExtensionType, CreateFromArray) {
   ASSERT_EQ(ext_arr->null_count(), 0);
 }
 
+template <typename T>
 void CheckSerializationRoundtrip(const std::shared_ptr<DataType>& ext_type) {
-  auto fst_type = internal::checked_pointer_cast<FixedShapeTensorType>(ext_type);
-  auto serialized = fst_type->Serialize();
+  auto type = internal::checked_pointer_cast<T>(ext_type);

Review Comment:
   Is this required? You're only calling methods on `ExtensionType`, so this cast should be sufficient:
   ```suggestion
     auto type = internal::checked_pointer_cast<ExtensionType>(ext_type);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406527448


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')
+
+    def to_tensor(self):
+        """
+        Convert variable shape tensor extension scalar to a pyarrow.Tensor.
+        """
+        cdef:
+            shared_ptr[CTensor] ctensor
+            vector[int64_t] strides
+            vector[c_string] dim_names
+
+            shared_ptr[CVariableShapeTensorType] typ = static_pointer_cast[CVariableShapeTensorType, CDataType](
+                self.wrapped.get().type)
+
+            shared_ptr[CDataType] ty = typ.get().value_type()
+            # TODO: this accesses the full buffer instead of a slice
+            shared_ptr[CBuffer] data = pyarrow_unwrap_buffer(self.value[1].values.buffers()[1])

Review Comment:
   Why not define a `GetTensor` operating on a scalar?
   ```c++
   class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
    public:
      Result<std::shared_ptr<Tensor>> GetTensor(const ExtensionScalar&);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1406515780


##########
python/pyarrow/scalar.pxi:
##########
@@ -1027,6 +1027,53 @@ cdef class ExtensionScalar(Scalar):
         return pyarrow_wrap_scalar(<shared_ptr[CScalar]> sp_scalar)
 
 
+cdef class VariableShapeTensorScalar(ExtensionScalar):
+    """
+    Concrete class for variable shape tensor extension scalar.
+    """
+
+    def to_numpy_ndarray(self):
+        # TODO: allow any permutation
+        """
+        Convert variable shape tensor extension scalar to a numpy array.
+        Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
+        """
+
+        if self.type.permutation is None or self.type.permutation == list(range(len(self.type.permutation))):
+            shape = self.value[0].values.to_pylist()
+            np_flat = np.asarray(self.value[1].values)
+            return np_flat.reshape(shape)
+        else:
+            raise ValueError(
+                'Only non-permuted tensors can be converted to numpy tensors.')

Review Comment:
   I don't know if this is only temporary, but https://numpy.org/doc/stable/reference/generated/numpy.lib.stride_tricks.as_strided.html should allow you to convert arbitrary permutations AFAICT.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1412725410


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):
+        """
+        Convert variable shape tensor extension array to list of numpy arrays.
+        """
+        tensors = []
+        for i in range(len(self.storage)):
+            tensors.append(self.get_tensor(i).to_numpy())
+
+        return tensors
+
+    def get_tensor(self, int64_t i):
+        """
+        Get i-th tensor from variable shape tensor extension array.
+
+        Parameters
+        ----------
+        i : int64_t
+            The index of the tensor to get.
+
+        Returns
+        -------
+        tensor : pyarrow.Tensor
+        """
+        cdef:
+            CVariableShapeTensorArray* ext_array = <CVariableShapeTensorArray*>(self.ap)
+            CResult[shared_ptr[CTensor]] ctensor
+        with nogil:
+            ctensor = ext_array.GetTensor(i)
+        return pyarrow_wrap_tensor(GetResultValue(ctensor))
+
+    @staticmethod
+    def from_numpy_ndarray(obj):
+        """
+        Convert a list of numpy arrays ndarrays to a variable shape tensor extension array.
+        The length of the list will become the length of the variable shape tensor array.
+
+        Numpy arrays needs to be C-contiguous in memory (``obj.flags["C_CONTIGUOUS"]==True``).
+
+        Parameters
+        ----------
+        obj : list(numpy.ndarray)
+
+        Examples
+        --------
+        >>> import pyarrow as pa
+        >>> import numpy as np
+        >>> ndarray_list = [
+        ...         np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+        ...         np.array([[7, 8]], dtype=np.float32),
+        ...     ]
+        >>> pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list)
+        <pyarrow.lib.VariableShapeTensorArray object at ...>
+        -- is_valid: all not null
+        -- child 0 type: fixed_size_list<item: uint32>[2]
+          [
+            [
+              2,
+              3
+            ],
+            [
+              1,
+              2
+            ]
+          ]
+        -- child 1 type: list<item: float>
+          [
+            [
+              1,
+              2,
+              3,
+              4,
+              5,
+              6
+            ],
+            [
+              7,
+              8
+            ]
+          ]
+        """
+        if not all([o.flags["C_CONTIGUOUS"] for o in obj]):
+            raise ValueError('The data in the numpy arrays need to be in a single, '
+                             'C-style contiguous segment.')

Review Comment:
   Agreed. I'll look into stride testing and this next week.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409380255


##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -2644,6 +2649,35 @@ cdef extern from "arrow/extension_type.h" namespace "arrow":
         shared_ptr[CArray] storage()
 
 
+cdef extern from "arrow/extension/variable_shape_tensor.h" namespace "arrow::extension":
+    cdef cppclass CVariableShapeTensorType \
+            " arrow::extension::VariableShapeTensorType"(CExtensionType):
+
+        CResult[shared_ptr[CTensor]] GetTensor(const shared_ptr[CExtensionScalar]& scalar) const
+
+        @staticmethod
+        CResult[shared_ptr[CDataType]] Make(const shared_ptr[CDataType]& value_type,
+                                            const uint32_t ndim,
+                                            const vector[int64_t]& permutation,
+                                            const vector[c_string]& dim_names,
+                                            const vector[optional[int64_t]]& uniform_shape)
+
+        CResult[shared_ptr[CDataType]] Deserialize(const shared_ptr[CDataType] storage_type,
+                                                   const c_string& serialized_data) const
+
+        c_string Serialize() const

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409320856


##########
cpp/src/arrow/extension/variable_shape_tensor.cc:
##########
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <sstream>
+
+#include "arrow/extension/fixed_shape_tensor.h"
+#include "arrow/extension/variable_shape_tensor.h"
+
+#include "arrow/array/array_nested.h"
+#include "arrow/array/array_primitive.h"
+#include "arrow/json/rapidjson_defs.h"  // IWYU pragma: keep
+#include "arrow/scalar.h"
+#include "arrow/tensor.h"
+#include "arrow/util/int_util_overflow.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/sort.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+namespace rj = arrow::rapidjson;
+
+namespace arrow {
+namespace extension {
+
+const Result<std::shared_ptr<Tensor>> VariableShapeTensorArray::GetTensor(
+    const int64_t i) const {
+  auto ext_arr = internal::checked_pointer_cast<StructArray>(this->storage());
+  auto ext_type = internal::checked_pointer_cast<VariableShapeTensorType>(this->type());
+  auto value_type = ext_type->value_type();
+  auto ndim = ext_type->ndim();
+  auto dim_names = ext_type->dim_names();
+  auto shapes =
+      std::static_pointer_cast<FixedSizeListArray>(ext_arr->field(0))->value_slice(i);
+
+  std::vector<int64_t> shape;
+  for (int64_t j = 0; j < ndim; ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shapes->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));
+  }
+
+  std::vector<int64_t> strides;
+  // TODO: optimize ComputeStrides for non-uniform tensors
+  ARROW_CHECK_OK(
+      internal::ComputeStrides(value_type, shape, ext_type->permutation(), &strides));
+
+  auto list_arr =
+      std::static_pointer_cast<ListArray>(ext_arr->field(1))->value_slice(i)->data();
+  auto byte_width = value_type->byte_width();
+  auto buffer = SliceBuffer(list_arr->buffers[1], list_arr->offset * byte_width,
+                            list_arr->length * byte_width);
+
+  return Tensor::Make(value_type, buffer, shape, strides, dim_names);
+}
+
+bool VariableShapeTensorType::ExtensionEquals(const ExtensionType& other) const {
+  if (extension_name() != other.extension_name()) {
+    return false;
+  }
+  const auto& other_ext = static_cast<const VariableShapeTensorType&>(other);
+  if (this->ndim() != other_ext.ndim()) {
+    return false;
+  }
+
+  auto is_permutation_trivial = [](const std::vector<int64_t>& permutation) {
+    for (size_t i = 1; i < permutation.size(); ++i) {
+      if (permutation[i - 1] + 1 != permutation[i]) {
+        return false;
+      }
+    }
+    return true;
+  };
+  const bool permutation_equivalent =
+      ((permutation_ == other_ext.permutation()) ||
+       (permutation_.empty() && is_permutation_trivial(other_ext.permutation())) ||
+       (is_permutation_trivial(permutation_) && other_ext.permutation().empty()));
+
+  return (storage_type()->Equals(other_ext.storage_type())) &&
+         (dim_names_ == other_ext.dim_names()) &&
+         (uniform_shape_ == other_ext.uniform_shape()) && permutation_equivalent;
+}
+
+std::string VariableShapeTensorType::Serialize() const {
+  rj::Document document;
+  document.SetObject();
+  rj::Document::AllocatorType& allocator = document.GetAllocator();
+
+  if (!permutation_.empty()) {
+    rj::Value permutation(rj::kArrayType);
+    for (auto v : permutation_) {
+      permutation.PushBack(v, allocator);
+    }
+    document.AddMember(rj::Value("permutation", allocator), permutation, allocator);
+  }
+
+  if (!dim_names_.empty()) {
+    rj::Value dim_names(rj::kArrayType);
+    for (std::string v : dim_names_) {
+      dim_names.PushBack(rj::Value{}.SetString(v.c_str(), allocator), allocator);
+    }
+    document.AddMember(rj::Value("dim_names", allocator), dim_names, allocator);
+  }
+
+  if (!uniform_shape_.empty()) {
+    rj::Value uniform_shape(rj::kArrayType);
+    for (auto v : uniform_shape_) {
+      if (v.has_value()) {
+        uniform_shape.PushBack(v.value(), allocator);
+      } else {
+        uniform_shape.PushBack(rj::Value{}.SetNull(), allocator);
+      }
+    }
+    document.AddMember(rj::Value("uniform_shape", allocator), uniform_shape, allocator);
+  }
+
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  document.Accept(writer);
+  return buffer.GetString();
+}
+
+Result<std::shared_ptr<DataType>> VariableShapeTensorType::Deserialize(
+    std::shared_ptr<DataType> storage_type, const std::string& serialized_data) const {
+  if (storage_type->id() != Type::STRUCT) {
+    return Status::Invalid("Expected Struct storage type, got ",
+                           storage_type->ToString());
+  }
+  auto value_type = storage_type->field(1)->type()->field(0)->type();
+  const size_t ndim =
+      std::static_pointer_cast<FixedSizeListType>(storage_type->field(0)->type())
+          ->list_size();
+
+  rj::Document document;
+  if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+    return Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+  }
+
+  std::vector<int64_t> permutation;
+  if (document.HasMember("permutation")) {
+    for (auto& x : document["permutation"].GetArray()) {
+      permutation.emplace_back(x.GetInt64());
+    }
+    if (permutation.size() != ndim) {
+      return Status::Invalid("Invalid permutation");
+    }
+  }
+  std::vector<std::string> dim_names;
+  if (document.HasMember("dim_names")) {
+    for (auto& x : document["dim_names"].GetArray()) {
+      dim_names.emplace_back(x.GetString());
+    }
+    if (dim_names.size() != ndim) {
+      return Status::Invalid("Invalid dim_names");
+    }
+  }
+
+  std::vector<std::optional<int64_t>> uniform_shape;
+  if (document.HasMember("uniform_shape")) {
+    for (auto& x : document["uniform_shape"].GetArray()) {
+      if (x.IsNull()) {
+        uniform_shape.emplace_back(std::nullopt);
+      } else {
+        uniform_shape.emplace_back(x.GetInt64());
+      }
+    }
+    if (uniform_shape.size() > ndim) {
+      return Status::Invalid("Invalid uniform_shape");
+    }
+  }
+
+  return variable_shape_tensor(value_type, static_cast<uint32_t>(ndim), permutation,
+                               dim_names, uniform_shape);
+}
+
+std::shared_ptr<Array> VariableShapeTensorType::MakeArray(
+    std::shared_ptr<ArrayData> data) const {
+  DCHECK_EQ(data->type->id(), Type::EXTENSION);
+  DCHECK_EQ("arrow.variable_shape_tensor",
+            static_cast<const ExtensionType&>(*data->type).extension_name());
+  return std::make_shared<ExtensionArray>(data);
+}
+
+Result<std::shared_ptr<Tensor>> VariableShapeTensorType::GetTensor(
+    const std::shared_ptr<ExtensionScalar>& scalar) const {
+  const auto tensor_scalar = internal::checked_pointer_cast<StructScalar>(scalar->value);
+  const auto fields = this->storage_type()->fields();
+
+  ARROW_ASSIGN_OR_RAISE(const auto shape_scalar, tensor_scalar->field(0));
+  ARROW_ASSIGN_OR_RAISE(const auto data, tensor_scalar->field(1));
+  const auto shape_array =
+      std::static_pointer_cast<FixedSizeListScalar>(shape_scalar)->value;
+
+  std::vector<int64_t> shape;
+  for (uint32_t j = 0; j < this->ndim(); ++j) {
+    ARROW_ASSIGN_OR_RAISE(auto size, shape_array->GetScalar(j));
+    shape.push_back(
+        static_cast<int64_t>(std::static_pointer_cast<UInt32Scalar>(size)->value));

Review Comment:
   Changed to:
   ```cpp
     if (!uniform_shape.empty() && uniform_shape.size() != static_cast<size_t>(ndim)) {
       return Status::Invalid("uniform_shape size must match ndim. Expected: ", ndim,
                              " Got: ", uniform_shape.size());
     }
     if (!uniform_shape.empty()) {
       for (const auto& v : uniform_shape) {
         if (v.has_value() && v.value() < 0) {
           return Status::Invalid("uniform_shape must have non-negative values");
         }
       }
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409313225


##########
cpp/src/arrow/extension/fixed_shape_tensor.h:
##########
@@ -15,9 +15,21 @@
 // specific language governing permissions and limitations
 // under the License.
 
+#pragma once
+
 #include "arrow/extension_type.h"
 
 namespace arrow {
+namespace internal {

Review Comment:
   Moved to `arrow/cpp/src/arrow/extension/tensor_internal.h`



##########
cpp/src/arrow/extension/variable_shape_tensor.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/extension_type.h"
+
+namespace arrow {
+namespace extension {
+
+class ARROW_EXPORT VariableShapeTensorArray : public ExtensionArray {
+ public:
+  using ExtensionArray::ExtensionArray;
+
+  /// \brief Get a Tensor of VariableShapeTensorArray at i
+  ///
+  /// This method will return a Tensor from VariableShapeTensorArray with strides
+  /// derived from shape and permutation of VariableShapeTensorType. Shape and
+  /// dim_names will be permuted according to permutation stored in the
+  /// VariableShapeTensorType metadata.
+  const Result<std::shared_ptr<Tensor>> GetTensor(const int64_t i) const;
+};
+
+/// \brief Concrete type class for variable-shape Tensor data.
+/// This is a canonical arrow extension type.
+/// See: https://arrow.apache.org/docs/format/CanonicalExtensions.html
+class ARROW_EXPORT VariableShapeTensorType : public ExtensionType {
+ public:
+  VariableShapeTensorType(const std::shared_ptr<DataType>& value_type,
+                          const uint32_t& ndim,
+                          const std::vector<int64_t>& permutation = {},
+                          const std::vector<std::string>& dim_names = {},
+                          const std::vector<std::optional<int64_t>>& uniform_shape = {})
+      : ExtensionType(struct_({::arrow::field("shape", fixed_size_list(uint32(), ndim)),
+                               ::arrow::field("data", list(value_type))})),
+        value_type_(value_type),
+        permutation_(permutation),
+        dim_names_(dim_names),
+        uniform_shape_(uniform_shape) {}

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1409363670


##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1353,6 +1386,70 @@ def test_tensor_class_methods():
         arr.to_numpy_ndarray()
 
 
+@pytest.mark.parametrize("value_type", (np.int8, np.int32, np.int64, np.float64))
+def test_variable_shape_tensor_class_methods(value_type):
+    ndim = 2
+    shape_type = pa.list_(pa.uint32(), ndim)
+    arrow_type = pa.from_numpy_dtype(value_type)
+    tensor_type = pa.variable_shape_tensor(
+        arrow_type,
+        ndim,
+        dim_names=["H", "W"],
+        permutation=[0, 1],
+        uniform_shape=[None, None],
+    )
+    fields = [pa.field("shape", shape_type), pa.field("data", pa.list_(arrow_type))]
+
+    shapes = pa.array([[2, 3], [2, 1]], shape_type)
+    values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(arrow_type))
+    struct_arr = pa.StructArray.from_arrays([shapes, values], fields=fields)
+    arr = pa.ExtensionArray.from_storage(tensor_type, struct_arr)
+    basic_arr = pa.ExtensionArray.from_storage(
+        pa.variable_shape_tensor(arrow_type, ndim), struct_arr
+    )
+
+    storage = pa.array(
+        [([2, 3], [1, 2, 3, 4, 5, 6]), ([2, 1], [7, 8])], type=pa.struct(fields)
+    )
+    assert pa.ExtensionArray.from_storage(tensor_type, storage).equals(arr)
+
+    assert arr.type == tensor_type
+
+    ndarray_list = [
+        np.array([[1, 2, 3], [4, 5, 6]], dtype=value_type),
+        np.array([[7], [8]], dtype=value_type),
+    ]
+    assert all(zip(x == y for x, y in zip(arr.to_numpy_ndarray(), ndarray_list)))

Review Comment:
   ```python
   >       assert arr.to_numpy_ndarray() == ndarray_list
   E       ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
   ```



##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1353,6 +1386,70 @@ def test_tensor_class_methods():
         arr.to_numpy_ndarray()
 
 
+@pytest.mark.parametrize("value_type", (np.int8, np.int32, np.int64, np.float64))
+def test_variable_shape_tensor_class_methods(value_type):
+    ndim = 2
+    shape_type = pa.list_(pa.uint32(), ndim)
+    arrow_type = pa.from_numpy_dtype(value_type)
+    tensor_type = pa.variable_shape_tensor(
+        arrow_type,
+        ndim,
+        dim_names=["H", "W"],
+        permutation=[0, 1],
+        uniform_shape=[None, None],
+    )
+    fields = [pa.field("shape", shape_type), pa.field("data", pa.list_(arrow_type))]
+
+    shapes = pa.array([[2, 3], [2, 1]], shape_type)
+    values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(arrow_type))
+    struct_arr = pa.StructArray.from_arrays([shapes, values], fields=fields)
+    arr = pa.ExtensionArray.from_storage(tensor_type, struct_arr)
+    basic_arr = pa.ExtensionArray.from_storage(
+        pa.variable_shape_tensor(arrow_type, ndim), struct_arr
+    )
+
+    storage = pa.array(
+        [([2, 3], [1, 2, 3, 4, 5, 6]), ([2, 1], [7, 8])], type=pa.struct(fields)
+    )
+    assert pa.ExtensionArray.from_storage(tensor_type, storage).equals(arr)
+
+    assert arr.type == tensor_type
+
+    ndarray_list = [
+        np.array([[1, 2, 3], [4, 5, 6]], dtype=value_type),
+        np.array([[7], [8]], dtype=value_type),
+    ]
+    assert all(zip(x == y for x, y in zip(arr.to_numpy_ndarray(), ndarray_list)))
+
+    assert pa.VariableShapeTensorArray.from_numpy_ndarray(ndarray_list).equals(
+        basic_arr
+    )
+    assert pa.VariableShapeTensorArray.from_numpy_ndarray(
+        arr.to_numpy_ndarray()
+    ).equals(basic_arr)
+
+    assert arr.to_pylist() == [
+        {"data": [1, 2, 3, 4, 5, 6], "shape": [2, 3]},
+        {"data": [7, 8], "shape": [2, 1]},
+    ]
+
+    expected_0 = np.array([[1, 2, 3], [4, 5, 6]], dtype=value_type)
+    expected_1 = np.array([[7], [8]], dtype=value_type)
+
+    np.testing.assert_array_equal(arr[0].to_tensor().to_numpy(), expected_0)
+    np.testing.assert_array_equal(arr[1].to_tensor().to_numpy(), expected_1)
+
+    np.testing.assert_array_equal(arr[0].to_numpy_ndarray(), expected_0)
+    np.testing.assert_array_equal(arr[1].to_numpy_ndarray(), expected_1)
+
+    assert arr[0].to_tensor().equals(
+        pa.Tensor.from_numpy(expected_0, dim_names=["H", "W"]))
+
+    # TODO: due to wrong offset this would return [[1], [2]] instead of [[7], [8]]

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1414433628


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):

Review Comment:
   Is this actually needed as a method to start with? 
   
   My expectation is that `to_numpy()` should ideally return this, but just as an object-dtype numpy array of numpy arrays, instead of a list of numpy arrays. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1416050030


##########
python/pyarrow/types.pxi:
##########
@@ -1591,6 +1591,114 @@ cdef class ExtensionType(BaseExtensionType):
         return ExtensionScalar
 
 
+cdef class VariableShapeTensorType(BaseExtensionType):
+    """
+    Concrete class for variable shape tensor extension type.
+
+    Examples
+    --------
+    Create an instance of variable shape tensor extension type:
+
+    >>> import pyarrow as pa
+    >>> pa.variable_shape_tensor(pa.int32(), 2)
+    VariableShapeTensorType(extension<arrow.variable_shape_tensor>)
+
+    Create an instance of variable shape tensor extension type with
+    permutation:
+
+    >>> tensor_type = pa.variable_shape_tensor(pa.int8(), 3,
+    ...                                     permutation=[0, 2, 1])
+    >>> tensor_type.permutation
+    [0, 2, 1]
+    """
+
+    cdef void init(self, const shared_ptr[CDataType]& type) except *:
+        BaseExtensionType.init(self, type)
+        self.tensor_ext_type = <const CVariableShapeTensorType*> type.get()
+
+    @property
+    def value_type(self):
+        """
+        Data type of an individual tensor.
+        """
+        return pyarrow_wrap_data_type(self.tensor_ext_type.value_type())
+
+    @property
+    def ndim(self):
+        """
+        Number of dimensions of the tensors.
+        """
+        return self.tensor_ext_type.ndim()
+
+    @property
+    def dim_names(self):
+        """
+        Explicit names of the dimensions.
+        """
+        list_of_bytes = self.tensor_ext_type.dim_names()
+        if len(list_of_bytes) != 0:
+            return [frombytes(x) for x in list_of_bytes]
+        else:
+            return None
+
+    @property
+    def permutation(self):
+        """
+        Indices of the dimensions ordering.
+        """
+        indices = self.tensor_ext_type.permutation()
+        if len(indices) != 0:
+            return indices
+        else:
+            return None
+
+    @property
+    def uniform_shape(self):
+        """
+        Shape over dimensions that are guaranteed to be constant.
+        """
+        cdef:
+            vector[optional[int64_t]] c_uniform_shape = self.tensor_ext_type.uniform_shape()
+            length = c_uniform_shape.size()
+
+        if length == 0:
+            return None
+
+        uniform_shape = []
+        for i in range(length):
+            if c_uniform_shape[i].has_value():
+                uniform_shape.append(c_uniform_shape[i].value())
+            else:
+                uniform_shape.append(None)
+
+        return uniform_shape
+
+    def __arrow_ext_serialize__(self):
+        """
+        Serialized representation of metadata to reconstruct the type object.
+        """
+        return self.tensor_ext_type.Serialize()
+
+    @classmethod
+    def __arrow_ext_deserialize__(self, storage_type, serialized):
+        """
+        Return an VariableShapeTensor type instance from the storage type and serialized
+        metadata.
+        """
+        return self.tensor_ext_type.Deserialize(storage_type, serialized)
+
+    def __arrow_ext_class__(self):
+        return VariableShapeTensorArray
+
+    def __reduce__(self):
+        return variable_shape_tensor, (self.value_type, self.ndim,
+                                       self.dim_names, self.permutation,
+                                       self.uniform_shape)

Review Comment:
   I think this should be removed in favor of a generic solution as a separate PR: see https://github.com/apache/arrow/pull/35933#issuecomment-1841288493



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r1427125716


##########
python/pyarrow/array.pxi:
##########
@@ -3586,6 +3586,156 @@ class FixedShapeTensorArray(ExtensionArray):
         )
 
 
+cdef class VariableShapeTensorArray(ExtensionArray):
+    """
+    Concrete class for variable shape tensor extension arrays.
+
+    Examples
+    --------
+    Define the extension type for tensor array
+
+    >>> import pyarrow as pa
+    >>> tensor_type = pa.variable_shape_tensor(pa.int32(), 2)
+
+    Create an extension array
+
+    >>> shapes = pa.array([[2, 3], [1, 2]], pa.list_(pa.uint32(), 2))
+    >>> values = pa.array([[1, 2, 3, 4, 5, 6], [7, 8]], pa.list_(pa.int32()))
+    >>> arr = pa.StructArray.from_arrays([shapes, values], names=["shape", "data"])
+    >>> pa.ExtensionArray.from_storage(tensor_type, arr)
+    <pyarrow.lib.VariableShapeTensorArray object at ...>
+    -- is_valid: all not null
+    -- child 0 type: fixed_size_list<item: uint32>[2]
+      [
+        [
+          2,
+          3
+        ],
+        [
+          1,
+          2
+        ]
+      ]
+    -- child 1 type: list<item: int32>
+      [
+        [
+          1,
+          2,
+          3,
+          4,
+          5,
+          6
+        ],
+        [
+          7,
+          8
+        ]
+      ]
+    """
+
+    def to_numpy_ndarray(self):

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1983381961

   Ping @pitrou 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++][Python] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-1952427250

   Ping @pitrou :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-2504459723

   @pitrou
   Oh, so github notifications are your todo list? :D


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3937761324

   Thanks for the review @pitrou! I think I addressed all comments and some more things I found. Could you please do another pass so we can merge this while it's fresh in our minds?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2835146701


##########
cpp/src/arrow/extension/tensor_internal.h:
##########
@@ -20,25 +20,17 @@
 #include <cstdint>
 #include <vector>
 
-#include "arrow/status.h"
-#include "arrow/util/print_internal.h"
+#include "arrow/array/array_nested.h"
 
 namespace arrow::internal {
 
-inline Status IsPermutationValid(const std::vector<int64_t>& permutation) {
-  const auto size = static_cast<int64_t>(permutation.size());
-  std::vector<uint8_t> dim_seen(size, 0);
+ARROW_EXPORT
+Status IsPermutationValid(const std::vector<int64_t>& permutation);
 
-  for (const auto p : permutation) {
-    if (p < 0 || p >= size || dim_seen[p] != 0) {
-      return Status::Invalid(
-          "Permutation indices for ", size,
-          " dimensional tensors must be unique and within [0, ", size - 1,
-          "] range. Got: ", ::arrow::internal::PrintVector{permutation, ","});
-    }
-    dim_seen[p] = 1;
-  }
-  return Status::OK();
-}
+ARROW_EXPORT
+Status ComputeStrides(const std::shared_ptr<DataType>& value_type,

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #38008:
URL: https://github.com/apache/arrow/pull/38008#discussion_r2835236051


##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1482,6 +1482,17 @@ def test_tensor_class_methods(np_type_str):
     assert result.to_tensor().shape == (1, 3, 2, 2)
     assert result.to_tensor().strides == (12 * bw, 1 * bw, 6 * bw, 2 * bw)
 
+    tensor_type = pa.fixed_shape_tensor(arrow_type, [2, 2, 3], permutation=[2, 1, 0])

Review Comment:
   I forget what this was about. I think it might just be additional test of the same path, but called from fixed_shape_tensor_here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38008:
URL: https://github.com/apache/arrow/pull/38008#issuecomment-3953466179

   By the way, do you want to expose it in Python soon after? That would allow validating against NumPy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38007: [C++] Add VariableShapeTensor implementation [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok merged PR #38008:
URL: https://github.com/apache/arrow/pull/38008


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org