You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AntoinePrv (via GitHub)" <gi...@apache.org> on 2026/09/01 14:05:49 UTC

[PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

AntoinePrv opened a new pull request, #51122:
URL: https://github.com/apache/arrow/pull/51122

   ### Rationale for this change
   
   ### What changes are included in this PR?
   
   ### Are these changes tested?
   Yes.
   
   ### Are there any user-facing changes?
   Yes, new APIs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3960288580


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +261,234 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ARROW_PREDICT_FALSE(ptr == nullptr)) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (ARROW_PREDICT_FALSE(out.ptr_->version.major != kVersion.major)) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", kVersion.major);
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim < 0)) {
+      return Status::Invalid("Invalid DLPack tensor: ndim must be >= 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().shape == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: shape must be non-null when ndim != 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().strides == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: strides must be non-null when ndim != 0");
+    }

Review Comment:
   @AntoinePrv This Copilot comment seems valid. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3932077157


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +263,217 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ptr == nullptr) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (out.ptr_->version.major != VERSION.major) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    return out;
+  }

Review Comment:
   CppDLTensor::TakeOwnership() only DCHECKs that ndim is non-negative and doesn't validate shape/strides pointers for ndim>0. In release builds a malformed/older-version DLPack tensor (e.g., negative ndim or null strides) can lead to huge spans and crashes when shape()/strides() are used.



##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +263,217 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ptr == nullptr) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (out.ptr_->version.major != VERSION.major) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  std::span<const int64_t> strides() const {
+    return {tensor().strides, static_cast<std::size_t>(ndim())};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+inline std::vector<int64_t> StridesInBytes(std::span<const int64_t> strides,
+                                           int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    out.push_back(s * byte_width);
+  }
+  return out;
+}
+
+Result<std::shared_ptr<Buffer>> ImportBuffer(CppDLTensor&& dl, bool copy) {
+  // DLPack strides are in number of elements, so is the size we compute from them.
+  ARROW_ASSIGN_OR_RAISE(const auto nelements,
+                        internal::ComputeTensorSize(dl.shape(), dl.strides(), 1));
+  const auto nbytes = nelements * dl.byte_width();
+  // DLPack mandates a null data pointer when the tensor holds no element, so there is
+  // neither anything to share nor to copy.

Review Comment:
   ImportBuffer computes `nbytes = nelements * dl.byte_width()` without an overflow check. If `nelements` is large (but still fits in int64), `nbytes` can overflow and lead to constructing a Buffer with an incorrect (wrapped) size, which is unsafe.



##########
python/pyarrow/array.pxi:
##########
@@ -2269,6 +2274,49 @@ cdef class Array(_PandasConvertible):
 
         return pyarrow_wrap_array(array)
 
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct an Array from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional
+            Designates where the resulting Array should reside, in the
+            format returned by :meth:`Array.__dlpack_device__`. When None,
+            the output Array occupies the same device as the source.
+            Default: None.
+        copy : bool, optional
+            Controls duplication behavior. True mandates copying; False
+            prohibits copying and raises ``BufferError`` if unavoidable;
+            None duplicates only when necessary. Default: None.
+
+        Returns
+        -------
+        Array
+            An Array housing the data from the input object, potentially
+            as a copy or view.
+        """
+        version = (DLPACK_VERSION.major, DLPACK_VERSION.minor)
+        pycapsule = x.__dlpack__(max_version=version, dl_device=device, copy=copy)
+        cdef DLManagedTensorVersioned* ptr = <DLManagedTensorVersioned*>PyCapsule_GetPointer(
+            pycapsule, "dltensor_versioned")

Review Comment:
   Array.from_dlpack calls PyCapsule_GetPointer() without first validating that the producer returned a PyCapsule. If __dlpack__ returns a non-capsule object, PyCapsule_GetPointer sets an error which is then overwritten by the custom ValueError, producing a misleading exception type/message.



##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +263,217 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ptr == nullptr) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (out.ptr_->version.major != VERSION.major) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  std::span<const int64_t> strides() const {
+    return {tensor().strides, static_cast<std::size_t>(ndim())};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+inline std::vector<int64_t> StridesInBytes(std::span<const int64_t> strides,
+                                           int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    out.push_back(s * byte_width);
+  }
+  return out;
+}
+
+Result<std::shared_ptr<Buffer>> ImportBuffer(CppDLTensor&& dl, bool copy) {
+  // DLPack strides are in number of elements, so is the size we compute from them.
+  ARROW_ASSIGN_OR_RAISE(const auto nelements,
+                        internal::ComputeTensorSize(dl.shape(), dl.strides(), 1));
+  const auto nbytes = nelements * dl.byte_width();
+  // DLPack mandates a null data pointer when the tensor holds no element, so there is
+  // neither anything to share nor to copy.
+  uint8_t* data =
+      (nbytes == 0) ? nullptr : dl.data_as<uint8_t>() + dl.tensor().byte_offset;
+
+  std::shared_ptr<Buffer> buffer = nullptr;
+  if (nbytes == 0) {
+    // DLPack data pointer may be null on empty tensors
+    buffer = std::make_shared<Buffer>(data, nbytes);
+  } else if (copy) {
+    ARROW_ASSIGN_OR_RAISE(buffer, MutableBuffer::CopyNonOwned(
+                                      {data, nbytes}, default_cpu_memory_manager()));
+  } else {
+    const bool readonly = dl.is_readonly();
+    // Trick to keep DLPack data alive taken from `Buffer::FromVector`.
+    auto deleter = [dl = std::move(dl)](auto* buffer) { delete buffer; };
+    if (readonly) {
+      buffer = {new Buffer{data, nbytes}, std::move(deleter)};
+    } else {
+      buffer = std::shared_ptr<MutableBuffer>{
+          new MutableBuffer{data, nbytes},
+          std::move(deleter),
+      };
+    }
+  }
+
+  return buffer;
+}
+
+}  // namespace
+
+Result<std::shared_ptr<Array>> ImportArrayVersioned(DLManagedTensorVersioned* unmanaged,
+                                                    bool copy) {
+  ARROW_ASSIGN_OR_RAISE(auto dl, CppDLTensor::TakeOwnership(unmanaged));
+
+  if (dl.tensor().device.device_type != kDLCPU) {
+    return Status::NotImplemented(
+        "DLPack support is implemented only for buffers on CPU device.");
+  }
+
+  if (dl.ndim() != 1 || dl.strides().front() != 1) {
+    return Status::Invalid(
+        "Only contiguous one dimensional tensor can be imported as arrays."
+        " Try importing to Tensor first.");
+  }

Review Comment:
   ImportArrayVersioned returns Status::Invalid for unsupported ndim/stride cases, but the tests (and Python API) expect a NotImplemented error for these unsupported DLPack tensor layouts. Returning Invalid will surface as ArrowInvalid/ValueError and will fail dlpack_test.cc expectations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on PR #51122:
URL: https://github.com/apache/arrow/pull/51122#issuecomment-5529511300

   @AlenkaF @rok @pitrou this is looking good


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3927039635


##########
cpp/src/arrow/tensor.cc:
##########
@@ -109,13 +109,40 @@ Status ComputeColumnMajorStrides(const FixedWidthType& type,
   return Status::OK();
 }
 
-}  // namespace internal
+Result<int64_t> ComputeTensorSize(std::span<const int64_t> shape,
+                                  std::span<const int64_t> strides, int64_t elem_size) {
+  // Check the largest offset can be computed without overflow
+  const size_t ndim = shape.size();
+  int64_t largest_offset = elem_size;
+  for (size_t i = 0; i < ndim; ++i) {
+    if (shape[i] == 0) continue;
+    if (strides[i] < 0) {
+      // TODO(mrkn): Support negative strides for sharing views
+      return Status::Invalid("negative strides not supported");
+    }
 
-namespace {
+    int64_t dim_offset = 0;
+    if (!internal::MultiplyWithOverflow(shape[i] - 1, strides[i], &dim_offset)) {
+      if (!internal::AddWithOverflow(largest_offset, dim_offset, &largest_offset)) {
+        continue;
+      }
+    }
 
-inline bool IsTensorStridesRowMajor(const std::shared_ptr<DataType>& type,
-                                    const std::vector<int64_t>& shape,
-                                    const std::vector<int64_t>& strides) {
+    return Status::Invalid(
+        "offsets computed from shape and strides would not fit in 64-bit integer");
+  }
+
+  // A dimension with no element means empty for which the preceding does not apply.
+  if (std::find(shape.begin(), shape.end(), 0) != shape.end()) {
+    return 0;
+  }
+  return largest_offset;
+}
+
+namespace {
+bool IsTensorStridesRowMajor(const std::shared_ptr<DataType>& type,
+                             const std::vector<int64_t>& shape,
+                             const std::vector<int64_t>& strides) {
   std::vector<int64_t> c_strides;

Review Comment:
   `IsTensorStridesRowMajor` / `IsTensorStridesColumnMajor` are defined inside an anonymous namespace nested under `arrow::internal` (starting at `namespace {`), but later calls use `internal::IsTensorStridesRowMajor/ColumnMajor` and `IsTensorStridesContiguous()` (in `arrow::internal`) calls them unqualified. Those lookups won’t find the nested-anonymous definitions, which should cause compile errors.



##########
cpp/src/arrow/c/dlpack.h:
##########
@@ -105,4 +108,33 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Array>& arr);
 ARROW_EXPORT
 Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t);
 
+/// \brief Import a DLPack tensor as an Arrow Array.
+///
+/// Same restrictions on data types as `ExportArrayVersioned`, and only row-major
+/// tensors are supported. Dimensions beyond the first are imported as nested
+/// fixed size lists.
+/// Takes ownership of the `DLManagedTensorVersioned` though it may point to shared data.

Review Comment:
   The ImportArrayVersioned documentation claims multidimensional tensors are imported as nested fixed-size lists, but the implementation rejects `ndim != 1` (see `ImportArrayVersioned` in `dlpack.cc`). The header comment should match the actual supported behavior.
   
   This issue also appears on line 132 of the same file.



##########
python/pyarrow/tensor.pxi:
##########
@@ -300,7 +302,50 @@ strides: {self.strides}"""
         buffer.strides = <Py_ssize_t *> cp.PyBytes_AsString(self._ssize_t_strides)
         buffer.suboffsets = NULL
 
-    def __dlpack__(self, stream=None, max_version=None, dl_device=None, copy=None):
+    def from_dlpack(x, /, *, device=None, copy=None):

Review Comment:
   `Tensor.from_dlpack` is a factory method but is not marked as `@staticmethod` (unlike `Tensor.from_numpy` and other `from_*` constructors). This makes binding/inspection inconsistent and allows accidental instance binding.



##########
python/pyarrow/array.pxi:
##########
@@ -2269,6 +2274,48 @@ cdef class Array(_PandasConvertible):
 
         return pyarrow_wrap_array(array)
 
+    def from_dlpack(x, /, *, device=None, copy=None):

Review Comment:
   `Array.from_dlpack` is a factory method but is not marked as `@staticmethod` like other `Array.from_*` constructors in this file. Without `@staticmethod`, it can also be (mis)bound as an instance method and the signature shown in docs/introspection can be confusing.



##########
python/pyarrow/tensor.pxi:
##########
@@ -300,7 +302,50 @@ strides: {self.strides}"""
         buffer.strides = <Py_ssize_t *> cp.PyBytes_AsString(self._ssize_t_strides)
         buffer.suboffsets = NULL
 
-    def __dlpack__(self, stream=None, max_version=None, dl_device=None, copy=None):
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct a Tensor from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method) or the array API's
+            ``__array_namespace__`` protocol.

Review Comment:
   The `Tensor.from_dlpack` docstring claims inputs may follow the array API `__array_namespace__` protocol, but the implementation only calls `x.__dlpack__(...)`. Either add support for the array API path or remove this from the docs to avoid misleading users.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3959453291


##########
python/pyarrow/tests/test_dlpack.py:
##########
@@ -374,3 +374,101 @@ def test_dlpack_cuda_not_supported():
     with pytest.raises(NotImplementedError, match="DLPack support is implemented "
                        "only for buffers on CPU device."):
         carr.__dlpack_device__()
+
+
+@requires_numpy_version("2.1.0")
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_tensor_from_dlpack(np_type):
+    def make_array():
+        base = np.arange(24, dtype=np_type).reshape((4, 6))
+        array = base[::2, 1::2]
+        assert not array.flags['C_CONTIGUOUS']
+        return array
+
+    # Non-contiguous, strided slice: DLPack carries explicit strides, so this
+    # should not need a copy on export.
+    tensor = pa.Tensor.from_dlpack(make_array())
+    assert isinstance(tensor, pa.Tensor)
+    gc.collect()  # Attempts to free input array memory
+    np.testing.assert_array_equal(tensor.to_numpy(), make_array(), strict=True)
+
+
+@requires_numpy_version("2.1.0")
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_array_from_dlpack(np_type):
+    expected = np.array([1, 2, 3, 4, 5], dtype=np_type)
+    arr = pa.Array.from_dlpack(expected)
+    arr.validate(full=True)
+    assert isinstance(arr, pa.Array)
+    np.testing.assert_array_equal(arr.to_numpy(), expected, strict=True)
+
+
+@requires_numpy_version("2.1.0")
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_fixed_shape_tensor_array_from_dlpack(np_type):
+    source = np.arange(12, dtype=np_type).reshape((3, 2, 2))
+    arr = pa.FixedShapeTensorArray.from_dlpack(source)
+    arr.validate(full=True)
+    assert arr.type == pa.fixed_shape_tensor(pa.from_numpy_dtype(np_type), [2, 2])
+    assert arr.to_pylist() == [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]]

Review Comment:
   Note that the underlying `FixedShapeTensorArray::FromTensor` predates this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3960288580


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +261,234 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ARROW_PREDICT_FALSE(ptr == nullptr)) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (ARROW_PREDICT_FALSE(out.ptr_->version.major != kVersion.major)) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", kVersion.major);
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim < 0)) {
+      return Status::Invalid("Invalid DLPack tensor: ndim must be >= 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().shape == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: shape must be non-null when ndim != 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().strides == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: strides must be non-null when ndim != 0");
+    }

Review Comment:
   @AntoinePrv This Copilot comment seems valid. Can you address it and make sure it's tested?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3927044410


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +263,217 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ptr == nullptr) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (out.ptr_->version.major != VERSION.major) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  std::span<const int64_t> strides() const {
+    return {tensor().strides, static_cast<std::size_t>(ndim())};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+inline std::vector<int64_t> StridesInBytes(std::span<const int64_t> strides,
+                                           int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    out.push_back(s * byte_width);
+  }
+  return out;
+}
+
+Result<std::shared_ptr<Buffer>> ImportBuffer(CppDLTensor&& dl, bool copy) {
+  // DLPack strides are in number of elements, so is the size we compute from them.
+  ARROW_ASSIGN_OR_RAISE(const auto nelements,
+                        internal::ComputeTensorSize(dl.shape(), dl.strides(), 1));
+  const auto nbytes = nelements * dl.byte_width();
+  // DLPack mandates a null data pointer when the tensor holds no element, so there is
+  // neither anything to share nor to copy.
+  uint8_t* data =
+      (nbytes == 0) ? nullptr : dl.data_as<uint8_t>() + dl.tensor().byte_offset;
+
+  std::shared_ptr<Buffer> buffer = nullptr;
+  if (nbytes == 0) {
+    // DLPack data pointer may be null on empty tensors
+    buffer = std::make_shared<Buffer>(data, nbytes);
+  } else if (copy) {
+    ARROW_ASSIGN_OR_RAISE(buffer, MutableBuffer::CopyNonOwned(
+                                      {data, nbytes}, default_cpu_memory_manager()));
+  } else {
+    const bool readonly = dl.is_readonly();
+    // Trick to keep DLPack data alive taken from `Buffer::FromVector`.

Review Comment:
   A new `Buffer` subclass would be cleaner than this IMHO.



##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -18,22 +18,37 @@
 #include "arrow/c/dlpack.h"
 
 #include <array>
+#include <functional>
 #include <memory>
+#include <numeric>
 #include <type_traits>
 #include <vector>
 
 #include "arrow/array/array_base.h"
+#include "arrow/array/util.h"
 #include "arrow/buffer.h"
 #include "arrow/c/dlpack_abi.h"
 #include "arrow/device.h"
 #include "arrow/tensor.h"
 #include "arrow/type.h"
 #include "arrow/type_traits.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/util/small_vector.h"
 
 namespace arrow::dlpack {
 
+extern const DLPackVersion VERSION = {

Review Comment:
   `kVersion` or `kDLPackVersion` rather than `VERSION`?



##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +261,238 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ARROW_PREDICT_FALSE(ptr == nullptr)) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (ARROW_PREDICT_FALSE(out.ptr_->version.major != VERSION.major)) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim < 0)) {
+      return Status::Invalid("Invalid DLPack tensor: ndim must be >= 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().shape == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: shape must be non-null when ndim != 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().strides == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: strides must be non-null when ndim != 0");
+    }
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  std::span<const int64_t> strides() const {
+    return {tensor().strides, static_cast<std::size_t>(ndim())};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+Result<std::vector<int64_t>> StridesInBytes(std::span<const int64_t> strides,
+                                            int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    int64_t stride_bytes = 0;
+    if (ARROW_PREDICT_FALSE(
+            internal::MultiplyWithOverflow(s, byte_width, &stride_bytes))) {
+      return Status::Invalid("Overflow computing DLPack tensor stride in bytes.");
+    }
+    out.push_back(stride_bytes);
+  }
+  return out;
+}
+
+Result<std::shared_ptr<Buffer>> ImportBuffer(CppDLTensor&& dl, bool copy) {
+  // DLPack strides are in number of elements, so is the size we compute from them.
+  ARROW_ASSIGN_OR_RAISE(const auto nelements,
+                        internal::ComputeTensorSize(dl.shape(), dl.strides(), 1));
+  int64_t nbytes = 0;
+  if (ARROW_PREDICT_FALSE(internal::MultiplyWithOverflow(
+          nelements, static_cast<int64_t>(dl.byte_width()), &nbytes))) {
+    return Status::Invalid("Overflow computing DLPack tensor size in bytes.");
+  }
+  // DLPack mandates a null data pointer when the tensor holds no element, so there is
+  // neither anything to share nor to copy.
+  uint8_t* data =
+      (nbytes == 0) ? nullptr : dl.data_as<uint8_t>() + dl.tensor().byte_offset;
+
+  std::shared_ptr<Buffer> buffer = nullptr;
+  if (nbytes == 0) {
+    // DLPack data pointer may be null on empty tensors
+    buffer = std::make_shared<Buffer>(data, nbytes);
+  } else if (copy) {
+    ARROW_ASSIGN_OR_RAISE(buffer, MutableBuffer::CopyNonOwned(
+                                      {data, nbytes}, default_cpu_memory_manager()));
+  } else {

Review Comment:
   Presumably, a caller setting `copy` wants to get an exclusive mutable copy of the tensor.
   
   Should we then inspect the DLPack flags for [`DLPACK_FLAG_BITMASK_IS_COPIED`](https://dmlc.github.io/dlpack/latest/c_api.html#c.DLPACK_FLAG_BITMASK_IS_COPIED)? In that case, a new copy doesn't need to be made, I think?
   
   (also check that `DLPACK_FLAG_BITMASK_READ_ONLY` is not set?)
   
   



##########
cpp/src/arrow/c/dlpack.h:
##########
@@ -105,4 +108,32 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Array>& arr);
 ARROW_EXPORT
 Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t);
 
+/// \brief Import a DLPack tensor as an Arrow Array.
+///
+/// Same restrictions on data types as `ExportArrayVersioned`, only row-major
+/// tensors are supported. Takes ownership of the `DLManagedTensorVersioned` in
+/// an error-safe fashion.
+///
+/// \param[in] raw DLPack tensor
+/// \param[in] copy Whether to copy the data instead of sharing it with the DLPack
+///            producer.
+/// \return An Arrow Array
+ARROW_EXPORT
+Result<std::shared_ptr<Array>> ImportArrayVersioned(DLManagedTensorVersioned* raw,

Review Comment:
   Is the "Versioned" suffix useful? We're not exposing any non-versioned API, and even then, we could just rely on function overloading.



##########
python/pyarrow/tests/test_dlpack.py:
##########
@@ -339,3 +339,81 @@ def test_dlpack_cuda_not_supported():
     with pytest.raises(NotImplementedError, match="DLPack support is implemented "
                        "only for buffers on CPU device."):
         carr.__dlpack_device__()
+
+
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_tensor_from_dlpack(np_type):
+    if Version(np.__version__) < Version("2.1.0"):
+        pytest.skip("Versioned DLPack capsules require numpy 2.1.0 or later")
+
+    # Non-contiguous, strided slice: DLPack carries explicit strides, so this
+    # should not need a copy on export.
+    base = np.arange(24, dtype=np_type).reshape((4, 6))
+    expected = base[::2, 1::2]
+    assert not expected.flags['C_CONTIGUOUS']
+    tensor = pa.Tensor.from_dlpack(expected)
+    assert isinstance(tensor, pa.Tensor)
+    np.testing.assert_array_equal(tensor.to_numpy(), expected, strict=True)
+
+
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_array_from_dlpack(np_type):
+    if Version(np.__version__) < Version("2.1.0"):
+        pytest.skip("Versioned DLPack capsules require numpy 2.1.0 or later")
+
+    expected = np.array([1, 2, 3, 4, 5], dtype=np_type)
+    arr = pa.Array.from_dlpack(expected)
+    assert isinstance(arr, pa.Array)
+    np.testing.assert_array_equal(arr.to_numpy(), expected, strict=True)
+
+
+@check_bytes_allocated
+def test_from_dlpack_zero_copy():
+    if Version(np.__version__) < Version("2.1.0"):
+        pytest.skip("Versioned DLPack capsules require numpy 2.1.0 or later")
+
+    expected = np.array([1, 2, 3], dtype=np.int64)
+    tensor = pa.Tensor.from_dlpack(expected)
+    result = tensor.to_numpy()
+    expected[0] = 100
+    # Zero-copy import: mutating the source is visible through the tensor.
+    assert result[0] == 100

Review Comment:
   Can we mutate `result` too?



##########
python/pyarrow/tests/test_dlpack.py:
##########
@@ -339,3 +339,81 @@ def test_dlpack_cuda_not_supported():
     with pytest.raises(NotImplementedError, match="DLPack support is implemented "
                        "only for buffers on CPU device."):
         carr.__dlpack_device__()
+
+
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_tensor_from_dlpack(np_type):
+    if Version(np.__version__) < Version("2.1.0"):
+        pytest.skip("Versioned DLPack capsules require numpy 2.1.0 or later")
+
+    # Non-contiguous, strided slice: DLPack carries explicit strides, so this
+    # should not need a copy on export.
+    base = np.arange(24, dtype=np_type).reshape((4, 6))
+    expected = base[::2, 1::2]
+    assert not expected.flags['C_CONTIGUOUS']
+    tensor = pa.Tensor.from_dlpack(expected)
+    assert isinstance(tensor, pa.Tensor)
+    np.testing.assert_array_equal(tensor.to_numpy(), expected, strict=True)

Review Comment:
   Should test that `tensor` is still valid after `expected` goes away?



##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +263,217 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ptr == nullptr) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (out.ptr_->version.major != VERSION.major) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  std::span<const int64_t> strides() const {
+    return {tensor().strides, static_cast<std::size_t>(ndim())};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+inline std::vector<int64_t> StridesInBytes(std::span<const int64_t> strides,
+                                           int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    out.push_back(s * byte_width);
+  }
+  return out;
+}
+
+Result<std::shared_ptr<Buffer>> ImportBuffer(CppDLTensor&& dl, bool copy) {
+  // DLPack strides are in number of elements, so is the size we compute from them.
+  ARROW_ASSIGN_OR_RAISE(const auto nelements,
+                        internal::ComputeTensorSize(dl.shape(), dl.strides(), 1));
+  const auto nbytes = nelements * dl.byte_width();
+  // DLPack mandates a null data pointer when the tensor holds no element, so there is
+  // neither anything to share nor to copy.
+  uint8_t* data =
+      (nbytes == 0) ? nullptr : dl.data_as<uint8_t>() + dl.tensor().byte_offset;
+
+  std::shared_ptr<Buffer> buffer = nullptr;
+  if (nbytes == 0) {
+    // DLPack data pointer may be null on empty tensors
+    buffer = std::make_shared<Buffer>(data, nbytes);
+  } else if (copy) {
+    ARROW_ASSIGN_OR_RAISE(buffer, MutableBuffer::CopyNonOwned(
+                                      {data, nbytes}, default_cpu_memory_manager()));
+  } else {
+    const bool readonly = dl.is_readonly();
+    // Trick to keep DLPack data alive taken from `Buffer::FromVector`.

Review Comment:
   That said, I understand that the subclassing strategy might not be scalable once we want to support non-CPU devices.



##########
python/pyarrow/tests/test_dlpack.py:
##########
@@ -339,3 +339,81 @@ def test_dlpack_cuda_not_supported():
     with pytest.raises(NotImplementedError, match="DLPack support is implemented "
                        "only for buffers on CPU device."):
         carr.__dlpack_device__()
+
+
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_tensor_from_dlpack(np_type):
+    if Version(np.__version__) < Version("2.1.0"):
+        pytest.skip("Versioned DLPack capsules require numpy 2.1.0 or later")
+
+    # Non-contiguous, strided slice: DLPack carries explicit strides, so this
+    # should not need a copy on export.
+    base = np.arange(24, dtype=np_type).reshape((4, 6))
+    expected = base[::2, 1::2]
+    assert not expected.flags['C_CONTIGUOUS']
+    tensor = pa.Tensor.from_dlpack(expected)
+    assert isinstance(tensor, pa.Tensor)
+    np.testing.assert_array_equal(tensor.to_numpy(), expected, strict=True)
+
+
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_array_from_dlpack(np_type):
+    if Version(np.__version__) < Version("2.1.0"):
+        pytest.skip("Versioned DLPack capsules require numpy 2.1.0 or later")
+
+    expected = np.array([1, 2, 3, 4, 5], dtype=np_type)
+    arr = pa.Array.from_dlpack(expected)

Review Comment:
   Call `arr.validate(full=True)`?



##########
python/pyarrow/array.pxi:
##########
@@ -2269,6 +2274,51 @@ cdef class Array(_PandasConvertible):
 
         return pyarrow_wrap_array(array)
 
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct an Array from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional

Review Comment:
   PyArrow also has its own Device class, can we add a TODO and/or open an issue to support it here?
   ```python
   >>> mm = pa.default_cpu_memory_manager()
   >>> mm.device
   <pyarrow.Device: CPUDevice()>
   >>> mm.device.device_type
   <DeviceAllocationType.CPU: 1>
   
   >>> import pyarrow.cuda as cu
   >>> ctx = cu.Context()
   >>> ctx.device
   <pyarrow.Device: CudaDevice(device_number=0, name="NVIDIA GeForce GT 1030")>
   >>> ctx.device.device_type
   <DeviceAllocationType.CUDA: 2>
   ```



##########
cpp/src/arrow/c/dlpack_test.cc:
##########
@@ -329,4 +329,353 @@ TYPED_TEST(TestExportTensor, TestTensorStrided) {
                            f_dlpack_strides);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+/// A DLPack tensor as a foreign library would produce it.
+struct ForeignTensor {
+  DLDataType dtype = {.code = kDLFloat, .bits = 32, .lanes = 1};
+  std::vector<int64_t> shape = {};
+  /// In number of elements, as mandated by DLPack.
+  std::vector<int64_t> strides = {};
+  std::vector<uint8_t> data = {};
+  DLDevice device = {.device_type = kDLCPU, .device_id = 0};
+  uint64_t byte_offset = 0;
+  uint64_t flags = 0;
+  /// Incremented when the consumer releases the tensor.
+  std::shared_ptr<int> deleted = std::make_shared<int>(0);
+
+  DLManagedTensorVersioned managed = {};
+};
+
+template <typename T>
+std::vector<uint8_t> ToBytes(const std::vector<T>& values) {
+  std::vector<uint8_t> bytes(values.size() * sizeof(T));
+  std::memcpy(bytes.data(), values.data(), bytes.size());
+  return bytes;
+}
+
+/// Hand out a DLPack tensor owning ``foreign``, releasing it through its deleter.
+DLManagedTensorVersioned* Produce(ForeignTensor foreign) {
+  auto owned = std::make_unique<ForeignTensor>(std::move(foreign));
+  owned->managed = {
+      .version = {.major = DLPACK_MAJOR_VERSION, .minor = DLPACK_MINOR_VERSION},
+      .manager_ctx = owned.get(),
+      .deleter =
+          [](DLManagedTensorVersioned* self) {
+            auto* ctx = static_cast<ForeignTensor*>(self->manager_ctx);
+            ++(*ctx->deleted);
+            delete ctx;
+          },
+      .flags = owned->flags,
+      .dl_tensor =
+          {
+              .data = owned->data.data(),
+              .device = owned->device,
+              .ndim = static_cast<int32_t>(owned->shape.size()),
+              .dtype = owned->dtype,
+              .shape = owned->shape.data(),
+              .strides = owned->strides.data(),
+              .byte_offset = owned->byte_offset,
+          },
+  };
+  return &owned.release()->managed;
+}
+
+template <bool kCopy>
+struct TensorConsumer {
+  using Imported = std::shared_ptr<Tensor>;
+  static constexpr bool copy = kCopy;
+  static constexpr const char* name = copy ? "TensorCopied" : "TensorShared";
+
+  static Result<Imported> Import(DLManagedTensorVersioned* raw) {
+    return ImportTensorVersioned(raw, copy);
+  }
+  static std::shared_ptr<DataType> ValueType(const Imported& t) { return t->type(); }
+  static const uint8_t* RawData(const Imported& t) { return t->raw_data(); }
+  static bool IsMutable(const Imported& t) { return t->is_mutable(); }
+  static int64_t Size(const Imported& t) { return t->size(); }
+};

Review Comment:
   Can we add a validation method that would call `Tensor::Validate` and `Array::ValidateFull`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3948619147


##########
cpp/src/arrow/c/dlpack.h:
##########
@@ -105,4 +108,32 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Array>& arr);
 ARROW_EXPORT
 Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t);
 
+/// \brief Import a DLPack tensor as an Arrow Array.
+///
+/// Same restrictions on data types as `ExportArrayVersioned`, only row-major
+/// tensors are supported. Takes ownership of the `DLManagedTensorVersioned` in
+/// an error-safe fashion.
+///
+/// \param[in] raw DLPack tensor
+/// \param[in] copy Whether to copy the data instead of sharing it with the DLPack
+///            producer.
+/// \return An Arrow Array
+ARROW_EXPORT
+Result<std::shared_ptr<Array>> ImportArrayVersioned(DLManagedTensorVersioned* raw,

Review Comment:
   Mostly to match the producer side `ExportArrayVersioned`, I feel it easier to match the importing and exporting functions that way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3958653503


##########
cpp/src/arrow/buffer.h:
##########
@@ -146,14 +147,66 @@ class ARROW_EXPORT Buffer {
     }
   }
 
-  /// \brief Construct an immutable buffer that takes ownership of the contents
+  /// \brief Default data accessor used by TakeOwnership.
+  struct DefaultGetData {

Review Comment:
   Can we just make this private?
   
   Or perhaps just use [`std::data`](https://en.cppreference.com/cpp/iterator/data)?



##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1708,6 +1708,26 @@ def test_tensor_class_methods(np_type_str):
     assert result.to_tensor().strides == (12 * bw, 1 * bw, 3 * bw, 6 * bw)
 
 
+@pytest.mark.numpy
+@pytest.mark.parametrize(
+    ("transpose", "permutation"),
+    [(False, [0, 1]), (True, [1, 0])]
+)
+def test_tensor_array_from_tensor(transpose, permutation):
+    arr = np.arange(24, dtype=np.int32).reshape(2, 3, 4)
+    arr = arr.transpose(0, 2, 1) if transpose else arr
+
+    result = pa.FixedShapeTensorArray.from_tensor(pa.Tensor.from_numpy(arr))

Review Comment:
   Also validate the result here?



##########
python/pyarrow/tests/test_dlpack.py:
##########
@@ -374,3 +374,101 @@ def test_dlpack_cuda_not_supported():
     with pytest.raises(NotImplementedError, match="DLPack support is implemented "
                        "only for buffers on CPU device."):
         carr.__dlpack_device__()
+
+
+@requires_numpy_version("2.1.0")
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_tensor_from_dlpack(np_type):
+    def make_array():
+        base = np.arange(24, dtype=np_type).reshape((4, 6))
+        array = base[::2, 1::2]
+        assert not array.flags['C_CONTIGUOUS']
+        return array
+
+    # Non-contiguous, strided slice: DLPack carries explicit strides, so this
+    # should not need a copy on export.
+    tensor = pa.Tensor.from_dlpack(make_array())
+    assert isinstance(tensor, pa.Tensor)
+    gc.collect()  # Attempts to free input array memory
+    np.testing.assert_array_equal(tensor.to_numpy(), make_array(), strict=True)
+
+
+@requires_numpy_version("2.1.0")
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_array_from_dlpack(np_type):
+    expected = np.array([1, 2, 3, 4, 5], dtype=np_type)
+    arr = pa.Array.from_dlpack(expected)
+    arr.validate(full=True)
+    assert isinstance(arr, pa.Array)
+    np.testing.assert_array_equal(arr.to_numpy(), expected, strict=True)
+
+
+@requires_numpy_version("2.1.0")
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_fixed_shape_tensor_array_from_dlpack(np_type):
+    source = np.arange(12, dtype=np_type).reshape((3, 2, 2))

Review Comment:
   Can we also test with a transposed (non-row major) source as well?



##########
python/pyarrow/tests/test_extension_type.py:
##########
@@ -1708,6 +1708,26 @@ def test_tensor_class_methods(np_type_str):
     assert result.to_tensor().strides == (12 * bw, 1 * bw, 3 * bw, 6 * bw)
 
 
+@pytest.mark.numpy
+@pytest.mark.parametrize(
+    ("transpose", "permutation"),
+    [(False, [0, 1]), (True, [1, 0])]
+)
+def test_tensor_array_from_tensor(transpose, permutation):
+    arr = np.arange(24, dtype=np.int32).reshape(2, 3, 4)
+    arr = arr.transpose(0, 2, 1) if transpose else arr

Review Comment:
   What if the first dimension is transposed too? I suspect we'll get an error, can we test that?



##########
python/pyarrow/tests/test_dlpack.py:
##########
@@ -374,3 +374,101 @@ def test_dlpack_cuda_not_supported():
     with pytest.raises(NotImplementedError, match="DLPack support is implemented "
                        "only for buffers on CPU device."):
         carr.__dlpack_device__()
+
+
+@requires_numpy_version("2.1.0")
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_tensor_from_dlpack(np_type):
+    def make_array():
+        base = np.arange(24, dtype=np_type).reshape((4, 6))
+        array = base[::2, 1::2]
+        assert not array.flags['C_CONTIGUOUS']
+        return array
+
+    # Non-contiguous, strided slice: DLPack carries explicit strides, so this
+    # should not need a copy on export.
+    tensor = pa.Tensor.from_dlpack(make_array())
+    assert isinstance(tensor, pa.Tensor)
+    gc.collect()  # Attempts to free input array memory
+    np.testing.assert_array_equal(tensor.to_numpy(), make_array(), strict=True)
+
+
+@requires_numpy_version("2.1.0")
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_array_from_dlpack(np_type):
+    expected = np.array([1, 2, 3, 4, 5], dtype=np_type)
+    arr = pa.Array.from_dlpack(expected)
+    arr.validate(full=True)
+    assert isinstance(arr, pa.Array)
+    np.testing.assert_array_equal(arr.to_numpy(), expected, strict=True)
+
+
+@requires_numpy_version("2.1.0")
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_fixed_shape_tensor_array_from_dlpack(np_type):
+    source = np.arange(12, dtype=np_type).reshape((3, 2, 2))
+    arr = pa.FixedShapeTensorArray.from_dlpack(source)
+    arr.validate(full=True)
+    assert arr.type == pa.fixed_shape_tensor(pa.from_numpy_dtype(np_type), [2, 2])
+    assert arr.to_pylist() == [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]]

Review Comment:
   Interesting. I'd have expected `[[[0, 1], [2, 3]], [[4, 5], [6, 7]], [[8, 9], [10, 11]]]`, which is also what NumPy returns. It's not really important, but slightly surprising IMHO.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3932596902


##########
python/pyarrow/array.pxi:
##########
@@ -2269,6 +2274,51 @@ cdef class Array(_PandasConvertible):
 
         return pyarrow_wrap_array(array)
 
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct an Array from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional
+            Designates where the resulting Array should reside, in the
+            format returned by :meth:`Array.__dlpack_device__`. When None,
+            the output Array occupies the same device as the source.
+            Default: None.
+        copy : bool, optional
+            Controls duplication behavior. True mandates copying; False
+            prohibits copying and raises ``BufferError`` if unavoidable;
+            None duplicates only when necessary. Default: None.
+
+        Returns
+        -------
+        Array
+            An Array housing the data from the input object, potentially
+            as a copy or view.
+        """
+        version = (DLPACK_VERSION.major, DLPACK_VERSION.minor)
+        pycapsule = x.__dlpack__(max_version=version, dl_device=device, copy=copy)
+        if not PyCapsule_CheckExact(pycapsule):
+            raise TypeError("DLPack producer did not return a PyCapsule")
+        cdef DLManagedTensorVersioned* ptr = <DLManagedTensorVersioned*>PyCapsule_GetPointer(
+            pycapsule, "dltensor_versioned")
+        if ptr == NULL:
+            raise ValueError(
+                'DLPack producer did not produce a "dltensor_versioned" PyCapsule')
+        # Mark the capsule as consumed so its destructor does not also invoke the deleter.
+        # ImportArrayVersionedFromDLPack will take ownership even if it errors (calling
+        # the deleter in that case).
+        PyCapsule_SetName(pycapsule, "used_dltensor_versioned")
+        with nogil:
+            # Copy handled on producer side
+            result = ImportArrayVersionedFromDLPack(ptr, False)
+        carray = GetResultValue(result)

Review Comment:
   In `Array.from_dlpack()`, `result` is assigned inside a `with nogil:` block but isn’t declared as a Cython cdef variable, which will fail to compile (and can’t use a Python variable without the GIL). Also check `PyCapsule_SetName` for failure; if it fails and you proceed, the capsule destructor may still call the DLPack deleter, risking double-free.



##########
python/pyarrow/tensor.pxi:
##########
@@ -300,7 +306,52 @@ strides: {self.strides}"""
         buffer.strides = <Py_ssize_t *> cp.PyBytes_AsString(self._ssize_t_strides)
         buffer.suboffsets = NULL
 
-    def __dlpack__(self, stream=None, max_version=None, dl_device=None, copy=None):
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct a Tensor from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional
+            Designates where the resulting Tensor should reside, in the
+            format returned by :meth:`Tensor.__dlpack_device__`. When None,
+            the output Tensor occupies the same device as the source.
+            Default: None.
+        copy : bool, optional
+            Controls duplication behavior. True mandates copying; False
+            prohibits copying and raises ``BufferError`` if unavoidable;
+            None duplicates only when necessary. Default: None.
+
+        Returns
+        -------
+        Tensor
+            A Tensor housing the data from the input object, potentially
+            as a copy or view.
+        """
+        version = (DLPACK_VERSION.major, DLPACK_VERSION.minor)
+        pycapsule = x.__dlpack__(max_version=version, dl_device=device, copy=copy)
+        if not PyCapsule_CheckExact(pycapsule):
+            raise TypeError("DLPack producer did not return a PyCapsule")
+        cdef DLManagedTensorVersioned* ptr = <DLManagedTensorVersioned*>PyCapsule_GetPointer(
+            pycapsule, "dltensor_versioned")
+        if ptr == NULL:
+            raise ValueError(
+                'DLPack producer did not produce a "dltensor_versioned" PyCapsule')
+        # Mark the capsule as consumed so its destructor does not also invoke the deleter.
+        # ImportTensorVersionedFromDLPack will take ownership even if it errors (calling
+        # the deleter in that case).
+        PyCapsule_SetName(pycapsule, "used_dltensor_versioned")
+        with nogil:
+            # Copy handled on producer side
+            result = ImportTensorVersionedFromDLPack(ptr, False)
+        ctensor = GetResultValue(result)

Review Comment:
   In `Tensor.from_dlpack()`, `result` is assigned inside a `with nogil:` block but isn’t declared as a Cython cdef variable, which will fail to compile (and can’t use a Python variable without the GIL). Also check `PyCapsule_SetName` for failure; if it fails and you proceed, the capsule destructor may still call the DLPack deleter, risking double-free.



##########
python/pyarrow/tests/test_dlpack.py:
##########
@@ -339,3 +339,81 @@ def test_dlpack_cuda_not_supported():
     with pytest.raises(NotImplementedError, match="DLPack support is implemented "
                        "only for buffers on CPU device."):
         carr.__dlpack_device__()
+
+
+@check_bytes_allocated
+@pytest.mark.parametrize('np_type',
+                         [np.uint8, np.uint16, np.uint32, np.uint64,
+                          np.int8, np.int16, np.int32, np.int64,
+                          np.float16, np.float32, np.float64])
+def test_tensor_from_dlpack(np_type):
+    if Version(np.__version__) < Version("2.1.0"):
+        pytest.skip("Versioned DLPack capsules require numpy 2.1.0 or later")
+
+    # Non-contiguous, strided slice: DLPack carries explicit strides, so this
+    # should not need a copy on export.

Review Comment:
   Comment says “copy on export” but this test is exercising *import* via `from_dlpack`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3955455617


##########
cpp/src/arrow/buffer.h:
##########
@@ -146,14 +147,66 @@ class ARROW_EXPORT Buffer {
     }
   }
 
-  /// \brief Construct an immutable buffer that takes ownership of the contents
+  /// \brief Default data accessor used by TakeOwnership.
+  struct DefaultGetData {
+    template <typename T>
+    auto* operator()(T& container) const {
+      return container.data();
+    }
+  };
+
+  /// \brief Construct an immutable buffer that takes ownership of a container.

Review Comment:
   The docstring says "Construct an immutable buffer" but the function can intentionally return a `MutableBuffer` depending on the owned container (and `FromString` / `FromVector` now do). Update the wording to avoid contradicting the behavior.



##########
python/pyarrow/array.pxi:
##########
@@ -2269,6 +2274,54 @@ cdef class Array(_PandasConvertible):
 
         return pyarrow_wrap_array(array)
 
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct an Array from an object implementing the DLPack protocol.
+        Only 1 dimensional contiguous tensor are accepted as input.
+        For multi-dimensional tensor, first use `Tensor.from_dpack`, then
+        convert to an array using `FixedShapeTensorArray.from_tensor()`.

Review Comment:
   Docstring has a typo and misleading guidance: it says `Tensor.from_dpack` (typo) and uses ungrammatical wording. This is user-facing API documentation for `Array.from_dlpack`.
   
   This issue also appears on line 2315 of the same file.



##########
python/pyarrow/tensor.pxi:
##########
@@ -300,7 +306,52 @@ strides: {self.strides}"""
         buffer.strides = <Py_ssize_t *> cp.PyBytes_AsString(self._ssize_t_strides)
         buffer.suboffsets = NULL
 
-    def __dlpack__(self, stream=None, max_version=None, dl_device=None, copy=None):
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct a Tensor from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional
+            Designates where the resulting Tensor should reside, in the
+            format returned by :meth:`Tensor.__dlpack_device__`. When None,
+            the output Tensor occupies the same device as the source.
+            Default: None.
+        copy : bool, optional
+            Controls duplication behavior. True mandates copying; False
+            prohibits copying and raises ``BufferError`` if unavoidable;
+            None duplicates only when necessary. Default: None.
+
+        Returns
+        -------
+        Tensor
+            A Tensor housing the data from the input object, potentially
+            as a copy or view.
+        """
+        version = (DLPACK_VERSION.major, DLPACK_VERSION.minor)
+        pycapsule = x.__dlpack__(max_version=version, dl_device=device, copy=copy)
+        if not PyCapsule_CheckExact(pycapsule):
+            raise TypeError("DLPack producer did not return a PyCapsule")
+        cdef DLManagedTensorVersioned* ptr = <DLManagedTensorVersioned*>PyCapsule_GetPointer(
+            pycapsule, "dltensor_versioned")
+        if ptr == NULL:
+            raise ValueError(
+                'DLPack producer did not produce a "dltensor_versioned" PyCapsule')
+        # Mark the capsule as consumed so its destructor does not also invoke the deleter.
+        # ImportTensorVersionedFromDLPack will take ownership even if it errors (calling
+        # the deleter in that case).
+        PyCapsule_SetName(pycapsule, "used_dltensor_versioned")
+        with nogil:

Review Comment:
   `PyCapsule_SetName` return value is ignored. If it fails, the capsule may still call its destructor deleter while Arrow also takes ownership, risking double-free. Check for failure and raise before importing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3959459845


##########
cpp/src/arrow/buffer.h:
##########
@@ -146,14 +147,66 @@ class ARROW_EXPORT Buffer {
     }
   }
 
-  /// \brief Construct an immutable buffer that takes ownership of the contents
+  /// \brief Default data accessor used by TakeOwnership.
+  struct DefaultGetData {

Review Comment:
   Using private because `std::data` is a function so it is harder to make it a default template type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3979362920


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +262,265 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ARROW_PREDICT_FALSE(ptr == nullptr)) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (ARROW_PREDICT_FALSE(out.ptr_->version.major != kVersion.major)) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", kVersion.major);
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim < 0)) {
+      return Status::Invalid("Invalid DLPack tensor: ndim must be >= 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().shape == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: shape must be non-null when ndim != 0");
+    }
+    // Null strides are handled as row major
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  /// Strides or empty span for old DLPack row-major convention.
+  std::span<const int64_t> strides() const {
+    if (auto strides = tensor().strides; strides != nullptr) {
+      return {strides, static_cast<std::size_t>(ndim())};
+    }
+    return {};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+  /// Number of element in this tensor's buffer.
+  ///
+  /// Possibly more elements than represented in the tensor for non-contiguous tensors.
+  ///
+  /// A zero dimensional tensor is a scalar, it holds a single element.
+  Result<int64_t> ComputeNumElements() const {
+    const auto strides = this->strides();
+    const auto shape = this->shape();
+    if (strides.size() > 0) {
+      // DLPack strides are in number of elements, so is the size we compute from them.
+      return internal::ComputeTensorSize(shape, strides, 1);
+    }
+    // DLPack <1.3 my set strides == nullptr for row major

Review Comment:
   ```suggestion
       // DLPack <1.3 may set strides == nullptr for row major
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #51122:
URL: https://github.com/apache/arrow/pull/51122#issuecomment-5620833582

   CI failures are unrelated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3948638410


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +263,217 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ptr == nullptr) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (out.ptr_->version.major != VERSION.major) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  std::span<const int64_t> strides() const {
+    return {tensor().strides, static_cast<std::size_t>(ndim())};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+inline std::vector<int64_t> StridesInBytes(std::span<const int64_t> strides,
+                                           int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    out.push_back(s * byte_width);
+  }
+  return out;
+}
+
+Result<std::shared_ptr<Buffer>> ImportBuffer(CppDLTensor&& dl, bool copy) {
+  // DLPack strides are in number of elements, so is the size we compute from them.
+  ARROW_ASSIGN_OR_RAISE(const auto nelements,
+                        internal::ComputeTensorSize(dl.shape(), dl.strides(), 1));
+  const auto nbytes = nelements * dl.byte_width();
+  // DLPack mandates a null data pointer when the tensor holds no element, so there is
+  // neither anything to share nor to copy.
+  uint8_t* data =
+      (nbytes == 0) ? nullptr : dl.data_as<uint8_t>() + dl.tensor().byte_offset;
+
+  std::shared_ptr<Buffer> buffer = nullptr;
+  if (nbytes == 0) {
+    // DLPack data pointer may be null on empty tensors
+    buffer = std::make_shared<Buffer>(data, nbytes);
+  } else if (copy) {
+    ARROW_ASSIGN_OR_RAISE(buffer, MutableBuffer::CopyNonOwned(
+                                      {data, nbytes}, default_cpu_memory_manager()));
+  } else {
+    const bool readonly = dl.is_readonly();
+    // Trick to keep DLPack data alive taken from `Buffer::FromVector`.

Review Comment:
   Why not, yes!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3954971716


##########
python/pyarrow/array.pxi:
##########
@@ -2269,6 +2274,51 @@ cdef class Array(_PandasConvertible):
 
         return pyarrow_wrap_array(array)
 
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct an Array from an object implementing the DLPack protocol.

Review Comment:
   Can we mention the 1-D restriction here ad point users to `FixedShapeTensorArray.from_tensor`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3950793347


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +261,238 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ARROW_PREDICT_FALSE(ptr == nullptr)) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (ARROW_PREDICT_FALSE(out.ptr_->version.major != VERSION.major)) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim < 0)) {
+      return Status::Invalid("Invalid DLPack tensor: ndim must be >= 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().shape == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: shape must be non-null when ndim != 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().strides == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: strides must be non-null when ndim != 0");
+    }
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  std::span<const int64_t> strides() const {
+    return {tensor().strides, static_cast<std::size_t>(ndim())};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+Result<std::vector<int64_t>> StridesInBytes(std::span<const int64_t> strides,
+                                            int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    int64_t stride_bytes = 0;
+    if (ARROW_PREDICT_FALSE(
+            internal::MultiplyWithOverflow(s, byte_width, &stride_bytes))) {
+      return Status::Invalid("Overflow computing DLPack tensor stride in bytes.");
+    }
+    out.push_back(stride_bytes);
+  }
+  return out;
+}
+
+Result<std::shared_ptr<Buffer>> ImportBuffer(CppDLTensor&& dl, bool copy) {
+  // DLPack strides are in number of elements, so is the size we compute from them.
+  ARROW_ASSIGN_OR_RAISE(const auto nelements,
+                        internal::ComputeTensorSize(dl.shape(), dl.strides(), 1));
+  int64_t nbytes = 0;
+  if (ARROW_PREDICT_FALSE(internal::MultiplyWithOverflow(
+          nelements, static_cast<int64_t>(dl.byte_width()), &nbytes))) {
+    return Status::Invalid("Overflow computing DLPack tensor size in bytes.");
+  }
+  // DLPack mandates a null data pointer when the tensor holds no element, so there is
+  // neither anything to share nor to copy.
+  uint8_t* data =
+      (nbytes == 0) ? nullptr : dl.data_as<uint8_t>() + dl.tensor().byte_offset;
+
+  std::shared_ptr<Buffer> buffer = nullptr;
+  if (nbytes == 0) {
+    // DLPack data pointer may be null on empty tensors
+    buffer = std::make_shared<Buffer>(data, nbytes);
+  } else if (copy) {
+    ARROW_ASSIGN_OR_RAISE(buffer, MutableBuffer::CopyNonOwned(
+                                      {data, nbytes}, default_cpu_memory_manager()));
+  } else {

Review Comment:
   @pitrou I've been thinking with this and:
   - First we do not need the copy parameter in `ImportArray` because we already can pass `copy` to the producer (which is what we do in Python).
   - Then without `copy` I believe we have no use for `DLPACK_FLAG_BITMASK_IS_COPIED`.
   
   What do you think, should we keep our `copy` parameter (currently unused) or always rely on the producer to copy?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3951081276


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +261,238 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ARROW_PREDICT_FALSE(ptr == nullptr)) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (ARROW_PREDICT_FALSE(out.ptr_->version.major != VERSION.major)) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim < 0)) {
+      return Status::Invalid("Invalid DLPack tensor: ndim must be >= 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().shape == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: shape must be non-null when ndim != 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().strides == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: strides must be non-null when ndim != 0");
+    }
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  std::span<const int64_t> strides() const {
+    return {tensor().strides, static_cast<std::size_t>(ndim())};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+Result<std::vector<int64_t>> StridesInBytes(std::span<const int64_t> strides,
+                                            int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    int64_t stride_bytes = 0;
+    if (ARROW_PREDICT_FALSE(
+            internal::MultiplyWithOverflow(s, byte_width, &stride_bytes))) {
+      return Status::Invalid("Overflow computing DLPack tensor stride in bytes.");
+    }
+    out.push_back(stride_bytes);
+  }
+  return out;
+}
+
+Result<std::shared_ptr<Buffer>> ImportBuffer(CppDLTensor&& dl, bool copy) {
+  // DLPack strides are in number of elements, so is the size we compute from them.
+  ARROW_ASSIGN_OR_RAISE(const auto nelements,
+                        internal::ComputeTensorSize(dl.shape(), dl.strides(), 1));
+  int64_t nbytes = 0;
+  if (ARROW_PREDICT_FALSE(internal::MultiplyWithOverflow(
+          nelements, static_cast<int64_t>(dl.byte_width()), &nbytes))) {
+    return Status::Invalid("Overflow computing DLPack tensor size in bytes.");
+  }
+  // DLPack mandates a null data pointer when the tensor holds no element, so there is
+  // neither anything to share nor to copy.
+  uint8_t* data =
+      (nbytes == 0) ? nullptr : dl.data_as<uint8_t>() + dl.tensor().byte_offset;
+
+  std::shared_ptr<Buffer> buffer = nullptr;
+  if (nbytes == 0) {
+    // DLPack data pointer may be null on empty tensors
+    buffer = std::make_shared<Buffer>(data, nbytes);
+  } else if (copy) {
+    ARROW_ASSIGN_OR_RAISE(buffer, MutableBuffer::CopyNonOwned(
+                                      {data, nbytes}, default_cpu_memory_manager()));
+  } else {

Review Comment:
   If it's currently unused then let's remove it for now?
   
   On the Python side, as you say, `__dlpack__` is called first.
   If there are some direct C/C++ uses of DLPack, then we may want to revisit later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #51122:
URL: https://github.com/apache/arrow/pull/51122


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3959674334


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +261,234 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ARROW_PREDICT_FALSE(ptr == nullptr)) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (ARROW_PREDICT_FALSE(out.ptr_->version.major != kVersion.major)) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", kVersion.major);
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim < 0)) {
+      return Status::Invalid("Invalid DLPack tensor: ndim must be >= 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().shape == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: shape must be non-null when ndim != 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().strides == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: strides must be non-null when ndim != 0");
+    }

Review Comment:
   `CppDLTensor::TakeOwnership` rejects tensors with `strides == nullptr` when `ndim != 0`, but `arrow/c/dlpack_abi.h` explicitly notes that (before DLPack v1.2) NULL strides are allowed to indicate contiguous data. Since the importer only checks `version.major`, this will reject valid v1.0/v1.1 producers (and would also make later `dl.strides().front()` accesses unsafe if the check is relaxed). Consider accepting NULL strides when `version.minor < 2` and treating them as compact row-major strides computed from `shape` (and still rejecting NULL strides for v1.2+).



##########
cpp/src/arrow/c/dlpack_test.cc:
##########
@@ -329,4 +329,361 @@ TYPED_TEST(TestExportTensor, TestTensorStrided) {
                            f_dlpack_strides);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+/// A DLPack tensor as a foreign library would produce it.
+struct ForeignTensor {
+  DLDataType dtype = {.code = kDLFloat, .bits = 32, .lanes = 1};
+  std::vector<int64_t> shape = {};
+  /// In number of elements, as mandated by DLPack.
+  std::vector<int64_t> strides = {};
+  std::vector<uint8_t> data = {};
+  DLDevice device = {.device_type = kDLCPU, .device_id = 0};
+  uint64_t byte_offset = 0;
+  uint64_t flags = 0;
+  /// Incremented when the consumer releases the tensor.
+  std::shared_ptr<int> deleted = std::make_shared<int>(0);
+
+  DLManagedTensorVersioned managed = {};
+};
+
+template <typename T>
+std::vector<uint8_t> ToBytes(const std::vector<T>& values) {
+  std::vector<uint8_t> bytes(values.size() * sizeof(T));
+  std::memcpy(bytes.data(), values.data(), bytes.size());
+  return bytes;
+}
+
+/// Hand out a DLPack tensor owning ``foreign``, releasing it through its deleter.
+DLManagedTensorVersioned* Produce(ForeignTensor foreign) {
+  auto owned = std::make_unique<ForeignTensor>(std::move(foreign));
+  owned->managed = {
+      .version = {.major = DLPACK_MAJOR_VERSION, .minor = DLPACK_MINOR_VERSION},
+      .manager_ctx = owned.get(),

Review Comment:
   The new consumer tests don't cover the pre-DLPack-v1.2 case where `dl_tensor.strides` may be NULL to indicate contiguous data (documented in `arrow/c/dlpack_abi.h`). Adding a regression test that sets `managed->version.minor = 1` and `managed->dl_tensor.strides = nullptr` for a compact row-major tensor would help ensure `Import{Array,Tensor}Versioned` stays compatible with older producers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3932494990


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +261,232 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ARROW_PREDICT_FALSE(ptr == nullptr)) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (ARROW_PREDICT_FALSE(out.ptr_->version.major != VERSION.major)) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim < 0)) {
+      return Status::Invalid("Invalid DLPack tensor: ndim must be >= 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().shape == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: shape must be non-null when ndim != 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().strides == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: strides must be non-null when ndim != 0");
+    }
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  std::span<const int64_t> strides() const {
+    return {tensor().strides, static_cast<std::size_t>(ndim())};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+inline std::vector<int64_t> StridesInBytes(std::span<const int64_t> strides,
+                                           int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    out.push_back(s * byte_width);
+  }
+  return out;
+}

Review Comment:
   `StridesInBytes` multiplies `int64_t` strides by `byte_width` using `s * byte_width`, which can overflow and is undefined behavior in C++. Since the inputs come from an external DLPack capsule, this should use `MultiplyWithOverflow` and fail (or force an error path) on overflow.



##########
python/pyarrow/array.pxi:
##########
@@ -2269,6 +2274,51 @@ cdef class Array(_PandasConvertible):
 
         return pyarrow_wrap_array(array)
 
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct an Array from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional
+            Designates where the resulting Array should reside, in the
+            format returned by :meth:`Array.__dlpack_device__`. When None,
+            the output Array occupies the same device as the source.
+            Default: None.
+        copy : bool, optional
+            Controls duplication behavior. True mandates copying; False
+            prohibits copying and raises ``BufferError`` if unavoidable;
+            None duplicates only when necessary. Default: None.
+
+        Returns
+        -------
+        Array
+            An Array housing the data from the input object, potentially
+            as a copy or view.
+        """
+        version = (DLPACK_VERSION.major, DLPACK_VERSION.minor)
+        pycapsule = x.__dlpack__(max_version=version, dl_device=device, copy=copy)
+        if not PyCapsule_CheckExact(pycapsule):
+            raise TypeError("DLPack producer did not return a PyCapsule")
+        cdef DLManagedTensorVersioned* ptr = <DLManagedTensorVersioned*>PyCapsule_GetPointer(
+            pycapsule, "dltensor_versioned")
+        if ptr == NULL:
+            raise ValueError(
+                'DLPack producer did not produce a "dltensor_versioned" PyCapsule')
+        # Mark the capsule as consumed so its destructor does not also invoke the deleter.
+        # ImportArrayVersionedFromDLPack will take ownership even if it errors (calling
+        # the deleter in that case).
+        PyCapsule_SetName(pycapsule, "used_dltensor_versioned")
+        with nogil:
+            # Copy handled on producer side
+            result = ImportArrayVersionedFromDLPack(ptr, False)
+        carray = GetResultValue(result)
+        return pyarrow_wrap_array(carray)

Review Comment:
   `copy` is documented as “True mandates copying”, but the implementation always calls `ImportArrayVersionedFromDLPack(ptr, False)`, so Arrow never enforces an owning copy. This can violate the contract for producers that ignore or only best-effort the `copy=` hint. Consider passing `copy is True` down to the C++ importer so `copy=True` reliably produces an owning buffer.



##########
python/pyarrow/tensor.pxi:
##########
@@ -300,7 +306,52 @@ strides: {self.strides}"""
         buffer.strides = <Py_ssize_t *> cp.PyBytes_AsString(self._ssize_t_strides)
         buffer.suboffsets = NULL
 
-    def __dlpack__(self, stream=None, max_version=None, dl_device=None, copy=None):
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct a Tensor from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional
+            Designates where the resulting Tensor should reside, in the
+            format returned by :meth:`Tensor.__dlpack_device__`. When None,
+            the output Tensor occupies the same device as the source.
+            Default: None.
+        copy : bool, optional
+            Controls duplication behavior. True mandates copying; False
+            prohibits copying and raises ``BufferError`` if unavoidable;
+            None duplicates only when necessary. Default: None.
+
+        Returns
+        -------
+        Tensor
+            A Tensor housing the data from the input object, potentially
+            as a copy or view.
+        """
+        version = (DLPACK_VERSION.major, DLPACK_VERSION.minor)
+        pycapsule = x.__dlpack__(max_version=version, dl_device=device, copy=copy)
+        if not PyCapsule_CheckExact(pycapsule):
+            raise TypeError("DLPack producer did not return a PyCapsule")
+        cdef DLManagedTensorVersioned* ptr = <DLManagedTensorVersioned*>PyCapsule_GetPointer(
+            pycapsule, "dltensor_versioned")
+        if ptr == NULL:
+            raise ValueError(
+                'DLPack producer did not produce a "dltensor_versioned" PyCapsule')
+        # Mark the capsule as consumed so its destructor does not also invoke the deleter.
+        # ImportTensorVersionedFromDLPack will take ownership even if it errors (calling
+        # the deleter in that case).
+        PyCapsule_SetName(pycapsule, "used_dltensor_versioned")
+        with nogil:
+            # Copy handled on producer side
+            result = ImportTensorVersionedFromDLPack(ptr, False)
+        ctensor = GetResultValue(result)
+        return pyarrow_wrap_tensor(ctensor)

Review Comment:
   `copy` is documented as “True mandates copying”, but the implementation always calls `ImportTensorVersionedFromDLPack(ptr, False)`, so Arrow never enforces an owning copy. This can violate the contract for producers that ignore or only best-effort the `copy=` hint. Consider passing `copy is True` down to the C++ importer so `copy=True` reliably produces an owning buffer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3948609240


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +263,217 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ptr == nullptr) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (out.ptr_->version.major != VERSION.major) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", VERSION.major);
+    }
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  std::span<const int64_t> strides() const {
+    return {tensor().strides, static_cast<std::size_t>(ndim())};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+inline std::vector<int64_t> StridesInBytes(std::span<const int64_t> strides,
+                                           int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    out.push_back(s * byte_width);
+  }
+  return out;
+}
+
+Result<std::shared_ptr<Buffer>> ImportBuffer(CppDLTensor&& dl, bool copy) {
+  // DLPack strides are in number of elements, so is the size we compute from them.
+  ARROW_ASSIGN_OR_RAISE(const auto nelements,
+                        internal::ComputeTensorSize(dl.shape(), dl.strides(), 1));
+  const auto nbytes = nelements * dl.byte_width();
+  // DLPack mandates a null data pointer when the tensor holds no element, so there is
+  // neither anything to share nor to copy.
+  uint8_t* data =
+      (nbytes == 0) ? nullptr : dl.data_as<uint8_t>() + dl.tensor().byte_offset;
+
+  std::shared_ptr<Buffer> buffer = nullptr;
+  if (nbytes == 0) {
+    // DLPack data pointer may be null on empty tensors
+    buffer = std::make_shared<Buffer>(data, nbytes);
+  } else if (copy) {
+    ARROW_ASSIGN_OR_RAISE(buffer, MutableBuffer::CopyNonOwned(
+                                      {data, nbytes}, default_cpu_memory_manager()));
+  } else {
+    const bool readonly = dl.is_readonly();
+    // Trick to keep DLPack data alive taken from `Buffer::FromVector`.

Review Comment:
   I does seem a bit overkill to subclass for that purpose.
   What about adding `Buffer::TakeOwnership(T&& container, uint8_t const* data, int64_t nbytes);` (where data points into container)? I can see it being universally useful to a C++ user.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3951135292


##########
python/pyarrow/array.pxi:
##########
@@ -2269,6 +2274,51 @@ cdef class Array(_PandasConvertible):
 
         return pyarrow_wrap_array(array)
 
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct an Array from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional

Review Comment:
   Well NumPy [supports a string](https://numpy.org/devdocs/reference/generated/numpy.from_dlpack.html), PyTorch [supports a `torch.device`](https://docs.pytorch.org/docs/main/dlpack.html), JAX [supports a `_jax.Device`](https://docs.jax.dev/en/latest/_autosummary/jax.dlpack.from_dlpack.html#jax.dlpack.from_dlpack). I think PyArrow can support a PyArrow device :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3951103443


##########
python/pyarrow/array.pxi:
##########
@@ -2269,6 +2274,51 @@ cdef class Array(_PandasConvertible):
 
         return pyarrow_wrap_array(array)
 
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct an Array from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional

Review Comment:
   That would be outside the DLPack spec. Do you think its reasonable?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51122:
URL: https://github.com/apache/arrow/pull/51122#issuecomment-5495231412

   :warning: GitHub issue #39295 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3951149114


##########
cpp/src/arrow/buffer.cc:
##########
@@ -148,22 +148,13 @@ Result<std::shared_ptr<Buffer>> Buffer::ViewOrCopy(
   return MemoryManager::CopyBuffer(source, to);
 }
 
-class StlStringBuffer : public Buffer {
- public:
-  explicit StlStringBuffer(std::string data) : input_(std::move(data)) {
-    if (!input_.empty()) {
-      data_ = reinterpret_cast<const uint8_t*>(input_.c_str());
-      size_ = static_cast<int64_t>(input_.size());
-      capacity_ = size_;
-    }
+std::shared_ptr<Buffer> Buffer::FromString(std::string data) {
+  if (data.empty()) {
+    return std::shared_ptr<Buffer>{new Buffer()};
   }
 
- private:
-  std::string input_;
-};
-
-std::shared_ptr<Buffer> Buffer::FromString(std::string data) {
-  return std::make_shared<StlStringBuffer>(std::move(data));
+  auto size_in_bytes = static_cast<int64_t>(data.size());
+  return TakeOwnership(std::move(data), size_in_bytes);
 }

Review Comment:
   Buffer::FromString() now delegates to Buffer::TakeOwnership() with the default get_data(), which returns a mutable pointer for std::string. This makes FromString() produce a mutable buffer (and exposes std::string storage as writable), contradicting the API contract and potentially allowing unsafe mutation through MutableBuffer APIs.



##########
python/pyarrow/array.pxi:
##########
@@ -2269,6 +2274,51 @@ cdef class Array(_PandasConvertible):
 
         return pyarrow_wrap_array(array)
 
+    @staticmethod
+    def from_dlpack(x, /, *, device=None, copy=None):
+        """
+        Construct an Array from an object implementing the DLPack protocol.
+
+        Parameters
+        ----------
+        x : object
+            The input object containing array data, following the DLPack
+            protocol (has a ``__dlpack__`` method).
+        device : tuple[enum.Enum, int], optional
+            Designates where the resulting Array should reside, in the
+            format returned by :meth:`Array.__dlpack_device__`. When None,
+            the output Array occupies the same device as the source.
+            Default: None.
+        copy : bool, optional
+            Controls duplication behavior. True mandates copying; False
+            prohibits copying and raises ``BufferError`` if unavoidable;
+            None duplicates only when necessary. Default: None.
+
+        Returns
+        -------
+        Array
+            An Array housing the data from the input object, potentially
+            as a copy or view.
+        """
+        version = (DLPACK_VERSION.major, DLPACK_VERSION.minor)
+        pycapsule = x.__dlpack__(max_version=version, dl_device=device, copy=copy)
+        if not PyCapsule_CheckExact(pycapsule):
+            raise TypeError("DLPack producer did not return a PyCapsule")
+        cdef DLManagedTensorVersioned* ptr = <DLManagedTensorVersioned*>PyCapsule_GetPointer(
+            pycapsule, "dltensor_versioned")
+        if ptr == NULL:
+            raise ValueError(
+                'DLPack producer did not produce a "dltensor_versioned" PyCapsule')
+        # Mark the capsule as consumed so its destructor does not also invoke the deleter.
+        # ImportArrayVersionedFromDLPack will take ownership even if it errors (calling
+        # the deleter in that case).
+        PyCapsule_SetName(pycapsule, "used_dltensor_versioned")

Review Comment:
   PyCapsule_SetName() return value is ignored. If renaming the capsule fails, proceeding to import risks double-free (capsule destructor + Import* deleter) or leaving an unconsumed capsule state. Check the return code and raise before calling ImportArrayVersionedFromDLPack (and apply the same pattern in Tensor.from_dlpack).



##########
cpp/src/arrow/c/dlpack.h:
##########
@@ -105,4 +108,32 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Array>& arr);
 ARROW_EXPORT
 Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t);
 
+/// \brief Import a DLPack tensor as an Arrow Array.
+///
+/// Same restrictions on data types as `ExportArrayVersioned`, only row-major
+/// tensors are supported. Takes ownership of the `DLManagedTensorVersioned` in
+/// an error-safe fashion.
+///
+/// \param[in] raw DLPack tensor
+/// \param[in] copy Whether to copy the data instead of sharing it with the DLPack
+///            producer.
+/// \return An Arrow Array
+ARROW_EXPORT
+Result<std::shared_ptr<Array>> ImportArrayVersioned(DLManagedTensorVersioned* raw,
+                                                    bool copy);
+
+/// \brief Import a DLPack tensor as an Arrow Tensor.
+///
+/// Same restrictions on data types as `ExportTensorVersioned`.
+/// Takes ownership of the `DLManagedTensorVersioned` in an error-safe fashion.
+/// If the DLPack input is marked as readonly, this will produce an immutable tensor.
+///
+/// \param[in] raw Arrow array

Review Comment:
   ImportTensorVersioned() docstring incorrectly says `raw` is an "Arrow array". The parameter is a DLPack tensor (DLManagedTensorVersioned*), which matters for API consumers reading the header.



##########
python/pyarrow/array.pxi:
##########
@@ -4969,6 +5019,32 @@ cdef class FixedShapeTensorArray(ExtensionArray):
 
         return self.to_tensor().to_numpy()
 
+    @staticmethod
+    def from_tensor(Tensor tensor not None):
+        """

Review Comment:
   FixedShapeTensorArray.from_tensor() is a new public Python API but there doesn't appear to be Python-level test coverage for it (existing tests cover from_numpy_ndarray() but not from_tensor()). Adding a small test that round-trips Tensor -> FixedShapeTensorArray -> Tensor (including strided/permuted tensors) would help prevent regressions in shape/permutation handling.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3966858460


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +262,265 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ARROW_PREDICT_FALSE(ptr == nullptr)) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (ARROW_PREDICT_FALSE(out.ptr_->version.major != kVersion.major)) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", kVersion.major);
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim < 0)) {
+      return Status::Invalid("Invalid DLPack tensor: ndim must be >= 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().shape == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: shape must be non-null when ndim != 0");
+    }
+    // Null strides are handled as row major
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  /// Strides or empty span for old DLPack row-major convention.
+  std::span<const int64_t> strides() const {
+    if (auto strides = tensor().strides; strides != nullptr) {
+      return {strides, static_cast<std::size_t>(ndim())};
+    }
+    return {};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+  /// Number of element in this tensor's buffer.
+  ///
+  /// Possibly more elements than represented in the tensor for non-contiguous tensors.
+  ///
+  /// A zero dimensional tensor is a scalar, it holds a single element.
+  Result<int64_t> ComputeNumElements() const {
+    const auto strides = this->strides();
+    const auto shape = this->shape();
+    if (strides.size() > 0) {
+      // DLPack strides are in number of elements, so is the size we compute from them.
+      return internal::ComputeTensorSize(shape, strides, 1);
+    }
+    // DLPack <1.3 my set strides == nullptr for row major
+    return std::reduce(shape.begin(), shape.end(), int64_t{1}, std::multiplies{});
+  }
+
+  /// Number of bytes needed to store this tensor data.
+  Result<int64_t> ComputeNumBytes() const {
+    ARROW_ASSIGN_OR_RAISE(const auto nelements, ComputeNumElements());
+    int64_t nbytes = 0;
+    if (ARROW_PREDICT_FALSE(internal::MultiplyWithOverflow(
+            nelements, static_cast<int64_t>(byte_width()), &nbytes))) {
+      return Status::Invalid("Overflow computing DLPack tensor size in bytes.");
+    }
+    return nbytes;
+  }
+
+ private:
+  struct Deleter {
+    void operator()(pointer_type ptr) {
+      // Null is valid in DLPack spec
+      if (auto del = ptr->deleter) {
+        del(ptr);
+      }
+    }
+  };
+
+  /// Make a safe wrapper that will delete the resource in case of exception.
+  std::unique_ptr<value_type, Deleter> ptr_;
+
+  explicit CppDLTensor(pointer_type ptr) : ptr_(ptr) {}
+};
+
+Result<std::shared_ptr<FixedWidthType>> DataTypeFromDLPack(DLDataType dtype) {
+  if (dtype.lanes != 1) {
+    return Status::TypeError("Only type with one lane are supported.");
+  }
+
+  auto constexpr as_fw = [](auto dt) {
+    return std::static_pointer_cast<FixedWidthType>(std::move(dt));
+  };
+
+  switch (dtype.code) {
+    case kDLInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(int8());
+        case 16:
+          return as_fw(int16());
+        case 32:
+          return as_fw(int32());
+        case 64:
+          return as_fw(int64());
+        default:
+          return Status::Invalid("unsupported integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLUInt: {
+      switch (dtype.bits) {
+        case 8:
+          return as_fw(uint8());
+        case 16:
+          return as_fw(uint16());
+        case 32:
+          return as_fw(uint32());
+        case 64:
+          return as_fw(uint64());
+        default:
+          return Status::Invalid("unsupported unsigned integer bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    case kDLFloat: {
+      switch (dtype.bits) {
+        case 16:
+          return as_fw(float16());
+        case 32:
+          return as_fw(float32());
+        case 64:
+          return as_fw(float64());
+        default:
+          return Status::Invalid("unsupported float bit width ",
+                                 static_cast<int>(dtype.bits));
+      }
+    }
+    default: {
+      return Status::Invalid("unsupported DLPack type ", static_cast<int>(dtype.code));
+    }
+  }
+}
+
+Result<std::vector<int64_t>> StridesInBytes(std::span<const int64_t> strides,
+                                            int64_t byte_width) {
+  std::vector<int64_t> out{};
+  out.reserve(strides.size());
+  for (const auto& s : strides) {
+    int64_t stride_bytes = 0;
+    if (ARROW_PREDICT_FALSE(
+            internal::MultiplyWithOverflow(s, byte_width, &stride_bytes))) {
+      return Status::Invalid("Overflow computing DLPack tensor stride in bytes.");
+    }
+    out.push_back(stride_bytes);
+  }
+  return out;
+}
+
+Result<std::shared_ptr<Buffer>> ImportBuffer(CppDLTensor&& dl) {
+  ARROW_ASSIGN_OR_RAISE(const int64_t nbytes, dl.ComputeNumBytes());
+
+  // DLPack mandates a null data pointer when the tensor holds no element, so there is
+  // neither anything to share nor to copy.
+  uint8_t* data =
+      (nbytes == 0) ? nullptr : dl.data_as<uint8_t>() + dl.tensor().byte_offset;

Review Comment:
   `ImportBuffer()` does pointer arithmetic on `dl.tensor().data` when `nbytes > 0`, but doesn’t validate that the DLPack producer actually provided a non-null data pointer. If a producer returns `data == nullptr` for a non-empty tensor, `dl.data_as<uint8_t>() + byte_offset` is undefined behavior. Add an explicit check and fail fast with a clear error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #51122:
URL: https://github.com/apache/arrow/pull/51122#discussion_r3966278046


##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -248,4 +262,266 @@ Result<DLDevice> ExportDevice(const std::shared_ptr<Tensor>& t) {
   return ExportDeviceImpl(t);
 }
 
+/***************
+ *  Consumers  *
+ ***************/
+
+namespace {
+
+class CppDLTensor {
+ public:
+  using value_type = DLManagedTensorVersioned;
+  using pointer_type = value_type*;
+
+  static Result<CppDLTensor> TakeOwnership(pointer_type ptr) {
+    if (ARROW_PREDICT_FALSE(ptr == nullptr)) {
+      return Status::Invalid("Received null pointer.");
+    }
+    // Create the wrapper before checking the version as the spec mandates that the
+    // deleter MUST be called on version major mismatch.
+    auto out = CppDLTensor(ptr);
+    if (ARROW_PREDICT_FALSE(out.ptr_->version.major != kVersion.major)) {
+      return Status::Invalid("Unsupported DLPack major version ", out.ptr_->version.major,
+                             ", expected ", kVersion.major);
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim < 0)) {
+      return Status::Invalid("Invalid DLPack tensor: ndim must be >= 0");
+    }
+    if (ARROW_PREDICT_FALSE(out.tensor().ndim != 0 && out.tensor().shape == nullptr)) {
+      return Status::Invalid(
+          "Invalid DLPack tensor: shape must be non-null when ndim != 0");
+    }
+    // Null strides are handled as row major
+    return out;
+  }
+
+  const DLTensor& tensor() const { return ptr_->dl_tensor; }
+
+  int64_t ndim() const {
+    DCHECK_GE(tensor().ndim, 0);
+    return tensor().ndim;
+  }
+
+  template <typename T>
+  T* data_as() {
+    return static_cast<T*>(tensor().data);
+  }
+
+  std::span<const int64_t> shape() const {
+    return {tensor().shape, static_cast<std::size_t>(ndim())};
+  }
+
+  /// Strides or empty span for old DLPack row-major convention.
+  std::span<const int64_t> strides() const {
+    if (auto strides = tensor().strides; strides != nullptr) {
+      return {strides, static_cast<std::size_t>(ndim())};
+    }
+    return {};
+  }
+
+  bool flag_is_set(uint8_t bits) const { return (ptr_->flags & bits) == bits; }
+
+  bool is_readonly() const { return flag_is_set(DLPACK_FLAG_BITMASK_READ_ONLY); }
+
+  int32_t byte_width() const { return tensor().dtype.bits / 8; }
+
+  /// Number of element in this tensor's buffer.
+  ///
+  /// Possibly more elements than represented in the tensor for non-contiguous tensors.
+  Result<int64_t> ComputeNumElements() const {
+    if (ndim() == 0) {
+      return 0;
+    }

Review Comment:
   `CppDLTensor::ComputeNumElements()` returns 0 when `ndim() == 0`, but DLPack allows `ndim == 0` to represent a scalar (shape/strides may be NULL) and that scalar still occupies 1 element. Returning 0 causes scalar tensors to be treated as empty and imported with a 0-byte buffer.
   
   Consider returning 1 element for `ndim() == 0` (consistent with other Arrow code paths that treat `ndim == 0` as a single value).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #51122:
URL: https://github.com/apache/arrow/pull/51122#issuecomment-5615373414

   @AntoinePrv I see the last comment from @pitrou has been addressed and feel this is ready to be merged if I am not missing anything.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-39295: [C++][Python] ConsumingDLPack on Arrays and Tensor [arrow]

Posted by "AntoinePrv (via GitHub)" <gi...@apache.org>.
AntoinePrv commented on PR #51122:
URL: https://github.com/apache/arrow/pull/51122#issuecomment-5584723939

   @pitrou @AlenkaF this is looking good.
   
   One missing thing is `FixedShapeScalarTensor.from_dlpack`, that I took the liberty to omit since it did not have any other `from_tensor`/`from_numpy` methods.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org