You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AlenkaF (via GitHub)" <gi...@apache.org> on 2023/10/26 12:43:05 UTC
[PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
AlenkaF opened a new pull request, #38472:
URL: https://github.com/apache/arrow/pull/38472
### Rationale for this change
DLPack is selected for Array API protocol so it is important to have it implemented for PyArrow Arrays also. This is possible for primitive type arrays with no validity buffer. Device support is not in scope of this PR (CPU only).
### What changes are included in this PR?
`__dlpack__` method on the base Array class with a check for number of buffers (only one buffer for the data).
### Are these changes tested?
Yes, tests are added to `test_array.py`.
### Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1425029258
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+#include "arrow/type_traits.h"
+
+namespace arrow::dlpack {
+
+namespace {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type.bit_width();
+ switch (type.id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ return dtype;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ return dtype;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ return dtype;
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ type.ToString());
+ }
+}
+
+} // namespace
+struct ManagerCtx {
+ std::shared_ptr<ArrayData> ref;
+ DLManagedTensor tensor;
+};
+
+Result<DLManagedTensor*> ExportArray(const std::shared_ptr<Array>& arr) {
+ // Define DLDevice struct nad check if array type is supported
+ // by the DLPack protocol at the same time. Raise TypeError if not.
+ // Supported data types: int, uint, float with no validity buffer.
+ ARROW_ASSIGN_OR_RAISE(auto device, ExportDevice(arr))
+
+ // Define the DLDataType struct
+ const DataType* arrow_type = arr->type().get();
+ ARROW_ASSIGN_OR_RAISE(auto dlpack_type, GetDLDataType(*arrow_type));
+
+ // Create ManagerCtx with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ std::unique_ptr<ManagerCtx> ctx(new ManagerCtx);
Review Comment:
Unless that `delete` call can then be replaced with a unique_ptr `reset()` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1411039504
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
Will do, thanks!! (Just finishing up the docs and then I will mark the PR ready for review and add a ping ❤️ )
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
Will do, thanks!! (Just finishing up the docs and then I will mark the PR ready for review and add a ping ❤️ )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1835497948
I am not sure about the failing builds:
- [AMD64 Ubuntu 22.04 C++ ASAN UBSAN](https://github.com/apache/arrow/actions/runs/7050402985/job/19191028912?pr=38472#step:6:3004) is failing due to a memory leak in the dlpack test with (not sure what is going on but am playing around with thest to see if I can fix it):
```
=================================================================
==5316==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 1144 byte(s) in 11 object(s) allocated from:
#0 0x55959388606d in operator new(unsigned long) (/build/cpp/debug/arrow-dlpack-test+0x10506d) (BuildId: 6eecfc46ecb410ea9bef89c63c4a74ec4e3333cc)
#1 0x7f47140c9efc in arrow::dlpack::ExportArray(std::shared_ptr<arrow::Array> const&, DLManagedTensor**) /arrow/cpp/src/arrow/c/dlpack.cc:87:29
#2 0x55959388e691 in arrow::dlpack::$_0::operator()(std::shared_ptr<arrow::Array> const&, std::shared_ptr<arrow::DataType>, DLDataTypeCode, long) const /arrow/cpp/src/arrow/c/dlpack_test.cc:41:3
#3 0x55959388ac56 in arrow::dlpack::TestExportArray_TestSupportedArray_Test::TestBody() /arrow/cpp/src/arrow/c/dlpack_test.cc:85:5
#4 0x7f47270e67fe in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.1500+0xda47fe) (BuildId: d0574061a61e73aebf3ce34a43c933ba7636c822)
```
- [Lint C++, Python, R, Docker, RAT](https://github.com/apache/arrow/actions/runs/7050402985/job/19191028912?pr=38472#logs) is failing but I am not sure if it is the old R linter error and I need to rebase. So will try with that.
- Other failing builds already have an issue opened and resolved, so will see if rebase helped: https://github.com/apache/arrow/issues/39003
Otherwise I think this PR is ready for review and will mark it as such after rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1416907727
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
After calling `release()`, I think the unique ptr itself becomes null? (https://en.cppreference.com/w/cpp/memory/unique_ptr/release) So then you can't use `ctx` afterwards to refer to it, but you have to use the raw pointer instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "kkraus14 (via GitHub)" <gi...@apache.org>.
kkraus14 commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1416086378
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
I do not know the rationale for this unfortunately.
@rgommers do you happen to know the rationale for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1409006731
##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -1199,6 +1199,60 @@ cdef extern from "arrow/api.h" namespace "arrow" nogil:
shared_ptr[CScalar] MakeNullScalar(shared_ptr[CDataType] type)
+cdef extern from "arrow/dlpack_structure.h" nogil:
+ cdef enum DLDeviceType:
Review Comment:
I think the structs and `DLDeviceType` all have to stay as they are interconnected and so I only removed `DLDataTypeCode`: https://github.com/apache/arrow/pull/38472/commits/3c35fb37ceeecbbd615daeae030060e1b2e20edf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1404011606
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+DLDataType getDLDataType(const Array& arr) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = arr.type()->bit_width();
+ switch (arr.type()->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ break;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ break;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ break;
+ case Type::BOOL:
+ dtype.code = DLDataTypeCode::kDLBool;
+ break;
+ default:
+ // TODO
+ break;
+ }
+ return dtype;
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+DLManagedTensor* toDLPack(const Array& arr) {
+ std::shared_ptr<ArrayData> array_ref = arr.data();
Review Comment:
Hm, I think I remember now why I chose `.data()` and not `.Slice()`. When accessing the buffer address of the data for `dl_tensor.data` pointer we also go through `array.data()` ([data_](https://github.com/apache/arrow/blob/5306b32370b4c5f5137defe90516a280d9c71796/cpp/src/arrow/array/array_base.h#L254)) so it looked to me like `ArrayData` is an underlying `Array` structure and it would be safe to use it as a reference.
@pitrou what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1408870138
##########
python/pyarrow/tests/test_array.py:
##########
@@ -3546,3 +3548,122 @@ def test_run_end_encoded_from_buffers():
with pytest.raises(ValueError):
pa.RunEndEncodedArray.from_buffers(ree_type, length, buffers,
1, offset, children)
+
+
+def PyCapsule_IsValid(capsule, name):
+ return ctypes.pythonapi.PyCapsule_IsValid(ctypes.py_object(capsule), name) == 1
+
+
+@pytest.mark.parametrize(
+ ('value_type', 'np_type'),
+ [
+ (pa.uint8(), np.uint8),
+ (pa.uint16(), np.uint16),
+ (pa.uint32(), np.uint32),
+ (pa.uint64(), np.uint64),
+ (pa.int8(), np.int8),
+ (pa.int16(), np.int16),
+ (pa.int32(), np.int32),
+ (pa.int64(), np.int64),
+ (pa.float32(), np.float32),
+ (pa.float64(), np.float64),
+ ]
+)
+def test_dlpack(value_type, np_type):
+ if Version(np.__version__) < Version("1.24.0"):
+ pytest.skip("No dlpack support in numpy versions older than 1.22.0, "
+ "strict keyward in assert_array_equal added in numpy version "
+ "1.24.0")
+
+ arr = pa.array([1, 2, 3], type=value_type)
+ DLTensor = arr.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([1, 2, 3], dtype=np_type)
+ result = np.from_dlpack(arr)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+ arr_sliced = arr.slice(1, 1)
+ DLTensor = arr_sliced.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([2], dtype=np_type)
+ result = np.from_dlpack(arr_sliced)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+ arr_sliced = arr.slice(0, 1)
+ DLTensor = arr_sliced.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([1], dtype=np_type)
+ result = np.from_dlpack(arr_sliced)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+ arr_sliced = arr.slice(1)
+ DLTensor = arr_sliced.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([2, 3], dtype=np_type)
+ result = np.from_dlpack(arr_sliced)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+ arr_zero = pa.array([], type=value_type)
+ DLTensor = arr_zero.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([], dtype=np_type)
+ result = np.from_dlpack(arr_zero)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+
+def test_dlpack_float_16():
+ if Version(np.__version__) < Version("1.24.0"):
+ pytest.skip("No dlpack support in numpy versions older than 1.22.0, "
+ "strict keyward in assert_array_equal added in numpy version "
+ "1.24.0")
+
+ expected = np.array([1, 2, 3], dtype=np.float16)
+ arr = pa.array(expected, type=pa.float16())
+ DLTensor = arr.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ result = np.from_dlpack(arr)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+
+def test_dlpack_not_supported():
+ if Version(np.__version__) < Version("1.22.0"):
+ pytest.skip("No dlpack support in numpy versions older than 1.22.0.")
+
+ with pytest.raises(TypeError, match="Can only use __dlpack__ "
+ "on arrays with no validity buffer."):
+ arr = pa.array([1, None, 3])
Review Comment:
Can you put the array creation before the `with` context? The message match will ensure we are not testing a wrong error, but ideally we only put in the context the part that raises the error we want to test
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError(
+ "Can only use __dlpack__ on arrays with no validity buffer.");
Review Comment:
```suggestion
"Can only use __dlpack__ on arrays with no nulls.");
```
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
So in what you implemented now, it's still the ExportArray function that allocates the struct (so the memory is not owned by the consumer or the capsule itself on the python side of the interface), right? But the struct is stored in `DLMTensorCtx` (the private data), and this gets deallocated by the deleter, which gets called by the consumer at the end (or at destruction of the capsule if not consumed), so that ensures it gets cleaned up?
##########
python/pyarrow/tests/test_array.py:
##########
@@ -3546,3 +3548,122 @@ def test_run_end_encoded_from_buffers():
with pytest.raises(ValueError):
pa.RunEndEncodedArray.from_buffers(ree_type, length, buffers,
1, offset, children)
+
+
+def PyCapsule_IsValid(capsule, name):
+ return ctypes.pythonapi.PyCapsule_IsValid(ctypes.py_object(capsule), name) == 1
+
+
+@pytest.mark.parametrize(
+ ('value_type', 'np_type'),
+ [
+ (pa.uint8(), np.uint8),
+ (pa.uint16(), np.uint16),
+ (pa.uint32(), np.uint32),
+ (pa.uint64(), np.uint64),
+ (pa.int8(), np.int8),
+ (pa.int16(), np.int16),
+ (pa.int32(), np.int32),
+ (pa.int64(), np.int64),
+ (pa.float32(), np.float32),
+ (pa.float64(), np.float64),
+ ]
+)
+def test_dlpack(value_type, np_type):
+ if Version(np.__version__) < Version("1.24.0"):
+ pytest.skip("No dlpack support in numpy versions older than 1.22.0, "
+ "strict keyward in assert_array_equal added in numpy version "
+ "1.24.0")
+
+ arr = pa.array([1, 2, 3], type=value_type)
+ DLTensor = arr.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([1, 2, 3], dtype=np_type)
+ result = np.from_dlpack(arr)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+ arr_sliced = arr.slice(1, 1)
+ DLTensor = arr_sliced.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([2], dtype=np_type)
+ result = np.from_dlpack(arr_sliced)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+ arr_sliced = arr.slice(0, 1)
+ DLTensor = arr_sliced.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([1], dtype=np_type)
+ result = np.from_dlpack(arr_sliced)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+ arr_sliced = arr.slice(1)
+ DLTensor = arr_sliced.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([2, 3], dtype=np_type)
+ result = np.from_dlpack(arr_sliced)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+ arr_zero = pa.array([], type=value_type)
+ DLTensor = arr_zero.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([], dtype=np_type)
+ result = np.from_dlpack(arr_zero)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+
+def test_dlpack_float_16():
Review Comment:
Is there a reason this one is not included in the parametrized test above? (slicing isn't tested this way)
##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -1199,6 +1199,60 @@ cdef extern from "arrow/api.h" namespace "arrow" nogil:
shared_ptr[CScalar] MakeNullScalar(shared_ptr[CDataType] type)
+cdef extern from "arrow/dlpack_structure.h" nogil:
+ cdef enum DLDeviceType:
Review Comment:
FWIW this isn't yet pushed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1423896057
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,49 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, pointing to a DLManagedTensor.
+ """
+ cdef CResult[DLManagedTensor*] c_dlm_tensor
+
+ if stream is None:
+ c_dlm_tensor = ExportToDLPack(pyarrow_unwrap_array(self))
+ dlm_tensor = GetResultValue(c_dlm_tensor)
+
+ return PyCapsule_New(dlm_tensor, 'dltensor', dlpack_pycapsule_deleter)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
+
+ Returns
+ -------
+ tuple : Tuple[DLDeviceType, int]
Review Comment:
Decided to do the same as Numpy at the end - returning `tuple[int, int]` =)
No need to complicate if it is not necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "kkraus14 (via GitHub)" <gi...@apache.org>.
kkraus14 commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1373958835
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+
+
+cdef void deleter(DLManagedTensor* tensor) with gil:
+ if tensor.manager_ctx is NULL:
+ return
+ free(tensor.dl_tensor.shape)
+ cpython.Py_DECREF(<Array>tensor.manager_ctx)
+ tensor.manager_ctx = NULL
+ free(tensor)
+
+
+cpdef object to_dlpack(Array arr) except *:
+ cdef DLManagedTensor* dlm_tensor = <DLManagedTensor*> malloc(sizeof(DLManagedTensor))
+
+ cdef DLTensor* dl_tensor = &dlm_tensor.dl_tensor
+ cdef intptr_t data_ptr = arr.buffers()[1].address
+ dl_tensor.data = <void*> data_ptr
+ dl_tensor.ndim = 1
+ cdef int64_t* shape = <int64_t*> malloc(sizeof(int64_t))
+ shape[0] = len(arr)
+ dl_tensor.shape = shape
+
+ dl_tensor.strides = NULL
+ dl_tensor.byte_offset = 0
+
+ cdef DLDevice* device = &dl_tensor.device
+ device.device_type = kDLCPU
+ device.device_id = 0
+
+ cdef DLDataType* dtype = &dl_tensor.dtype
+ if arr.type in [uint8(), uint16(), uint32(), uint64()]:
+ dtype.code = kDLUInt
+ elif arr.type in [int8(), int16(), int32(), int64()]:
+ dtype.code = kDLInt
+ elif arr.type in [float16(), float32(), float64()]:
+ dtype.code = kDLFloat
+ elif arr.type == bool_():
+ dtype.code = kDLBool
+
+ dtype.lanes = <uint16_t>1
+ dtype.bits = <uint8_t>arr.type.bit_width
+
+ dlm_tensor.manager_ctx = <void*>arr
+ cpython.Py_INCREF(arr)
Review Comment:
+1 to moving this to C++ and wrapping the result in PyArrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1373332663
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
Review Comment:
You seem to have got this from CuPy?
According to the spec, there are a couple more operations needed. See the C sample here:
https://dmlc.github.io/dlpack/latest/python_spec.html#implementation
Also, I've filed https://github.com/dmlc/dlpack/issues/130 about this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1822857464
TODO: C++ tests, tests with arrays with offsets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1402461018
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
It seems fragile to return a raw pointer. Instead, you could probably take an input pointer allocated by the caller. cc @kkraus14 for confirmation.
```suggestion
Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor* out);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417044907
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
Note that it is about the data pointer here, while your link to the dlpack code points to the strides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1418783538
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ cdef PyObject* err_type
+ cdef PyObject* err_value
+ cdef PyObject* err_traceback
+
+ # Do nothing if the capsule has been consumed
+ if cpython.PyCapsule_IsValid(dltensor, "used_dltensor"):
+ return
+
+ # An exception may be in-flight, we must save it in case
+ # we create another one
+ cpython.PyErr_Fetch(&err_type, &err_value, &err_traceback)
+
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+ else:
+ cpython.PyErr_WriteUnraisable(dltensor)
Review Comment:
https://github.com/apache/arrow/pull/38472/commits/d232de444793a2f9afcb06ccaf4c354b2a969638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "kkraus14 (via GitHub)" <gi...@apache.org>.
kkraus14 commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1413230389
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
Can we use a smart ptr here and then handle it in the deleter instead of using raw `new` / `delete` calls?
As it stands for example if someone passes a GPU buffer, we allocate this and then raise later which ends up leaking this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1415621962
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
Yes, in the DLPack Python spec just above the Warning at the end of the page: https://dmlc.github.io/dlpack/latest/python_spec.html#syntax-for-data-interchange-with-dlpack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417097381
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,40 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, containing a DLPackManagedTensor.
+ """
+ if stream is None:
+ return to_dlpack(self)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
Review Comment:
Yes, I have (added to the docs examples - those are not being tested yet, right?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "leofang (via GitHub)" <gi...@apache.org>.
leofang commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417771530
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
Indeed. I will send a PR to DLPAck repo later. Good suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1424980116
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+#include "arrow/type_traits.h"
+
+namespace arrow::dlpack {
+
+namespace {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type.bit_width();
+ switch (type.id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ return dtype;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ return dtype;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ return dtype;
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ type.ToString());
+ }
+}
+
+} // namespace
+struct ManagerCtx {
+ std::shared_ptr<ArrayData> ref;
+ DLManagedTensor tensor;
+};
+
+Result<DLManagedTensor*> ExportArray(const std::shared_ptr<Array>& arr) {
+ // Define DLDevice struct nad check if array type is supported
+ // by the DLPack protocol at the same time. Raise TypeError if not.
+ // Supported data types: int, uint, float with no validity buffer.
+ ARROW_ASSIGN_OR_RAISE(auto device, ExportDevice(arr))
+
+ // Define the DLDataType struct
+ const DataType* arrow_type = arr->type().get();
+ ARROW_ASSIGN_OR_RAISE(auto dlpack_type, GetDLDataType(*arrow_type));
+
+ // Create ManagerCtx with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ std::unique_ptr<ManagerCtx> ctx(new ManagerCtx);
+ ctx->ref = array_ref;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
+ if (arr->length() == 0) {
+ ctx->tensor.dl_tensor.data = NULL;
+ } else if (arr->offset() > 0) {
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ARROW_ASSIGN_OR_RAISE(auto sliced_buffer,
+ SliceBufferSafe(array_ref->buffers[1], start));
+ ctx->tensor.dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(sliced_buffer->address()));
+ } else {
+ ctx->tensor.dl_tensor.data = const_cast<void*>(
+ reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+ }
+
+ ctx->tensor.dl_tensor.device = device;
+ ctx->tensor.dl_tensor.ndim = 1;
+ ctx->tensor.dl_tensor.dtype = dlpack_type;
+ ctx->tensor.dl_tensor.shape = const_cast<int64_t*>(&array_ref->length);
+ ctx->tensor.dl_tensor.strides = NULL;
+ ctx->tensor.dl_tensor.byte_offset = 0;
+
+ ctx->tensor.manager_ctx = ctx.get();
+ ctx->tensor.deleter = [](struct DLManagedTensor* self) {
+ delete reinterpret_cast<ManagerCtx*>(self->manager_ctx);
+ };
+ return &ctx.release()->tensor;
+}
+
+Result<DLDevice> ExportDevice(const std::shared_ptr<Array>& arr) {
+ // Check if array is supported by the DLPack protocol.
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use DLPack on arrays with no nulls.");
+ }
+ const DataType* arrow_type = arr->type().get();
+ if (arrow_type->id() == Type::BOOL) {
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ }
+ if (!is_integer(arrow_type->id()) && !is_floating(arrow_type->id())) {
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ arrow_type->ToString());
+ }
+
+ // Define DLDevice struct
+ DLDevice device;
+ if (arr->data()->buffers[1]->device_type() == DeviceAllocationType::kCPU) {
Review Comment:
Yes, that should be safe. There is a test in Python that checks that: https://github.com/apache/arrow/pull/38472/files#diff-1368d4aa013b54f08fa17e63822e01f6a879ae0ddafb3c1ba6e3e919750c5d7eR113
```python
arr = pa.array([])
with pytest.raises(TypeError, match="DataType is not compatible with DLPack spec"):
np.from_dlpack(arr)
```
but I will add it to C++ also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "kkraus14 (via GitHub)" <gi...@apache.org>.
kkraus14 commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1402493364
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
Looks like there's a mix of how other projects do it. Returning a raw pointer:
* https://github.com/rapidsai/cudf/blob/cfa2d513667edabda6c4487f15f251f757f0c94d/cpp/include/cudf/interop.hpp#L88
* https://github.com/pytorch/pytorch/blob/d6578b36781b0402163e230e8838a802bf51ba0f/aten/src/ATen/DLConvertor.h#L14
* https://github.com/tensorflow/tensorflow/blob/44e4863a120282b43f4f2e4eca4312935c881522/tensorflow/c/eager/dlpack.h#L28
Taking an input pointer allocated by the caller:
* https://github.com/apache/tvm/blob/bce82432f98a93356f393166ae50834cb177caa9/include/tvm/runtime/c_runtime_api.h#L562
I think @pitrou's proposal here makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1403580873
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+DLDataType getDLDataType(const Array& arr) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = arr.type()->bit_width();
+ switch (arr.type()->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ break;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ break;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ break;
+ case Type::BOOL:
+ dtype.code = DLDataTypeCode::kDLBool;
+ break;
+ default:
+ // TODO
+ break;
+ }
+ return dtype;
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+DLManagedTensor* toDLPack(const Array& arr) {
+ std::shared_ptr<ArrayData> array_ref = arr.data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+ dlm_tensor->dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+
+ DLDevice ctx;
+ ctx.device_id = 0;
+ ctx.device_type = DLDeviceType::kDLCPU;
Review Comment:
This makes sense yes. We planned to support only CPU device at first. But I should not hardcode it like I did but validate they they are in fact CPU buffers and raise if not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [C++][Python] DLPack implementation for Arrow Arrays (producer) [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1864483153
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 6c326db6a5686a78bc77be662b61236ddbfc66dc.
There was 1 benchmark result indicating a performance regression:
- Commit Run on `ursa-i9-9960x` at [2023-12-20 02:11:37Z](https://conbench.ursa.dev/compare/runs/d6c8ed4639714492b425c0af4ff9ce51...b2fc9c4d3a464886865427730f14c36c/)
- [`tpch` (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-05, scale_factor=1](https://conbench.ursa.dev/compare/benchmarks/0658237af44f78d68000ea4a94bdf51c...0658261406fc74228000e433e12daced)
The [full Conbench report](https://github.com/apache/arrow/runs/19825300063) has more details. It also includes information about 10 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1402456236
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
This could go into `arrow/c/`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1402089809
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
@pitrou any advice on where to put this in the C++ codebase? Just as top-level files in /arrow, or does it fit better somewhere else?
##########
cpp/src/arrow/dlpack_structure.h:
##########
Review Comment:
Small suggestion to rename this to `dlpack_abi.h`
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+DLDataType getDLDataType(const Array& arr) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = arr.type()->bit_width();
+ switch (arr.type()->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ break;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ break;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ break;
+ case Type::BOOL:
+ dtype.code = DLDataTypeCode::kDLBool;
+ break;
+ default:
+ // TODO
+ break;
+ }
+ return dtype;
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+DLManagedTensor* toDLPack(const Array& arr) {
Review Comment:
If we want to have this usable from C++ as well, I think we should add some validation of the input Array here? (i.e. checking it has no nulls; that it is a primitive type will implicitly be done by getDLDataType)
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+DLDataType getDLDataType(const Array& arr) {
Review Comment:
You can probably pass the type directory here, since that's the only thing needed
##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -1199,6 +1199,60 @@ cdef extern from "arrow/api.h" namespace "arrow" nogil:
shared_ptr[CScalar] MakeNullScalar(shared_ptr[CDataType] type)
+cdef extern from "arrow/dlpack_structure.h" nogil:
+ cdef enum DLDeviceType:
Review Comment:
I think this can be removed now (or at least part of it) now the bulk of the implementation doesn't live in cython?
##########
python/pyarrow/tests/test_array.py:
##########
@@ -3546,3 +3548,59 @@ def test_run_end_encoded_from_buffers():
with pytest.raises(ValueError):
pa.RunEndEncodedArray.from_buffers(ree_type, length, buffers,
1, offset, children)
+
+
+def PyCapsule_IsValid(capsule, name):
+ return ctypes.pythonapi.PyCapsule_IsValid(ctypes.py_object(capsule), name) == 1
+
+
+@pytest.mark.parametrize(
+ 'value_type',
+ [pa.uint8(), pa.uint32(), pa.int16(), pa.float32()]
+)
+def test_dlpack(value_type):
+ if Version(np.__version__) < Version("1.22.0"):
+ pytest.skip("No dlpack support in numpy versions older than 1.22.0.")
+ arr = pa.array([1, 2, 3], type=value_type)
+
+ DLTensor = arr.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+
+ expected = np.array([1, 2, 3])
+ result = np.from_dlpack(arr)
+ np.testing.assert_array_equal(result, expected)
Review Comment:
`assert_array_equal` doesn't require the dtypes to be the same? We should also assert that part
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ cdef PyObject* err_type
+ cdef PyObject* err_value
+ cdef PyObject* err_traceback
+
+ if cpython.PyCapsule_IsValid(dltensor, "used_dltensor"):
+ return
+
+ cpython.PyErr_Fetch(&err_type, &err_value, &err_traceback)
Review Comment:
Can you add some comments about why this fetch/restore is done? (I see in the example on https://dmlc.github.io/dlpack/latest/python_spec.html there are some comments)
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
Review Comment:
Can you expand this doc note? (parameter section, maybe point to the DLPack webpage, restrictions about what the Array can be, ..)
##########
cpp/src/arrow/dlpack_structure.h:
##########
Review Comment:
Can you also add to the comment the exact commit from which you copied it from the repo? (eg see https://github.com/numpy/numpy/blob/main/numpy/_core/src/common/dlpack/dlpack.h)
##########
python/pyarrow/array.pxi:
##########
@@ -21,6 +21,8 @@ import os
import warnings
from cython import sizeof
+include "_dlpack.pxi"
Review Comment:
You can move this to lib.pyx where the other includes live
##########
python/pyarrow/tests/test_array.py:
##########
@@ -3546,3 +3548,59 @@ def test_run_end_encoded_from_buffers():
with pytest.raises(ValueError):
pa.RunEndEncodedArray.from_buffers(ree_type, length, buffers,
1, offset, children)
+
+
+def PyCapsule_IsValid(capsule, name):
+ return ctypes.pythonapi.PyCapsule_IsValid(ctypes.py_object(capsule), name) == 1
+
+
+@pytest.mark.parametrize(
+ 'value_type',
+ [pa.uint8(), pa.uint32(), pa.int16(), pa.float32()]
Review Comment:
I suppose you can expand this parametrization to every single supported dtype
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* toDLPack(const Array& arr);
Review Comment:
Small nit, but following the namin scheme from the C Data bridge, we can maybe call this "ExportToDLPack"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1831505761
Revision: 09573789ba3e2a835254691c0267206021a5878d
Submitted crossbow builds: [ursacomputing/crossbow @ actions-7886714998](https://github.com/ursacomputing/crossbow/branches/all?query=actions-7886714998)
|Task|Status|
|----|------|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/7030376310/job/19129799897)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1835602469
> * [AMD64 Ubuntu 22.04 C++ ASAN UBSAN](https://github.com/apache/arrow/actions/runs/7050402985/job/19191028912?pr=38472#step:6:3004) is failing due to a memory leak in the dlpack test with (not sure what is going on but am playing around with thest to see if I can fix it):
I assume this is because we don't yet have a consumer, and in those C++ tests you are exporting the array without consuming it. In that case you will have to call the deleter explicitly (in the python tests, by consuming it by numpy, it's numpy that will ensure to call the deleter when the numpy array goes out of scope)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1835702976
@pitrou I think this is ready for review =)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "rgommers (via GitHub)" <gi...@apache.org>.
rgommers commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417350931
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
That's because most of it wasn't new content, but extensively reviewed over several PRs, including by the lead DLPack maintainer.
Digging into the largest of those PRs did help me find the relevant data pointer comment: https://github.com/data-apis/array-api/pull/106#discussion_r557091538:
"_it'd be also nice to note if the array is of zero size, in the DLTensor its data pointer should set to nullptr (or 0). In the events where strides are not specified, this is the only effective way to inform the consumer that a zero-size array is coming. Numba/CuPy received a few bug reports when CAI v0/v1 were not clear about this, thus leading to the v2 revision._"
I'm not sure that the above is actually true, because there is a `shape` field as well. But checking `data` for being NULL is perhaps easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "leofang (via GitHub)" <gi...@apache.org>.
leofang commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417451657
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
I think the convention came from the old days of CUDA Array Interface where either `None` or `0` could be used, but I guess it doesn't really matter to a C interface. Indeed, `shape` can be used to check whether an array is 0-size, but why bother asking the consumer to do an extra check when the producer already has this knowledge?
From the ergonomics perspective, we know the pointer to a 0-size array is meaningless and shouldn't be used, and all libraries handle this case differently (ex: NumPy would return a pointer address that likely comes from its private memory reservation, while CuPy returns null), so setting it to 0 universally was the safest thing to align all libraries and tell the consumer "do not use the pointer."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417046322
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
@rgommers
> It doesn't mandate that it must be NULL, but rather that it must be "NULL or 0".
0 in a pointer context means NULL to me, so I'm not sure which distinction you're drawing here. I might miss a subtlety, but the spec should probably be clearer if it doesn't mean "must be NULL".
> even though in theory any values for strides works
The highlighted text talks about _the data pointer_, not about strides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1373354245
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+
+
+cdef void deleter(DLManagedTensor* tensor) with gil:
+ if tensor.manager_ctx is NULL:
+ return
+ free(tensor.dl_tensor.shape)
+ cpython.Py_DECREF(<Array>tensor.manager_ctx)
+ tensor.manager_ctx = NULL
+ free(tensor)
+
+
+cpdef object to_dlpack(Array arr) except *:
+ cdef DLManagedTensor* dlm_tensor = <DLManagedTensor*> malloc(sizeof(DLManagedTensor))
Review Comment:
Hmm, according to the latest spec contents, it seems we should use `DLManagedTensorVersioned` instead?
https://dmlc.github.io/dlpack/latest/c_api.html#_CPPv424DLManagedTensorVersioned
> This is the current standard DLPack exchange data structure.
Though [neither NumPy](https://github.com/cupy/cupy/blob/main/cupy/_core/dlpack.pyx) [nor CuPy](https://github.com/cupy/cupy/blob/main/cupy/_core/dlpack.pyx) seem to handle it.
@kkraus14 Do you have any insight about the status of the new versioned DLPack ABI? Is it just WIP?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "kkraus14 (via GitHub)" <gi...@apache.org>.
kkraus14 commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1373958622
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+
+
+cdef void deleter(DLManagedTensor* tensor) with gil:
+ if tensor.manager_ctx is NULL:
+ return
+ free(tensor.dl_tensor.shape)
+ cpython.Py_DECREF(<Array>tensor.manager_ctx)
+ tensor.manager_ctx = NULL
+ free(tensor)
+
+
+cpdef object to_dlpack(Array arr) except *:
+ cdef DLManagedTensor* dlm_tensor = <DLManagedTensor*> malloc(sizeof(DLManagedTensor))
Review Comment:
As far as I know it was added somewhat recently (Jan 2023) and hasn't been widely adopted yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1849656892
> (Can't see the reason for the linter build to keep failing though ...)
```
INFO:archery:Running Docker linter
apache-rat license violation: cpp/src/arrow/c/dlpack_abi.h
```
I don't know if it is fine to add this file to `dev/release/rat_exclude_files.txt` to ignore it in the license check, or whether we should add the standard license header to this file. If it's considered "third party work" (given we simply copied it), then I assume the first, but then we need to add a note to NOTICE.txt?
Although for header files that are _meant_ to be included, it's maybe not needed to consider them as third party code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1402159784
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+DLDataType getDLDataType(const Array& arr) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = arr.type()->bit_width();
+ switch (arr.type()->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ break;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ break;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ break;
+ case Type::BOOL:
+ dtype.code = DLDataTypeCode::kDLBool;
+ break;
+ default:
+ // TODO
+ break;
+ }
+ return dtype;
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+DLManagedTensor* toDLPack(const Array& arr) {
Review Comment:
Need to push, have it locally 👍
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+DLDataType getDLDataType(const Array& arr) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = arr.type()->bit_width();
+ switch (arr.type()->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ break;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ break;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ break;
+ case Type::BOOL:
+ dtype.code = DLDataTypeCode::kDLBool;
+ break;
+ default:
+ // TODO
+ break;
+ }
+ return dtype;
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+DLManagedTensor* toDLPack(const Array& arr) {
Review Comment:
Need to push, have it locally 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [C++][Python] DLPack implementation for Arrow Arrays (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1863214048
@github-actions crossbow submit *cuda*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1410998337
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
@AlenkaF I had missed this was in the spec. This seems ok on the principle. Feel free to ping me when there's new code to review :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1831423351
For the cuda test you might need to trigger some crossbow build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1415456409
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,40 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, containing a DLPackManagedTensor.
+ """
+ if stream is None:
+ return to_dlpack(self)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
Review Comment:
Ah, it seems this is somewhat document on the Array API side (not on the DLPack page itself): https://data-apis.org/array-api/latest/API_specification/generated/array_api.array.__dlpack_device__.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1415539648
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,40 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, containing a DLPackManagedTensor.
+ """
+ if stream is None:
+ return to_dlpack(self)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
Review Comment:
I have missed it too. It is in the second item in the enumerated list at `Syntax for data interchange with DLPack`:
https://dmlc.github.io/dlpack/latest/python_spec.html#syntax-for-data-interchange-with-dlpack
Numpy doesn't use it in their `from_dlpack(x)` but they have it implemented for the array object:
https://github.com/numpy/numpy/blob/eabb962d38d8a871b2d76a9198846b49897365eb/numpy/array_api/_array_object.py#L488-L493
PyTorch checks for `__dlpack_device__` in Python:
https://github.com/pytorch/pytorch/blob/89569be2bdb608e64a190c5498f36b711ddd3e8d/torch/utils/dlpack.py#L100-L101
but I am not sure if they have something similar in C++ (I think not).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417099176
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,131 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow::dlpack {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type.bit_width();
+ switch (type.id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ return dtype;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ return dtype;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ return dtype;
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ type.ToString());
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
Review Comment:
Will be as part of https://github.com/apache/arrow/pull/38472#discussion_r1414170664.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "leofang (via GitHub)" <gi...@apache.org>.
leofang commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1421961780
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
xref: https://github.com/dmlc/dlpack/pull/133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417052191
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,131 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow::dlpack {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type.bit_width();
+ switch (type.id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ return dtype;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ return dtype;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ return dtype;
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ type.ToString());
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
Review Comment:
The shape can be removed now from the context?
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,93 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+The DLPack Protocol is a stable in-memory data structure
+that allows exchange between major frameworks working
+with multidimensional arrays or tensors. It is
+designed for cross hardware support meaning it allows exchange
+of data on devices other than the CPU (e.g. GPU).
+
+DLPack protocol had been
+`selected as the Python array API standard <https://data-apis.org/array-api/latest/design_topics/data_interchange.html#dlpack-an-in-memory-tensor-structure>`_
+by the
+`Consortium for Python Data API Standards <https://data-apis.org/>`_
+in order to enable device aware data interchange between array/tensor
+libraries in the Python ecosystem. See more about the standard
+in the
+`protocol documentation <https://data-apis.org/array-api/latest/index.html>`_
+and more about the DLPack in the
Review Comment:
```suggestion
and more about DLPack in the
```
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,93 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+The DLPack Protocol is a stable in-memory data structure
+that allows exchange between major frameworks working
+with multidimensional arrays or tensors. It is
+designed for cross hardware support meaning it allows exchange
+of data on devices other than the CPU (e.g. GPU).
+
+DLPack protocol had been
+`selected as the Python array API standard <https://data-apis.org/array-api/latest/design_topics/data_interchange.html#dlpack-an-in-memory-tensor-structure>`_
+by the
+`Consortium for Python Data API Standards <https://data-apis.org/>`_
+in order to enable device aware data interchange between array/tensor
+libraries in the Python ecosystem. See more about the standard
+in the
+`protocol documentation <https://data-apis.org/array-api/latest/index.html>`_
+and more about the DLPack in the
+`Python Specification for DLPack <https://dmlc.github.io/dlpack/latest/python_spec.html#python-spec>`_.
+
+Implementation of DLPack in PyArrow
+-----------------------------------
+
+Producing side of the DLPack Protocol is implemented for ``pa.Array``
Review Comment:
```suggestion
The producing side of the DLPack Protocol is implemented for ``pa.Array``
```
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,93 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+The DLPack Protocol is a stable in-memory data structure
+that allows exchange between major frameworks working
+with multidimensional arrays or tensors. It is
+designed for cross hardware support meaning it allows exchange
+of data on devices other than the CPU (e.g. GPU).
+
+DLPack protocol had been
+`selected as the Python array API standard <https://data-apis.org/array-api/latest/design_topics/data_interchange.html#dlpack-an-in-memory-tensor-structure>`_
+by the
+`Consortium for Python Data API Standards <https://data-apis.org/>`_
+in order to enable device aware data interchange between array/tensor
+libraries in the Python ecosystem. See more about the standard
+in the
+`protocol documentation <https://data-apis.org/array-api/latest/index.html>`_
+and more about the DLPack in the
+`Python Specification for DLPack <https://dmlc.github.io/dlpack/latest/python_spec.html#python-spec>`_.
+
+Implementation of DLPack in PyArrow
+-----------------------------------
+
+Producing side of the DLPack Protocol is implemented for ``pa.Array``
+and can be used to interchange data between PyArrow and other tensor
+libraries. The data structures that are supported in the implementation
+of the protocol are integer, unsigned integer and float arrays. The
+protocol has no missing data support meaning PyArrow arrays with
+missing values cannot be used to transfer data through the DLPack
+protocol. Currently Arrow implementation of the protocol only supports
Review Comment:
```suggestion
protocol. Currently, the Arrow implementation of the protocol only supports
```
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,93 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+The DLPack Protocol is a stable in-memory data structure
+that allows exchange between major frameworks working
+with multidimensional arrays or tensors. It is
+designed for cross hardware support meaning it allows exchange
+of data on devices other than the CPU (e.g. GPU).
+
+DLPack protocol had been
+`selected as the Python array API standard <https://data-apis.org/array-api/latest/design_topics/data_interchange.html#dlpack-an-in-memory-tensor-structure>`_
+by the
+`Consortium for Python Data API Standards <https://data-apis.org/>`_
+in order to enable device aware data interchange between array/tensor
+libraries in the Python ecosystem. See more about the standard
+in the
+`protocol documentation <https://data-apis.org/array-api/latest/index.html>`_
+and more about the DLPack in the
+`Python Specification for DLPack <https://dmlc.github.io/dlpack/latest/python_spec.html#python-spec>`_.
+
+Implementation of DLPack in PyArrow
+-----------------------------------
+
+Producing side of the DLPack Protocol is implemented for ``pa.Array``
+and can be used to interchange data between PyArrow and other tensor
+libraries. The data structures that are supported in the implementation
+of the protocol are integer, unsigned integer and float arrays. The
Review Comment:
```suggestion
libraries. Supported data types are integer, unsigned integer and float. The
```
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,93 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+The DLPack Protocol is a stable in-memory data structure
Review Comment:
maybe link "DLPack Protocol" to their main github https://github.com/dmlc/dlpack ?
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
Review Comment:
Since we also have other capsule deleters for the C Data Interface, let's name this more explicit
```suggestion
cdef void dlpack_pycapsule_deleter(object dltensor) noexcept:
```
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,93 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+The DLPack Protocol is a stable in-memory data structure
+that allows exchange between major frameworks working
+with multidimensional arrays or tensors. It is
+designed for cross hardware support meaning it allows exchange
+of data on devices other than the CPU (e.g. GPU).
+
+DLPack protocol had been
+`selected as the Python array API standard <https://data-apis.org/array-api/latest/design_topics/data_interchange.html#dlpack-an-in-memory-tensor-structure>`_
+by the
+`Consortium for Python Data API Standards <https://data-apis.org/>`_
+in order to enable device aware data interchange between array/tensor
+libraries in the Python ecosystem. See more about the standard
+in the
+`protocol documentation <https://data-apis.org/array-api/latest/index.html>`_
+and more about the DLPack in the
+`Python Specification for DLPack <https://dmlc.github.io/dlpack/latest/python_spec.html#python-spec>`_.
+
+Implementation of DLPack in PyArrow
+-----------------------------------
+
+Producing side of the DLPack Protocol is implemented for ``pa.Array``
+and can be used to interchange data between PyArrow and other tensor
+libraries. The data structures that are supported in the implementation
+of the protocol are integer, unsigned integer and float arrays. The
+protocol has no missing data support meaning PyArrow arrays with
+missing values cannot be used to transfer data through the DLPack
Review Comment:
```suggestion
missing values cannot be transferred through the DLPack
```
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,40 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, containing a DLPackManagedTensor.
+ """
+ if stream is None:
+ return to_dlpack(self)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
Review Comment:
@AlenkaF have you checked it locally that this works with pytorch (or another tensor library that might check `__dlpack_device__` (ideally we would have integration tests for this part as well instead of only with numpy, but this alone is probably not worth adding a heavy test dependency on pytorch)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1850255958
I would simply add the file to `dev/release/rat_exclude_files.txt`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1374292854
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+
+
+cdef void deleter(DLManagedTensor* tensor) with gil:
+ if tensor.manager_ctx is NULL:
+ return
+ free(tensor.dl_tensor.shape)
+ cpython.Py_DECREF(<Array>tensor.manager_ctx)
+ tensor.manager_ctx = NULL
+ free(tensor)
+
+
+cpdef object to_dlpack(Array arr) except *:
+ cdef DLManagedTensor* dlm_tensor = <DLManagedTensor*> malloc(sizeof(DLManagedTensor))
Review Comment:
Ah, you're right. Let's stick with `DLManangedTensor` for now, then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1406081098
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ cdef PyObject* err_type
+ cdef PyObject* err_value
+ cdef PyObject* err_traceback
+
+ if cpython.PyCapsule_IsValid(dltensor, "used_dltensor"):
+ return
+
+ cpython.PyErr_Fetch(&err_type, &err_value, &err_traceback)
Review Comment:
https://github.com/apache/arrow/pull/38472/commits/51b64802eb635d28fd5f71c0ae103d34365e3b67
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1407700375
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_bitmap() != NULLPTR) {
+ return Status::TypeError(
+ "Can only use __dlpack__ on arrays with no validity buffer.");
+ }
Review Comment:
I think it would be fine to relax this to a null_count of zero? (i.e. if the array has a validity buffer, but just without any nulls, we can still export zero-copy the data buffer)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1406046478
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
I think this should do: https://github.com/apache/arrow/pull/38472/commits/b3c0a8c8fe73d61c241c797e851384d623455c1c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1410498838
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
Let me summarize what conclusion we arrived at when talking today:
`ExportArray` allocates the `DLManagedTensor` struct that is stored in `DLMTensorCtx`. When PyCapsule calls the deleter (or any C++ consumer) at the end, the deleter deletes this `DLMTensorCtx` context which in turn also cleans up the `DLManagedTensor`, see https://dmlc.github.io/dlpack/latest/c_api.html#_CPPv4N15DLManagedTensor7deleterE:
> Destructor - this should be called to destruct the manager_ctx which backs the [DLManagedTensor](https://dmlc.github.io/dlpack/latest/c_api.html#structDLManagedTensor). [...]. The destructors deletes the argument self as well.
@pitrou you think we got this right? 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417051061
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
Perhaps something like this, then (still untested!):
```c++
ctx->tensor->manager_ctx = ctx.get();
ctx->tensor->deleter = [](struct DLManagedTensor *self) {
delete reinterpret_cast<ManagerCtx*>(self->manager_ctx);
};
return &ctx.release()->tensor;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1415464102
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
Review Comment:
> Do we plan to later support `fixed_shape_tensor` extension arrays?
See https://github.com/apache/arrow/issues/38868 for that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1423610414
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,49 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, pointing to a DLManagedTensor.
+ """
+ cdef CResult[DLManagedTensor*] c_dlm_tensor
+
+ if stream is None:
+ c_dlm_tensor = ExportToDLPack(pyarrow_unwrap_array(self))
+ dlm_tensor = GetResultValue(c_dlm_tensor)
+
+ return PyCapsule_New(dlm_tensor, 'dltensor', dlpack_pycapsule_deleter)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
+
+ Returns
+ -------
+ tuple : Tuple[DLDeviceType, int]
Review Comment:
Yeah, it currently returns an int.
Numpy returns a `tuple[int, Literall[0]]`: https://github.com/numpy/numpy/blob/0be4154648cd6e90aae32246349085f7ca8f2b13/numpy/typing/tests/data/reveal/ndarray_misc.pyi#L34
but the documentation is clear about returning `Tuple[Enum, int]`:
https://data-apis.org/array-api/latest/API_specification/generated/array_api.array.__dlpack_device__.html
Will update code to follow the array-api docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [C++][Python] DLPack implementation for Arrow Arrays (producer) [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1863216928
Revision: 49a978fcef620e6c2f9d68cf3b9ebae03c5a1020
Submitted crossbow builds: [ursacomputing/crossbow @ actions-b8525f4884](https://github.com/ursacomputing/crossbow/branches/all?query=actions-b8525f4884)
|Task|Status|
|----|------|
|conda-linux-aarch64-cuda-py3|[](https://github.com/ursacomputing/crossbow/runs/19795378668)|
|conda-linux-ppc64le-cuda-py3|[](https://github.com/ursacomputing/crossbow/runs/19795378003)|
|conda-linux-x64-cuda-py3|[](https://github.com/ursacomputing/crossbow/runs/19795377555)|
|conda-win-x64-cuda-py3|[](https://github.com/ursacomputing/crossbow/runs/19795378547)|
|test-cuda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7265518460/job/19795376877)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/7265518448/job/19795376614)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1402174495
##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -1199,6 +1199,60 @@ cdef extern from "arrow/api.h" namespace "arrow" nogil:
shared_ptr[CScalar] MakeNullScalar(shared_ptr[CDataType] type)
+cdef extern from "arrow/dlpack_structure.h" nogil:
+ cdef enum DLDeviceType:
Review Comment:
Just need to push, have it locally 👍
##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -1199,6 +1199,60 @@ cdef extern from "arrow/api.h" namespace "arrow" nogil:
shared_ptr[CScalar] MakeNullScalar(shared_ptr[CDataType] type)
+cdef extern from "arrow/dlpack_structure.h" nogil:
+ cdef enum DLDeviceType:
Review Comment:
Just need to push, have it locally 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1402500695
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
The semantics of a raw pointer are not obvious to say the least. Who is responsible to deallocate it and how?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1403594145
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+DLDataType getDLDataType(const Array& arr) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = arr.type()->bit_width();
+ switch (arr.type()->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ break;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ break;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ break;
+ case Type::BOOL:
+ dtype.code = DLDataTypeCode::kDLBool;
+ break;
+ default:
+ // TODO
+ break;
+ }
+ return dtype;
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+DLManagedTensor* toDLPack(const Array& arr) {
+ std::shared_ptr<ArrayData> array_ref = arr.data();
Review Comment:
I thought not, but am not really sure. AFAIU this is only needed for a reference so I can just as well use `Slice` on an `Array` which for sure doesn't do any copies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1373483599
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+
+
+cdef void deleter(DLManagedTensor* tensor) with gil:
+ if tensor.manager_ctx is NULL:
+ return
+ free(tensor.dl_tensor.shape)
+ cpython.Py_DECREF(<Array>tensor.manager_ctx)
+ tensor.manager_ctx = NULL
+ free(tensor)
+
+
+cpdef object to_dlpack(Array arr) except *:
+ cdef DLManagedTensor* dlm_tensor = <DLManagedTensor*> malloc(sizeof(DLManagedTensor))
+
+ cdef DLTensor* dl_tensor = &dlm_tensor.dl_tensor
+ cdef intptr_t data_ptr = arr.buffers()[1].address
+ dl_tensor.data = <void*> data_ptr
+ dl_tensor.ndim = 1
+ cdef int64_t* shape = <int64_t*> malloc(sizeof(int64_t))
+ shape[0] = len(arr)
+ dl_tensor.shape = shape
+
+ dl_tensor.strides = NULL
+ dl_tensor.byte_offset = 0
+
+ cdef DLDevice* device = &dl_tensor.device
+ device.device_type = kDLCPU
+ device.device_id = 0
+
+ cdef DLDataType* dtype = &dl_tensor.dtype
+ if arr.type in [uint8(), uint16(), uint32(), uint64()]:
+ dtype.code = kDLUInt
+ elif arr.type in [int8(), int16(), int32(), int64()]:
+ dtype.code = kDLInt
+ elif arr.type in [float16(), float32(), float64()]:
+ dtype.code = kDLFloat
+ elif arr.type == bool_():
+ dtype.code = kDLBool
+
+ dtype.lanes = <uint16_t>1
+ dtype.bits = <uint8_t>arr.type.bit_width
+
+ dlm_tensor.manager_ctx = <void*>arr
+ cpython.Py_INCREF(arr)
Review Comment:
Will move the implementation of `to_dlpack ` to the C++ 👍
I will probably have to move the check for the buffers (only primitive types without validity bitmasks) to C++ also, right?
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+
+
+cdef void deleter(DLManagedTensor* tensor) with gil:
+ if tensor.manager_ctx is NULL:
+ return
+ free(tensor.dl_tensor.shape)
+ cpython.Py_DECREF(<Array>tensor.manager_ctx)
+ tensor.manager_ctx = NULL
+ free(tensor)
+
+
+cpdef object to_dlpack(Array arr) except *:
+ cdef DLManagedTensor* dlm_tensor = <DLManagedTensor*> malloc(sizeof(DLManagedTensor))
+
+ cdef DLTensor* dl_tensor = &dlm_tensor.dl_tensor
+ cdef intptr_t data_ptr = arr.buffers()[1].address
+ dl_tensor.data = <void*> data_ptr
+ dl_tensor.ndim = 1
+ cdef int64_t* shape = <int64_t*> malloc(sizeof(int64_t))
+ shape[0] = len(arr)
+ dl_tensor.shape = shape
+
+ dl_tensor.strides = NULL
+ dl_tensor.byte_offset = 0
+
+ cdef DLDevice* device = &dl_tensor.device
+ device.device_type = kDLCPU
+ device.device_id = 0
+
+ cdef DLDataType* dtype = &dl_tensor.dtype
+ if arr.type in [uint8(), uint16(), uint32(), uint64()]:
+ dtype.code = kDLUInt
+ elif arr.type in [int8(), int16(), int32(), int64()]:
+ dtype.code = kDLInt
+ elif arr.type in [float16(), float32(), float64()]:
+ dtype.code = kDLFloat
+ elif arr.type == bool_():
+ dtype.code = kDLBool
+
+ dtype.lanes = <uint16_t>1
+ dtype.bits = <uint8_t>arr.type.bit_width
+
+ dlm_tensor.manager_ctx = <void*>arr
+ cpython.Py_INCREF(arr)
Review Comment:
Will move the implementation of `to_dlpack ` to the C++ 👍
I will probably have to move the check for the buffers (only primitive types without validity bitmasks) to C++ also, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "kkraus14 (via GitHub)" <gi...@apache.org>.
kkraus14 commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1402244230
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+DLDataType getDLDataType(const Array& arr) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = arr.type()->bit_width();
+ switch (arr.type()->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ break;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ break;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ break;
+ case Type::BOOL:
+ dtype.code = DLDataTypeCode::kDLBool;
+ break;
+ default:
+ // TODO
+ break;
+ }
+ return dtype;
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+DLManagedTensor* toDLPack(const Array& arr) {
+ std::shared_ptr<ArrayData> array_ref = arr.data();
Review Comment:
Does this make a deep copy of the underlying data?
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+DLDataType getDLDataType(const Array& arr) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = arr.type()->bit_width();
+ switch (arr.type()->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ break;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ break;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ break;
+ case Type::BOOL:
+ dtype.code = DLDataTypeCode::kDLBool;
+ break;
+ default:
+ // TODO
+ break;
+ }
+ return dtype;
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+DLManagedTensor* toDLPack(const Array& arr) {
+ std::shared_ptr<ArrayData> array_ref = arr.data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+ dlm_tensor->dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+
+ DLDevice ctx;
+ ctx.device_id = 0;
+ ctx.device_type = DLDeviceType::kDLCPU;
Review Comment:
What happens if someone hands an array backed by `CudaBuffer` buffers?
I think we need to validate these are CPU buffers, or better yet support the cases where we have non-CPU buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1410498838
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
Let me summarize what conclusion we arrived at when talking today:
`ExportArray` allocates the `DLManagedTensor` struct that is stored in `DLMTensorCtx`. When PyCapsule calls the deleter (or any C++ consumer) at the end, the deleter removes the context which in turn also cleans up the `DLManagedTensor`, see https://dmlc.github.io/dlpack/latest/c_api.html#_CPPv4N15DLManagedTensor7deleterE:
> Destructor - this should be called to destruct the manager_ctx which backs the [DLManagedTensor](https://dmlc.github.io/dlpack/latest/c_api.html#structDLManagedTensor). [...]. The destructors deletes the argument self as well.
@pitrou you think we got this right? 😊
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
Let me summarize what conclusion we arrived at when talking today:
`ExportArray` allocates the `DLManagedTensor` struct that is stored in `DLMTensorCtx`. When PyCapsule calls the deleter (or any C++ consumer) at the end, the deleter removes the context which in turn also cleans up the `DLManagedTensor`, see https://dmlc.github.io/dlpack/latest/c_api.html#_CPPv4N15DLManagedTensor7deleterE:
> Destructor - this should be called to destruct the manager_ctx which backs the [DLManagedTensor](https://dmlc.github.io/dlpack/latest/c_api.html#structDLManagedTensor). [...]. The destructors deletes the argument self as well.
@pitrou you think we got this right? 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1408989636
##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -1199,6 +1199,60 @@ cdef extern from "arrow/api.h" namespace "arrow" nogil:
shared_ptr[CScalar] MakeNullScalar(shared_ptr[CDataType] type)
+cdef extern from "arrow/dlpack_structure.h" nogil:
+ cdef enum DLDeviceType:
Review Comment:
Sorry, misunderstood this one 😊
##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -1199,6 +1199,60 @@ cdef extern from "arrow/api.h" namespace "arrow" nogil:
shared_ptr[CScalar] MakeNullScalar(shared_ptr[CDataType] type)
+cdef extern from "arrow/dlpack_structure.h" nogil:
+ cdef enum DLDeviceType:
Review Comment:
Sorry, misunderstood this one 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1415454572
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,40 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, containing a DLPackManagedTensor.
+ """
+ if stream is None:
+ return to_dlpack(self)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
Review Comment:
There is exactly one mention of `__dlpack_device__` on that page (just the mention that it exists in bullet point no 2), but unfortunately without any explanation of what it is or should return.
So implementation defined, eg https://github.com/numpy/numpy/blob/eabb962d38d8a871b2d76a9198846b49897365eb/numpy/_core/src/multiarray/dlpack.c#L279-L287 or https://github.com/pytorch/pytorch/blob/89569be2bdb608e64a190c5498f36b711ddd3e8d/torch/_tensor.py#L1484-L1502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "rgommers (via GitHub)" <gi...@apache.org>.
rgommers commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417094740
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
Argh sorry, it's early. Did a bit more digging, and there is no requirement documented in `dlpack.h`, nor a review comment on those Python docs when they were added. The one single mention of the data pointer being NULL I can find is [this comment](https://github.com/dmlc/dlpack/issues/51#issuecomment-849829016) by @leofang. Leo, do you have any thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1416907727
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
After calling `release()`, I think the unique ptr itself (`ctx`) becomes null? (https://en.cppreference.com/w/cpp/memory/unique_ptr/release) So then you can't use `ctx` afterwards to refer to it, but you have to use the raw pointer instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [C++][Python] DLPack implementation for Arrow Arrays (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #38472:
URL: https://github.com/apache/arrow/pull/38472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [C++][Python] DLPack implementation for Arrow Arrays (producer) [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1863216881
Revision: 49a978fcef620e6c2f9d68cf3b9ebae03c5a1020
Submitted crossbow builds: [ursacomputing/crossbow @ actions-bbd691007d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-bbd691007d)
|Task|Status|
|----|------|
|test-alpine-linux-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517410/job/19795372746)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517387/job/19795372512)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517448/job/19795373058)|
|test-conda-cpp-valgrind|[](https://github.com/ursacomputing/crossbow/runs/19795375114)|
|test-cuda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517412/job/19795372854)|
|test-debian-11-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517348/job/19795372468)|
|test-debian-11-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517452/job/19795372965)|
|test-fedora-38-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517565/job/19795373497)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517495/job/19795373126)|
|test-ubuntu-20.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517352/job/19795372287)|
|test-ubuntu-20.04-cpp-minimal-with-formats|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517567/job/19795373507)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517378/job/19795372264)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517470/job/19795372980)|
|test-ubuntu-22.04-cpp-20|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517355/job/19795372285)|
|test-ubuntu-22.04-cpp-no-threading|[](https://github.com/ursacomputing/crossbow/actions/runs/7265517349/job/19795372446)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [C++][Python] DLPack implementation for Arrow Arrays (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1863324906
Thank you @pitrou for all the suggestions and help!
Have to add this was a very nice way to learn some C++ ❤️
Listing possible follow-up issues:
- https://github.com/apache/arrow/issues/39294
- https://github.com/apache/arrow/issues/38868
- https://github.com/apache/arrow/issues/39295
- https://github.com/apache/arrow/issues/39296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1850263119
Will do 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1422637533
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,49 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, pointing to a DLManagedTensor.
+ """
+ cdef CResult[DLManagedTensor*] c_dlm_tensor
+
+ if stream is None:
+ c_dlm_tensor = ExportToDLPack(pyarrow_unwrap_array(self))
+ dlm_tensor = GetResultValue(c_dlm_tensor)
Review Comment:
You can probably simplify this
```suggestion
if stream is None:
dlm_tensor = GetResultValue(ExportToDLPack(self.sp_array))
```
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
Review Comment:
This is not used anymore, is it?
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+#include "arrow/type_traits.h"
+
+namespace arrow::dlpack {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type.bit_width();
+ switch (type.id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ return dtype;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ return dtype;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ return dtype;
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ type.ToString());
+ }
+}
+
+struct ManagerCtx {
+ std::shared_ptr<ArrayData> ref;
+ DLManagedTensor tensor;
+};
+
+Result<DLManagedTensor*> ExportArray(const std::shared_ptr<Array>& arr) {
+ // Define DLDevice struct nad check if array type is supported
+ // by the DLPack protocol at the same time. Raise TypeError if not.
+ // Supported data types: int, uint, float with no validity buffer.
+ ARROW_ASSIGN_OR_RAISE(auto device, ExportDevice(arr))
+
+ // Define the DLDataType struct
+ const DataType* arrow_type = arr->type().get();
+ ARROW_ASSIGN_OR_RAISE(auto dlpack_type, GetDLDataType(*arrow_type));
+
+ // Create ManagerCtx with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ std::unique_ptr<ManagerCtx> ctx(new ManagerCtx);
+ ctx->ref = array_ref;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
+ if (arr->length() == 0) {
+ ctx->tensor.dl_tensor.data = NULL;
+ } else if (arr->offset() > 0) {
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ARROW_ASSIGN_OR_RAISE(auto sliced_buffer,
+ SliceBufferSafe(array_ref->buffers[1], start));
+ ctx->tensor.dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(sliced_buffer->address()));
+ } else {
+ ctx->tensor.dl_tensor.data = const_cast<void*>(
+ reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+ }
+
+ ctx->tensor.dl_tensor.device = device;
+ ctx->tensor.dl_tensor.ndim = 1;
+ ctx->tensor.dl_tensor.dtype = dlpack_type;
+ ctx->tensor.dl_tensor.shape = const_cast<int64_t*>(&array_ref->length);
+ ctx->tensor.dl_tensor.strides = NULL;
+ ctx->tensor.dl_tensor.byte_offset = 0;
+
+ // return dlm_tensor;
Review Comment:
Does this line need to be removed now?
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+#include "arrow/type_traits.h"
+
+namespace arrow::dlpack {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type.bit_width();
+ switch (type.id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ return dtype;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ return dtype;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ return dtype;
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ type.ToString());
+ }
+}
+
+struct ManagerCtx {
+ std::shared_ptr<ArrayData> ref;
+ DLManagedTensor tensor;
+};
+
+Result<DLManagedTensor*> ExportArray(const std::shared_ptr<Array>& arr) {
+ // Define DLDevice struct nad check if array type is supported
+ // by the DLPack protocol at the same time. Raise TypeError if not.
+ // Supported data types: int, uint, float with no validity buffer.
+ ARROW_ASSIGN_OR_RAISE(auto device, ExportDevice(arr))
+
+ // Define the DLDataType struct
+ const DataType* arrow_type = arr->type().get();
+ ARROW_ASSIGN_OR_RAISE(auto dlpack_type, GetDLDataType(*arrow_type));
+
+ // Create ManagerCtx with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ std::unique_ptr<ManagerCtx> ctx(new ManagerCtx);
+ ctx->ref = array_ref;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
+ if (arr->length() == 0) {
+ ctx->tensor.dl_tensor.data = NULL;
+ } else if (arr->offset() > 0) {
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ARROW_ASSIGN_OR_RAISE(auto sliced_buffer,
+ SliceBufferSafe(array_ref->buffers[1], start));
+ ctx->tensor.dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(sliced_buffer->address()));
+ } else {
+ ctx->tensor.dl_tensor.data = const_cast<void*>(
+ reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+ }
+
+ ctx->tensor.dl_tensor.device = device;
+ ctx->tensor.dl_tensor.ndim = 1;
+ ctx->tensor.dl_tensor.dtype = dlpack_type;
+ ctx->tensor.dl_tensor.shape = const_cast<int64_t*>(&array_ref->length);
+ ctx->tensor.dl_tensor.strides = NULL;
+ ctx->tensor.dl_tensor.byte_offset = 0;
+
+ // return dlm_tensor;
+ ctx->tensor.manager_ctx = ctx.get();
+ ctx->tensor.deleter = [](struct DLManagedTensor* self) {
+ delete reinterpret_cast<ManagerCtx*>(self->manager_ctx);
+ };
+ return &ctx.release()->tensor;
+}
+
+Result<DLDevice> ExportDevice(const std::shared_ptr<Array>& arr) {
+ // Check if array is supported by the DLPack protocol.
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use DLPack on arrays with no nulls.");
+ }
+ const DataType* arrow_type = arr->type().get();
+ if (arrow_type->id() == Type::BOOL) {
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ }
+ if (!is_integer(arrow_type->id()) && !is_unsigned_integer(arrow_type->id()) &&
+ !is_floating(arrow_type->id())) {
Review Comment:
I think this can be simplified to:
```suggestion
if (!is_integer(arrow_type->id()) && !is_floating(arrow_type->id())) {
```
##########
cpp/src/arrow/c/dlpack_test.cc:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+
+namespace arrow::dlpack {
+
+class TestExportArray : public ::testing::Test {
+ public:
+ void SetUp() {}
+};
+
+auto check_dlptensor = [](const std::shared_ptr<Array>& arr,
+ std::shared_ptr<DataType> arrow_type,
+ DLDataTypeCode dlpack_type, int64_t length) {
+ ASSERT_OK_AND_ASSIGN(auto dlmtensor, arrow::dlpack::ExportArray(arr));
+ auto dltensor = dlmtensor->dl_tensor;
+
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ASSERT_OK_AND_ASSIGN(auto sliced_buffer,
+ SliceBufferSafe(arr->data()->buffers[1], start));
+ ASSERT_EQ(sliced_buffer->data(), dltensor.data);
+
+ ASSERT_EQ(0, dltensor.byte_offset);
+ ASSERT_EQ(NULL, dltensor.strides);
+ ASSERT_EQ(length, dltensor.shape[0]);
+ ASSERT_EQ(1, dltensor.ndim);
+
+ ASSERT_EQ(dlpack_type, dltensor.dtype.code);
+
+ ASSERT_EQ(arrow_type->bit_width(), dltensor.dtype.bits);
+ ASSERT_EQ(1, dltensor.dtype.lanes);
+ ASSERT_EQ(DLDeviceType::kDLCPU, dltensor.device.device_type);
+ ASSERT_EQ(0, dltensor.device.device_id);
+
+ ASSERT_OK_AND_ASSIGN(auto device, arrow::dlpack::ExportDevice(arr));
+ ASSERT_EQ(DLDeviceType::kDLCPU, device.device_type);
+ ASSERT_EQ(0, device.device_id);
+
+ dlmtensor->deleter(dlmtensor);
+};
+
+TEST_F(TestExportArray, TestSupportedArray) {
+ random::RandomArrayGenerator gen(0);
+
+ std::vector<std::shared_ptr<DataType>> arrow_types = {
+ int8(), uint8(), int16(), uint16(), int32(), uint32(),
+ int64(), uint64(), float16(), float32(), float64(),
+ };
+
+ std::vector<DLDataTypeCode> dlpack_types = {
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt,
+ DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt,
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLFloat,
+ DLDataTypeCode::kDLFloat, DLDataTypeCode::kDLFloat,
+ };
+
+ for (int64_t i = 0; i < 11; ++i) {
Review Comment:
Hard-coding array lengths like this is fragile.
How about something like:
```c++
std::vector<std::pair<std::shared_ptr<DataType>, DLDataTypeCode>> cases = {
{int8(), DLDataTypeCode::kDLInt}, ...
};
for (auto [arrow_type, dlpack_type] : cases) {
...
```
##########
cpp/src/arrow/c/dlpack_test.cc:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+
+namespace arrow::dlpack {
+
+class TestExportArray : public ::testing::Test {
+ public:
+ void SetUp() {}
+};
+
+auto check_dlptensor = [](const std::shared_ptr<Array>& arr,
+ std::shared_ptr<DataType> arrow_type,
+ DLDataTypeCode dlpack_type, int64_t length) {
+ ASSERT_OK_AND_ASSIGN(auto dlmtensor, arrow::dlpack::ExportArray(arr));
+ auto dltensor = dlmtensor->dl_tensor;
+
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ASSERT_OK_AND_ASSIGN(auto sliced_buffer,
+ SliceBufferSafe(arr->data()->buffers[1], start));
+ ASSERT_EQ(sliced_buffer->data(), dltensor.data);
+
+ ASSERT_EQ(0, dltensor.byte_offset);
+ ASSERT_EQ(NULL, dltensor.strides);
+ ASSERT_EQ(length, dltensor.shape[0]);
+ ASSERT_EQ(1, dltensor.ndim);
+
+ ASSERT_EQ(dlpack_type, dltensor.dtype.code);
+
+ ASSERT_EQ(arrow_type->bit_width(), dltensor.dtype.bits);
+ ASSERT_EQ(1, dltensor.dtype.lanes);
+ ASSERT_EQ(DLDeviceType::kDLCPU, dltensor.device.device_type);
+ ASSERT_EQ(0, dltensor.device.device_id);
+
+ ASSERT_OK_AND_ASSIGN(auto device, arrow::dlpack::ExportDevice(arr));
+ ASSERT_EQ(DLDeviceType::kDLCPU, device.device_type);
+ ASSERT_EQ(0, device.device_id);
+
+ dlmtensor->deleter(dlmtensor);
+};
+
+TEST_F(TestExportArray, TestSupportedArray) {
+ random::RandomArrayGenerator gen(0);
+
+ std::vector<std::shared_ptr<DataType>> arrow_types = {
+ int8(), uint8(), int16(), uint16(), int32(), uint32(),
+ int64(), uint64(), float16(), float32(), float64(),
+ };
+
+ std::vector<DLDataTypeCode> dlpack_types = {
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt,
+ DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt,
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLFloat,
+ DLDataTypeCode::kDLFloat, DLDataTypeCode::kDLFloat,
+ };
+
+ for (int64_t i = 0; i < 11; ++i) {
+ const std::shared_ptr<Array> array = gen.ArrayOf(arrow_types[i], 10, 0);
Review Comment:
You also don't need a random generator here...
##########
cpp/src/arrow/c/dlpack_test.cc:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+
+namespace arrow::dlpack {
+
+class TestExportArray : public ::testing::Test {
+ public:
+ void SetUp() {}
+};
+
+auto check_dlptensor = [](const std::shared_ptr<Array>& arr,
+ std::shared_ptr<DataType> arrow_type,
+ DLDataTypeCode dlpack_type, int64_t length) {
+ ASSERT_OK_AND_ASSIGN(auto dlmtensor, arrow::dlpack::ExportArray(arr));
+ auto dltensor = dlmtensor->dl_tensor;
+
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ASSERT_OK_AND_ASSIGN(auto sliced_buffer,
+ SliceBufferSafe(arr->data()->buffers[1], start));
+ ASSERT_EQ(sliced_buffer->data(), dltensor.data);
+
+ ASSERT_EQ(0, dltensor.byte_offset);
+ ASSERT_EQ(NULL, dltensor.strides);
+ ASSERT_EQ(length, dltensor.shape[0]);
+ ASSERT_EQ(1, dltensor.ndim);
+
+ ASSERT_EQ(dlpack_type, dltensor.dtype.code);
+
+ ASSERT_EQ(arrow_type->bit_width(), dltensor.dtype.bits);
+ ASSERT_EQ(1, dltensor.dtype.lanes);
+ ASSERT_EQ(DLDeviceType::kDLCPU, dltensor.device.device_type);
+ ASSERT_EQ(0, dltensor.device.device_id);
+
+ ASSERT_OK_AND_ASSIGN(auto device, arrow::dlpack::ExportDevice(arr));
+ ASSERT_EQ(DLDeviceType::kDLCPU, device.device_type);
+ ASSERT_EQ(0, device.device_id);
+
+ dlmtensor->deleter(dlmtensor);
+};
+
+TEST_F(TestExportArray, TestSupportedArray) {
+ random::RandomArrayGenerator gen(0);
+
+ std::vector<std::shared_ptr<DataType>> arrow_types = {
+ int8(), uint8(), int16(), uint16(), int32(), uint32(),
+ int64(), uint64(), float16(), float32(), float64(),
+ };
+
+ std::vector<DLDataTypeCode> dlpack_types = {
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt,
+ DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt,
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLFloat,
+ DLDataTypeCode::kDLFloat, DLDataTypeCode::kDLFloat,
+ };
+
+ for (int64_t i = 0; i < 11; ++i) {
+ const std::shared_ptr<Array> array = gen.ArrayOf(arrow_types[i], 10, 0);
+ check_dlptensor(array, arrow_types[i], dlpack_types[i], 10);
+ ASSERT_OK_AND_ASSIGN(auto sliced_1, array->SliceSafe(1, 5));
+ check_dlptensor(sliced_1, arrow_types[i], dlpack_types[i], 5);
+ ASSERT_OK_AND_ASSIGN(auto sliced_2, array->SliceSafe(0, 5));
+ check_dlptensor(sliced_2, arrow_types[i], dlpack_types[i], 5);
+ ASSERT_OK_AND_ASSIGN(auto sliced_3, array->SliceSafe(3));
+ check_dlptensor(sliced_3, arrow_types[i], dlpack_types[i], 7);
+ }
+}
+
+TEST_F(TestExportArray, TestUnSupportedArray) {
+ random::RandomArrayGenerator gen(0);
Review Comment:
You don't really need a random generator for this :-)
```c++
const std::shared_ptr<Array> array_with_null = ArrayFromJSON(int8(), "[1, null]");
const std::shared_ptr<Array> array_string = ArrayFromJSON(utf8(), "[]");
const std::shared_ptr<Array> array_boolean = ArrayFromJSON(boolean(), "[true]");
```
##########
cpp/src/arrow/c/dlpack_test.cc:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+
+namespace arrow::dlpack {
+
+class TestExportArray : public ::testing::Test {
+ public:
+ void SetUp() {}
+};
+
+auto check_dlptensor = [](const std::shared_ptr<Array>& arr,
+ std::shared_ptr<DataType> arrow_type,
+ DLDataTypeCode dlpack_type, int64_t length) {
+ ASSERT_OK_AND_ASSIGN(auto dlmtensor, arrow::dlpack::ExportArray(arr));
+ auto dltensor = dlmtensor->dl_tensor;
+
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ASSERT_OK_AND_ASSIGN(auto sliced_buffer,
+ SliceBufferSafe(arr->data()->buffers[1], start));
+ ASSERT_EQ(sliced_buffer->data(), dltensor.data);
+
+ ASSERT_EQ(0, dltensor.byte_offset);
+ ASSERT_EQ(NULL, dltensor.strides);
+ ASSERT_EQ(length, dltensor.shape[0]);
+ ASSERT_EQ(1, dltensor.ndim);
+
+ ASSERT_EQ(dlpack_type, dltensor.dtype.code);
+
+ ASSERT_EQ(arrow_type->bit_width(), dltensor.dtype.bits);
+ ASSERT_EQ(1, dltensor.dtype.lanes);
+ ASSERT_EQ(DLDeviceType::kDLCPU, dltensor.device.device_type);
+ ASSERT_EQ(0, dltensor.device.device_id);
+
+ ASSERT_OK_AND_ASSIGN(auto device, arrow::dlpack::ExportDevice(arr));
+ ASSERT_EQ(DLDeviceType::kDLCPU, device.device_type);
+ ASSERT_EQ(0, device.device_id);
+
+ dlmtensor->deleter(dlmtensor);
+};
+
+TEST_F(TestExportArray, TestSupportedArray) {
+ random::RandomArrayGenerator gen(0);
+
+ std::vector<std::shared_ptr<DataType>> arrow_types = {
+ int8(), uint8(), int16(), uint16(), int32(), uint32(),
+ int64(), uint64(), float16(), float32(), float64(),
+ };
+
+ std::vector<DLDataTypeCode> dlpack_types = {
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt,
+ DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt,
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLFloat,
+ DLDataTypeCode::kDLFloat, DLDataTypeCode::kDLFloat,
+ };
+
+ for (int64_t i = 0; i < 11; ++i) {
+ const std::shared_ptr<Array> array = gen.ArrayOf(arrow_types[i], 10, 0);
+ check_dlptensor(array, arrow_types[i], dlpack_types[i], 10);
+ ASSERT_OK_AND_ASSIGN(auto sliced_1, array->SliceSafe(1, 5));
+ check_dlptensor(sliced_1, arrow_types[i], dlpack_types[i], 5);
+ ASSERT_OK_AND_ASSIGN(auto sliced_2, array->SliceSafe(0, 5));
+ check_dlptensor(sliced_2, arrow_types[i], dlpack_types[i], 5);
+ ASSERT_OK_AND_ASSIGN(auto sliced_3, array->SliceSafe(3));
+ check_dlptensor(sliced_3, arrow_types[i], dlpack_types[i], 7);
+ }
Review Comment:
Can we also try to test that memory is properly deallocated?
For example:
```c++
TEST_F(TestExportArray, TestSupportedArray) {
const auto allocated_bytes = default_memory_pool()->bytes_allocated();
... // function body
ASSERT_EQ(allocated_bytes, default_memory_pool()->bytes_allocated());
```
##########
python/pyarrow/tests/test_dlpack.py:
##########
@@ -0,0 +1,129 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import ctypes
+import pytest
+
+import numpy as np
+
+import pyarrow as pa
+from pyarrow.vendored.version import Version
+
+
+def PyCapsule_IsValid(capsule, name):
+ return ctypes.pythonapi.PyCapsule_IsValid(ctypes.py_object(capsule), name) == 1
+
+
+def check_dlpack_export(arr, expected_arr):
+ DLTensor = arr.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+
+ result = np.from_dlpack(arr)
+ np.testing.assert_array_equal(result, expected_arr, strict=True)
+
+ assert arr.__dlpack_device__() == (1, 0)
+
+
Review Comment:
Can we also test that memory gets properly deallocated during these tests?
Something like:
```python
def check_bytes_allocated(f):
@functools.wraps(f)
def wrapper(*args, **kwargs):
allocated_bytes = pa.bytes_allocated()
try:
return f(*args, **kwargs)
finally:
assert pa.bytes_allocated() == allocated_bytes
return wrapper
@check_bytes_allocated
def test_dlpack(value_type, np_type):
....
```
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+#include "arrow/type_traits.h"
+
+namespace arrow::dlpack {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
Review Comment:
If this function is not exposed publicly, then it should be enclosed in the anonymous namespace.
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
Review Comment:
These are unused as well?
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,49 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, pointing to a DLManagedTensor.
+ """
+ cdef CResult[DLManagedTensor*] c_dlm_tensor
+
+ if stream is None:
+ c_dlm_tensor = ExportToDLPack(pyarrow_unwrap_array(self))
+ dlm_tensor = GetResultValue(c_dlm_tensor)
+
+ return PyCapsule_New(dlm_tensor, 'dltensor', dlpack_pycapsule_deleter)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
+
+ Returns
+ -------
+ tuple : Tuple[DLDeviceType, int]
+ Tuple with enumerator specifying the type of the device
+ and index of the device which is 0 by default for CPU.
+ """
+ cdef CResult[DLDevice] c_device
+ c_device = ExportDevice(pyarrow_unwrap_array(self))
+ device = GetResultValue(c_device)
Review Comment:
```suggestion
device = GetResultValue(ExportDevice(self.sp_array))
```
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,49 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, pointing to a DLManagedTensor.
+ """
+ cdef CResult[DLManagedTensor*] c_dlm_tensor
+
+ if stream is None:
+ c_dlm_tensor = ExportToDLPack(pyarrow_unwrap_array(self))
+ dlm_tensor = GetResultValue(c_dlm_tensor)
+
+ return PyCapsule_New(dlm_tensor, 'dltensor', dlpack_pycapsule_deleter)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
+
+ Returns
+ -------
+ tuple : Tuple[DLDeviceType, int]
Review Comment:
Does it really return a "DLDeviceType"? In the tests, the result is compared to an int.
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,49 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, pointing to a DLManagedTensor.
+ """
+ cdef CResult[DLManagedTensor*] c_dlm_tensor
+
+ if stream is None:
+ c_dlm_tensor = ExportToDLPack(pyarrow_unwrap_array(self))
+ dlm_tensor = GetResultValue(c_dlm_tensor)
+
+ return PyCapsule_New(dlm_tensor, 'dltensor', dlpack_pycapsule_deleter)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
Review Comment:
Hmm, can the docstring actually describe the "operation"?
Something like:
```suggestion
Return the DLPack device this arrays resides on.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "kkraus14 (via GitHub)" <gi...@apache.org>.
kkraus14 commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1424719762
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+#include "arrow/type_traits.h"
+
+namespace arrow::dlpack {
+
+namespace {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type.bit_width();
+ switch (type.id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ return dtype;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ return dtype;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ return dtype;
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ type.ToString());
+ }
+}
+
+} // namespace
+struct ManagerCtx {
+ std::shared_ptr<ArrayData> ref;
+ DLManagedTensor tensor;
+};
+
+Result<DLManagedTensor*> ExportArray(const std::shared_ptr<Array>& arr) {
+ // Define DLDevice struct nad check if array type is supported
+ // by the DLPack protocol at the same time. Raise TypeError if not.
+ // Supported data types: int, uint, float with no validity buffer.
+ ARROW_ASSIGN_OR_RAISE(auto device, ExportDevice(arr))
+
+ // Define the DLDataType struct
+ const DataType* arrow_type = arr->type().get();
+ ARROW_ASSIGN_OR_RAISE(auto dlpack_type, GetDLDataType(*arrow_type));
+
+ // Create ManagerCtx with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ std::unique_ptr<ManagerCtx> ctx(new ManagerCtx);
Review Comment:
nitpick: I think it's best practice to use `std::make_unique` instead of wrapping `new` call. Any reason we can't use `make_unique` here?
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+#include "arrow/type_traits.h"
+
+namespace arrow::dlpack {
+
+namespace {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type.bit_width();
+ switch (type.id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ return dtype;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ return dtype;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ return dtype;
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ type.ToString());
+ }
+}
+
+} // namespace
+struct ManagerCtx {
+ std::shared_ptr<ArrayData> ref;
+ DLManagedTensor tensor;
+};
+
+Result<DLManagedTensor*> ExportArray(const std::shared_ptr<Array>& arr) {
+ // Define DLDevice struct nad check if array type is supported
+ // by the DLPack protocol at the same time. Raise TypeError if not.
+ // Supported data types: int, uint, float with no validity buffer.
+ ARROW_ASSIGN_OR_RAISE(auto device, ExportDevice(arr))
+
+ // Define the DLDataType struct
+ const DataType* arrow_type = arr->type().get();
+ ARROW_ASSIGN_OR_RAISE(auto dlpack_type, GetDLDataType(*arrow_type));
+
+ // Create ManagerCtx with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ std::unique_ptr<ManagerCtx> ctx(new ManagerCtx);
+ ctx->ref = array_ref;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
+ if (arr->length() == 0) {
+ ctx->tensor.dl_tensor.data = NULL;
+ } else if (arr->offset() > 0) {
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ARROW_ASSIGN_OR_RAISE(auto sliced_buffer,
+ SliceBufferSafe(array_ref->buffers[1], start));
+ ctx->tensor.dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(sliced_buffer->address()));
+ } else {
+ ctx->tensor.dl_tensor.data = const_cast<void*>(
+ reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+ }
+
+ ctx->tensor.dl_tensor.device = device;
+ ctx->tensor.dl_tensor.ndim = 1;
+ ctx->tensor.dl_tensor.dtype = dlpack_type;
+ ctx->tensor.dl_tensor.shape = const_cast<int64_t*>(&array_ref->length);
+ ctx->tensor.dl_tensor.strides = NULL;
+ ctx->tensor.dl_tensor.byte_offset = 0;
+
+ ctx->tensor.manager_ctx = ctx.get();
+ ctx->tensor.deleter = [](struct DLManagedTensor* self) {
+ delete reinterpret_cast<ManagerCtx*>(self->manager_ctx);
+ };
+ return &ctx.release()->tensor;
+}
+
+Result<DLDevice> ExportDevice(const std::shared_ptr<Array>& arr) {
+ // Check if array is supported by the DLPack protocol.
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use DLPack on arrays with no nulls.");
+ }
+ const DataType* arrow_type = arr->type().get();
+ if (arrow_type->id() == Type::BOOL) {
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ }
+ if (!is_integer(arrow_type->id()) && !is_floating(arrow_type->id())) {
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ arrow_type->ToString());
+ }
+
+ // Define DLDevice struct
+ DLDevice device;
+ if (arr->data()->buffers[1]->device_type() == DeviceAllocationType::kCPU) {
Review Comment:
I think it should be safe based on the type checks above, but just confirming if we had a type of `NA` that we'd error before hitting here, since I imagine that accessing `buffers[1]` wouldn't be safe on that type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1849348288
Thank you for all you comments @kkraus14 @pitrou @jorisvandenbossche ! ❤️
I think this PR is ready for another round of review.
(Can't see the reason for the linter build to keep failing though ...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1859685222
@pitrou would you have some time to run through this PR again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1831545968
Revision: 8551650e629eae5d4131e54c76a94c691cffd000
Submitted crossbow builds: [ursacomputing/crossbow @ actions-34584f1b70](https://github.com/ursacomputing/crossbow/branches/all?query=actions-34584f1b70)
|Task|Status|
|----|------|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/7030667128/job/19130661223)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1831542420
@github-actions crossbow submit test-cuda-python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1410489744
##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -1199,6 +1199,60 @@ cdef extern from "arrow/api.h" namespace "arrow" nogil:
shared_ptr[CScalar] MakeNullScalar(shared_ptr[CDataType] type)
+cdef extern from "arrow/dlpack_structure.h" nogil:
+ cdef enum DLDeviceType:
Review Comment:
Done: https://github.com/apache/arrow/pull/38472/commits/ee6c7fac3e6a150bf8a3cad4a6c1f15121144194
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1410714596
##########
cpp/src/arrow/c/dlpack_test.cc:
##########
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+// using ExportArray = arrow::dlpack::ExportArray;
+
+class TestExportArray : public ::testing::Test {
+ public:
+ void SetUp() {}
+};
+
+static std::vector<std::shared_ptr<DataType>> TestExportArrayAgainstTheseTypes() {
+ return {
+ int8(), uint8(), int16(), uint16(), int32(), uint32(),
+ int64(), uint64(), float16(), float32(), float64(),
+ };
+}
+
+static std::vector<DLDataTypeCode> TestExpectedDLPackDataTypes() {
+ return {
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt,
+ DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt,
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLFloat,
+ DLDataTypeCode::kDLFloat, DLDataTypeCode::kDLFloat,
+ };
+}
+
+auto check_dlptensor = [](const std::shared_ptr<Array>& arr,
+ std::shared_ptr<DataType> arrow_type,
+ DLDataTypeCode dlpack_type, int64_t length) {
+ DLManagedTensor* dlmtensor;
+ ASSERT_OK(arrow::dlpack::ExportArray(arr, &dlmtensor));
+ auto dltensor = dlmtensor->dl_tensor;
+
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ASSERT_OK_AND_ASSIGN(auto sliced_buffer,
+ SliceBufferSafe(arr->data()->buffers[1], start));
+ ASSERT_EQ(sliced_buffer->data(), dltensor.data);
+
+ ASSERT_EQ(0, dltensor.byte_offset);
+ ASSERT_EQ(NULL, dltensor.strides);
+ ASSERT_EQ(length, dltensor.shape[0]);
+ ASSERT_EQ(1, dltensor.ndim);
+
+ ASSERT_EQ(dlpack_type, dltensor.dtype.code);
+
+ ASSERT_EQ(arrow_type->bit_width(), dltensor.dtype.bits);
+ ASSERT_EQ(1, dltensor.dtype.lanes);
+ ASSERT_EQ(DLDeviceType::kDLCPU, dltensor.device.device_type);
+ ASSERT_EQ(0, dltensor.device.device_id);
+};
+
+TEST_F(TestExportArray, TestSupportedArray) {
+ random::RandomArrayGenerator gen(0);
+
+ for (int64_t i = 0; i < 11; ++i) {
+ const std::shared_ptr<Array> array =
+ gen.ArrayOf(TestExportArrayAgainstTheseTypes()[i], 10, 0);
+ check_dlptensor(array, TestExportArrayAgainstTheseTypes()[i],
Review Comment:
You can probably define the vector once before the for loop, and then access the i'th element from that variable within the for loop.
Also, if this is only used for one test, I assume we could also define the actual vector data (the list of types) here within the test, instead of calling a function that returns the vector?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1409364946
##########
python/pyarrow/includes/libarrow.pxd:
##########
@@ -1199,6 +1199,60 @@ cdef extern from "arrow/api.h" namespace "arrow" nogil:
shared_ptr[CScalar] MakeNullScalar(shared_ptr[CDataType] type)
+cdef extern from "arrow/dlpack_structure.h" nogil:
+ cdef enum DLDeviceType:
Review Comment:
Ah, yes. Now I think you can also declare only those fields that you need, i.e. the deleter (the actual definition of the struct still lives in the .h file):
```
ctypedef struct DLManagedTensor:
void (*deleter)(DLManagedTensor*)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1402457065
##########
cpp/src/arrow/dlpack.h:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+
+namespace arrow {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// Returns pointer to the DLManagedTensor class defined by
+// the DLPack protocol
+ARROW_EXPORT
+DLManagedTensor* ExportToDLPack(const std::shared_ptr<Array>& arr);
Review Comment:
You could put this in a `arrow::dlpack` namespace, so that you don't have to put "DLPack" everywhere in method names etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1414141534
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,40 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, containing a DLPackManagedTensor.
+ """
+ if stream is None:
+ return to_dlpack(self)
+ else:
+ raise NotImplementedError(
+ "Only stream=None is supported."
+ )
+
+ def __dlpack_device__(self):
+ """
+ Performs the operation __dlpack_device__.
Review Comment:
Where is this specified? I see nothing in https://dmlc.github.io/dlpack/latest/python_spec.html
##########
cpp/src/arrow/c/dlpack.h:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+
+namespace arrow {
+
+namespace dlpack {
Review Comment:
Nit: can use C++17 nested namespace declarations
```suggestion
namespace arrow::dlpack {
```
(you'll have to change the closing braces as well)
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
Review Comment:
1. naming style, 2. Result instead of out-ptr, 3. no need for shared_ptr here :-)
```suggestion
Result<DLDataType> GetDLDataType(const DataType& type) {
```
##########
cpp/src/arrow/c/dlpack.h:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// DLMangedTensor is produced from an array as defined by
+/// the DLPack protocol, see https://dmlc.github.io/dlpack/latest/.
+///
+/// Data types for which the protocol is supported are
+/// primitive data types without NullType, BooleanType and
+/// Decimal types.
+///
+/// DLPack protocol only supports arrays with one contiguous
+/// memory region which means Arrow Arrays with validity buffers
+/// are not supported.
+///
+/// \param[in] arr Arrow array
+/// \param[out] out DLManagedTensor struct
+/// \return Status
+ARROW_EXPORT
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out);
+
+/// \brief Get DLDevice with enumerator specifying the
+/// type of the device data is stored on and index of the
+/// device which is 0 by default for CPU.
+///
+/// \param[in] arr Arrow array
+/// \param[out] out DLDevice struct
+/// \return DLDevice
+ARROW_EXPORT
+Status ExportDevice(const std::shared_ptr<Array>& arr, DLDevice* out);
Review Comment:
```suggestion
Result<DLDevice> ExportDevice(const std::shared_ptr<Array>& arr);
```
##########
cpp/src/arrow/c/dlpack_test.cc:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+
+namespace arrow {
+
+namespace dlpack {
Review Comment:
Nit
```suggestion
namespace arrow::dlpack {
```
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
Review Comment:
Do we plan to later support `fixed_shape_tensor` extension arrays?
##########
cpp/src/arrow/c/dlpack.h:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+/// \brief DLPack protocol for producing DLManagedTensor
+///
+/// DLMangedTensor is produced from an array as defined by
+/// the DLPack protocol, see https://dmlc.github.io/dlpack/latest/.
+///
+/// Data types for which the protocol is supported are
+/// primitive data types without NullType, BooleanType and
+/// Decimal types.
+///
+/// DLPack protocol only supports arrays with one contiguous
+/// memory region which means Arrow Arrays with validity buffers
+/// are not supported.
+///
+/// \param[in] arr Arrow array
+/// \param[out] out DLManagedTensor struct
+/// \return Status
+ARROW_EXPORT
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out);
Review Comment:
Instead of returning a Status and writing into a separate out pointer, we should simply return a Result:
```suggestion
Result<DLManagedTensor*> ExportArray(const std::shared_ptr<Array>& arr);
```
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
+ if (arr->length() == 0) {
+ dlm_tensor->dl_tensor.data = NULL;
+ } else if (arr->offset() > 0) {
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ARROW_ASSIGN_OR_RAISE(auto sliced_buffer,
+ SliceBufferSafe(array_ref->buffers[1], start));
+ dlm_tensor->dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(sliced_buffer->address()));
+ } else {
+ dlm_tensor->dl_tensor.data = const_cast<void*>(
+ reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+ }
+
+ // Define DLDevice struct
+ DLDevice ctx;
+ if (array_ref->buffers[1]->device_type() == DeviceAllocationType::kCPU) {
+ ctx.device_id = 0;
+ ctx.device_type = DLDeviceType::kDLCPU;
+ } else {
+ return Status::NotImplemented(
+ "DLPack support is implemented only for buffers on CPU device.");
+ }
+ dlm_tensor->dl_tensor.device = ctx;
+
+ dlm_tensor->dl_tensor.ndim = 1;
+ dlm_tensor->dl_tensor.dtype = arr_type;
+ std::vector<int64_t>* shape_arr = &DLMTensor->shape;
+ shape_arr->resize(1);
+ (*shape_arr)[0] = arr->length();
+ dlm_tensor->dl_tensor.shape = shape_arr->data();
+ dlm_tensor->dl_tensor.strides = NULL;
+ dlm_tensor->dl_tensor.byte_offset = 0;
+
+ *out = dlm_tensor;
+ return Status::OK();
+}
+
+Status ExportDevice(const std::shared_ptr<Array>& arr, DLDevice* out) {
+ DLDevice device;
+ if (arr->data()->buffers[1]->device_type() == DeviceAllocationType::kCPU) {
Review Comment:
We shouldn't blindly access buffer #1.
Instead we should first check that the array type is supported by DLPack, and raise a `TypeError` if not.
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ cdef PyObject* err_type
+ cdef PyObject* err_value
+ cdef PyObject* err_traceback
+
+ # Do nothing if the capsule has been consumed
+ if cpython.PyCapsule_IsValid(dltensor, "used_dltensor"):
+ return
+
+ # An exception may be in-flight, we must save it in case
+ # we create another one
+ cpython.PyErr_Fetch(&err_type, &err_value, &err_traceback)
+
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+ else:
+ cpython.PyErr_WriteUnraisable(dltensor)
+
+ # Set the error indicator from err_type, err_value, err_traceback
+ cpython.PyErr_Restore(err_type, err_value, err_traceback)
+
+
+cpdef object to_dlpack(Array arr) except *:
+
+ cdef DLManagedTensor* dlm_tensor
+ check_status(ExportToDLPack(pyarrow_unwrap_array(arr), &dlm_tensor))
+
+ return PyCapsule_New(dlm_tensor, 'dltensor', pycapsule_deleter)
+
+cpdef object dlpack_device(Array arr) except *:
+
+ cdef DLDevice device
+ check_status(ExportDevice(pyarrow_unwrap_array(arr), &device))
+
+ return (device.device_type, device.device_id)
Review Comment:
This code could go directly in the Array method implementations, I don't think it's useful to have them here.
##########
python/pyarrow/tests/test_array.py:
##########
@@ -3546,3 +3548,123 @@ def test_run_end_encoded_from_buffers():
with pytest.raises(ValueError):
pa.RunEndEncodedArray.from_buffers(ree_type, length, buffers,
1, offset, children)
+
+
+def PyCapsule_IsValid(capsule, name):
+ return ctypes.pythonapi.PyCapsule_IsValid(ctypes.py_object(capsule), name) == 1
+
+
+@pytest.mark.parametrize(
+ ('value_type', 'np_type'),
+ [
+ (pa.uint8(), np.uint8),
+ (pa.uint16(), np.uint16),
+ (pa.uint32(), np.uint32),
+ (pa.uint64(), np.uint64),
+ (pa.int8(), np.int8),
+ (pa.int16(), np.int16),
+ (pa.int32(), np.int32),
+ (pa.int64(), np.int64),
+ (pa.float16(), np.float16),
+ (pa.float32(), np.float32),
+ (pa.float64(), np.float64),
+ ]
+)
+def test_dlpack(value_type, np_type):
+ if Version(np.__version__) < Version("1.24.0"):
+ pytest.skip("No dlpack support in numpy versions older than 1.22.0, "
+ "strict keyward in assert_array_equal added in numpy version "
Review Comment:
```suggestion
"strict keyword in assert_array_equal added in numpy version "
```
##########
python/pyarrow/array.pxi:
##########
@@ -1778,6 +1778,40 @@ cdef class Array(_PandasConvertible):
return pyarrow_wrap_array(array)
+ def __dlpack__(self, stream=None):
+ """Export a primitive array as a DLPack capsule.
+
+ Parameters
+ ----------
+ stream : int, optional
+ A Python integer representing a pointer to a stream. Currently not supported.
+ Stream is provided by the consumer to the producer to instruct the producer
+ to ensure that operations can safely be performed on the array.
+
+ Returns
+ -------
+ capsule : PyCapsule
+ A DLPack capsule for the array, containing a DLPackManagedTensor.
Review Comment:
```suggestion
A DLPack capsule for the array, pointing to a DLManagedTensor.
```
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
Review Comment:
DLPack also doesn't support temporals, strings, etc.
```suggestion
return Status::TypeError(
"DataType is not compatible with DLPack spec: ", type.ToString());
```
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
+ if (arr->length() == 0) {
+ dlm_tensor->dl_tensor.data = NULL;
+ } else if (arr->offset() > 0) {
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ARROW_ASSIGN_OR_RAISE(auto sliced_buffer,
+ SliceBufferSafe(array_ref->buffers[1], start));
+ dlm_tensor->dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(sliced_buffer->address()));
+ } else {
+ dlm_tensor->dl_tensor.data = const_cast<void*>(
+ reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+ }
+
+ // Define DLDevice struct
+ DLDevice ctx;
+ if (array_ref->buffers[1]->device_type() == DeviceAllocationType::kCPU) {
+ ctx.device_id = 0;
+ ctx.device_type = DLDeviceType::kDLCPU;
+ } else {
+ return Status::NotImplemented(
+ "DLPack support is implemented only for buffers on CPU device.");
+ }
+ dlm_tensor->dl_tensor.device = ctx;
+
+ dlm_tensor->dl_tensor.ndim = 1;
+ dlm_tensor->dl_tensor.dtype = arr_type;
+ std::vector<int64_t>* shape_arr = &DLMTensor->shape;
+ shape_arr->resize(1);
+ (*shape_arr)[0] = arr->length();
+ dlm_tensor->dl_tensor.shape = shape_arr->data();
Review Comment:
This is not necessary, we could simply point to the existing length and remove `DLMTensorCtx::shape`:
```suggestion
dlm_tensor->dl_tensor.shape = const_cast<int64_t*>(&array_ref->length);
```
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
Review Comment:
Let's not use the Python spelling as this could be used directly from third-party C++ code:
```suggestion
return Status::TypeError("Can only use DLPack on arrays with no nulls.");
```
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ cdef PyObject* err_type
+ cdef PyObject* err_value
+ cdef PyObject* err_traceback
+
+ # Do nothing if the capsule has been consumed
+ if cpython.PyCapsule_IsValid(dltensor, "used_dltensor"):
+ return
+
+ # An exception may be in-flight, we must save it in case
+ # we create another one
+ cpython.PyErr_Fetch(&err_type, &err_value, &err_traceback)
+
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+ else:
+ cpython.PyErr_WriteUnraisable(dltensor)
Review Comment:
This writes the raised exception but nothing is raised by `PyCapsule_IsValid`. Instead, you can probably call `PyCapsule_GetPointer` directly and check the result...
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
+ if (arr->length() == 0) {
+ dlm_tensor->dl_tensor.data = NULL;
+ } else if (arr->offset() > 0) {
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ARROW_ASSIGN_OR_RAISE(auto sliced_buffer,
+ SliceBufferSafe(array_ref->buffers[1], start));
+ dlm_tensor->dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(sliced_buffer->address()));
+ } else {
+ dlm_tensor->dl_tensor.data = const_cast<void*>(
+ reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+ }
+
+ // Define DLDevice struct
+ DLDevice ctx;
+ if (array_ref->buffers[1]->device_type() == DeviceAllocationType::kCPU) {
+ ctx.device_id = 0;
+ ctx.device_type = DLDeviceType::kDLCPU;
+ } else {
+ return Status::NotImplemented(
+ "DLPack support is implemented only for buffers on CPU device.");
+ }
Review Comment:
We should not duplicate this code since this is already implemented in `ExportDevice` below...
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
Is this specified somewhere?
##########
python/pyarrow/tests/test_array.py:
##########
@@ -3546,3 +3548,123 @@ def test_run_end_encoded_from_buffers():
with pytest.raises(ValueError):
pa.RunEndEncodedArray.from_buffers(ree_type, length, buffers,
1, offset, children)
+
+
+def PyCapsule_IsValid(capsule, name):
+ return ctypes.pythonapi.PyCapsule_IsValid(ctypes.py_object(capsule), name) == 1
+
+
+@pytest.mark.parametrize(
+ ('value_type', 'np_type'),
+ [
+ (pa.uint8(), np.uint8),
+ (pa.uint16(), np.uint16),
+ (pa.uint32(), np.uint32),
+ (pa.uint64(), np.uint64),
+ (pa.int8(), np.int8),
+ (pa.int16(), np.int16),
+ (pa.int32(), np.int32),
+ (pa.int64(), np.int64),
+ (pa.float16(), np.float16),
+ (pa.float32(), np.float32),
+ (pa.float64(), np.float64),
+ ]
+)
+def test_dlpack(value_type, np_type):
+ if Version(np.__version__) < Version("1.24.0"):
+ pytest.skip("No dlpack support in numpy versions older than 1.22.0, "
+ "strict keyward in assert_array_equal added in numpy version "
+ "1.24.0")
+
+ expected = np.array([1, 2, 3], dtype=np_type)
+ arr = pa.array(expected, type=value_type)
+ DLTensor = arr.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ result = np.from_dlpack(arr)
+ np.testing.assert_array_equal(result, expected, strict=True)
+
+ assert arr.__dlpack_device__() == (1, 0)
+
+ arr_sliced = arr.slice(1, 1)
+ DLTensor = arr_sliced.__dlpack__()
+ assert PyCapsule_IsValid(DLTensor, b"dltensor") is True
+ expected = np.array([2], dtype=np_type)
+ result = np.from_dlpack(arr_sliced)
+ np.testing.assert_array_equal(result, expected, strict=True)
Review Comment:
Can we avoid such copy-pasting in tests? There could be some test helper(s) such that you would write, e.g.:
```suggestion
check_dlpack_export(arr_sliced, expected=np.array([2], dtype=np_type))
```
##########
python/pyarrow/tests/test_array.py:
##########
@@ -3546,3 +3548,123 @@ def test_run_end_encoded_from_buffers():
with pytest.raises(ValueError):
pa.RunEndEncodedArray.from_buffers(ree_type, length, buffers,
1, offset, children)
+
Review Comment:
Hmm, can we move all these tests into a separate `test_dlpack.py`?
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
Indeed, something like this:
```c++
struct ManagerCtx {
std::shared_ptr<ArrayData> ref;
DLManagedTensor tensor;
};
std::unique_ptr<ManagerCtx> ctx(new ManagerCtx);
ctx->ref = array_ref;
// fill struct fields
...
// end of function
ctx->tensor->manager_ctx = ctx.release();
ctx->tensor->deleter = [](struct DLManagedTensor *self) {
delete reinterpret_cast<ManagerCtx*>(self->manager_ctx);
};
return &ctx->tensor;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417048761
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
Ahem, sorry for the bad suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1373289089
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
Review Comment:
Added noexcept here to see if Cython 3 would be happy.
I do not think any Python exceptions can raise, but not sure ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1373359435
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+
+
+cdef void deleter(DLManagedTensor* tensor) with gil:
+ if tensor.manager_ctx is NULL:
+ return
+ free(tensor.dl_tensor.shape)
+ cpython.Py_DECREF(<Array>tensor.manager_ctx)
+ tensor.manager_ctx = NULL
+ free(tensor)
+
+
+cpdef object to_dlpack(Array arr) except *:
+ cdef DLManagedTensor* dlm_tensor = <DLManagedTensor*> malloc(sizeof(DLManagedTensor))
+
+ cdef DLTensor* dl_tensor = &dlm_tensor.dl_tensor
+ cdef intptr_t data_ptr = arr.buffers()[1].address
+ dl_tensor.data = <void*> data_ptr
+ dl_tensor.ndim = 1
+ cdef int64_t* shape = <int64_t*> malloc(sizeof(int64_t))
+ shape[0] = len(arr)
+ dl_tensor.shape = shape
+
+ dl_tensor.strides = NULL
+ dl_tensor.byte_offset = 0
+
+ cdef DLDevice* device = &dl_tensor.device
+ device.device_type = kDLCPU
+ device.device_id = 0
+
+ cdef DLDataType* dtype = &dl_tensor.dtype
+ if arr.type in [uint8(), uint16(), uint32(), uint64()]:
+ dtype.code = kDLUInt
+ elif arr.type in [int8(), int16(), int32(), int64()]:
+ dtype.code = kDLInt
+ elif arr.type in [float16(), float32(), float64()]:
+ dtype.code = kDLFloat
+ elif arr.type == bool_():
+ dtype.code = kDLBool
+
+ dtype.lanes = <uint16_t>1
+ dtype.bits = <uint8_t>arr.type.bit_width
+
+ dlm_tensor.manager_ctx = <void*>arr
+ cpython.Py_INCREF(arr)
Review Comment:
Hmm... this is probably ok for a prototype, but I think we would rather track the underlying C++ array here?
@kkraus14 What do you think?
Also, to be honest, I would probably write this file in C++ rather than Cython, personally. It would probably be more natural, but I may be biased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1374289604
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+
+
+cdef void deleter(DLManagedTensor* tensor) with gil:
+ if tensor.manager_ctx is NULL:
+ return
+ free(tensor.dl_tensor.shape)
+ cpython.Py_DECREF(<Array>tensor.manager_ctx)
+ tensor.manager_ctx = NULL
+ free(tensor)
+
+
+cpdef object to_dlpack(Array arr) except *:
+ cdef DLManagedTensor* dlm_tensor = <DLManagedTensor*> malloc(sizeof(DLManagedTensor))
Review Comment:
The Python part of the spec still mentions to use DLManagedTensor, and not DLManagedTensorVersioned (https://dmlc.github.io/dlpack/latest/python_spec.html).
So maybe on the C++ side we could support both, but at least for the python PyCapsule side we need DLManagedTensor, to my understanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1852053787
@pitrou thank you so much for the reviews! I addressed all your comments so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1422886863
##########
cpp/src/arrow/c/dlpack_test.cc:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/testing/random.h"
+
+namespace arrow::dlpack {
+
+class TestExportArray : public ::testing::Test {
+ public:
+ void SetUp() {}
+};
+
+auto check_dlptensor = [](const std::shared_ptr<Array>& arr,
+ std::shared_ptr<DataType> arrow_type,
+ DLDataTypeCode dlpack_type, int64_t length) {
+ ASSERT_OK_AND_ASSIGN(auto dlmtensor, arrow::dlpack::ExportArray(arr));
+ auto dltensor = dlmtensor->dl_tensor;
+
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ASSERT_OK_AND_ASSIGN(auto sliced_buffer,
+ SliceBufferSafe(arr->data()->buffers[1], start));
+ ASSERT_EQ(sliced_buffer->data(), dltensor.data);
+
+ ASSERT_EQ(0, dltensor.byte_offset);
+ ASSERT_EQ(NULL, dltensor.strides);
+ ASSERT_EQ(length, dltensor.shape[0]);
+ ASSERT_EQ(1, dltensor.ndim);
+
+ ASSERT_EQ(dlpack_type, dltensor.dtype.code);
+
+ ASSERT_EQ(arrow_type->bit_width(), dltensor.dtype.bits);
+ ASSERT_EQ(1, dltensor.dtype.lanes);
+ ASSERT_EQ(DLDeviceType::kDLCPU, dltensor.device.device_type);
+ ASSERT_EQ(0, dltensor.device.device_id);
+
+ ASSERT_OK_AND_ASSIGN(auto device, arrow::dlpack::ExportDevice(arr));
+ ASSERT_EQ(DLDeviceType::kDLCPU, device.device_type);
+ ASSERT_EQ(0, device.device_id);
+
+ dlmtensor->deleter(dlmtensor);
+};
+
+TEST_F(TestExportArray, TestSupportedArray) {
+ random::RandomArrayGenerator gen(0);
+
+ std::vector<std::shared_ptr<DataType>> arrow_types = {
+ int8(), uint8(), int16(), uint16(), int32(), uint32(),
+ int64(), uint64(), float16(), float32(), float64(),
+ };
+
+ std::vector<DLDataTypeCode> dlpack_types = {
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt,
+ DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt,
+ DLDataTypeCode::kDLInt, DLDataTypeCode::kDLUInt, DLDataTypeCode::kDLFloat,
+ DLDataTypeCode::kDLFloat, DLDataTypeCode::kDLFloat,
+ };
+
+ for (int64_t i = 0; i < 11; ++i) {
+ const std::shared_ptr<Array> array = gen.ArrayOf(arrow_types[i], 10, 0);
+ check_dlptensor(array, arrow_types[i], dlpack_types[i], 10);
+ ASSERT_OK_AND_ASSIGN(auto sliced_1, array->SliceSafe(1, 5));
+ check_dlptensor(sliced_1, arrow_types[i], dlpack_types[i], 5);
+ ASSERT_OK_AND_ASSIGN(auto sliced_2, array->SliceSafe(0, 5));
+ check_dlptensor(sliced_2, arrow_types[i], dlpack_types[i], 5);
+ ASSERT_OK_AND_ASSIGN(auto sliced_3, array->SliceSafe(3));
+ check_dlptensor(sliced_3, arrow_types[i], dlpack_types[i], 7);
+ }
+}
+
+TEST_F(TestExportArray, TestUnSupportedArray) {
+ random::RandomArrayGenerator gen(0);
Review Comment:
Yeah, will not explain why I decided to use it :D
Will change, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [C++][Python] DLPack implementation for Arrow Arrays (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1863213860
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417114824
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
Works 👍 Thank you!
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
Works 👍 Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1416911334
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
Ah, I thought it only returns a "Pointer to the managed object" and releases when the deleter is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "leofang (via GitHub)" <gi...@apache.org>.
leofang commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417451657
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
I think the convention came from the old days of CUDA Array Interface where either `None` or `0` could be used (https://github.com/numba/numba/issues/4140), but I guess it doesn't really matter to a C interface. Indeed, `shape` can be used to check whether an array is 0-size, but why bother asking the consumer to do an extra check when the producer already has this knowledge?
From the ergonomics perspective, we know the pointer to a 0-size array is meaningless and shouldn't be used, and all libraries handle this case differently (ex: NumPy would return a pointer address that likely comes from its private memory reservation, while CuPy returns null), so setting it to 0 universally was the safest thing to align all libraries and tell the consumer "do not use the pointer."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "rgommers (via GitHub)" <gi...@apache.org>.
rgommers commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417031796
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
It doesn't mandate that it must be NULL, but rather that it must be "NULL or 0". That seems reasonable to me, even though in theory any values for strides works. It is reasonable because there is enough evidence that in practice users regularly make mistakes with these strides - which is why for a long time NumPy had a "relaxed strides" build option to put garbage values in the strides so mistakes with them could be caught.
These Python API docs here simply transcribed what was already contained in a code comment: https://github.com/dmlc/dlpack/blame/main/include/dlpack/dlpack.h#L227. This seems to have been the requirement in DLPack from the start, see https://github.com/dmlc/dlpack/issues/35.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1406060174
##########
cpp/src/arrow/dlpack_structure.h:
##########
Review Comment:
https://github.com/apache/arrow/pull/38472/commits/98392e70b3af71762949a9c6095b16d5d16310eb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1407902250
##########
cpp/src/arrow/dlpack.cc:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/dlpack_structure.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+DLDataType getDLDataType(const Array& arr) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = arr.type()->bit_width();
+ switch (arr.type()->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ break;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ break;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ break;
+ case Type::BOOL:
+ dtype.code = DLDataTypeCode::kDLBool;
+ break;
+ default:
+ // TODO
+ break;
+ }
+ return dtype;
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+DLManagedTensor* toDLPack(const Array& arr) {
+ std::shared_ptr<ArrayData> array_ref = arr.data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+ dlm_tensor->dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+
+ DLDevice ctx;
+ ctx.device_id = 0;
+ ctx.device_type = DLDeviceType::kDLCPU;
Review Comment:
Added a check for CPU device: https://github.com/apache/arrow/pull/38472/commits/6c886fdec91e06fd5c93bf43ff0e3a2f133f45ca
It would be easy to add support for other cases in a follow-up I think. Not sure about the tests though =)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1412356415
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,82 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+Producing side of the DLPack Protocol is implemented for ``pa.Array``
+and can be used to interchange data between PyArrow and other tensor
+libraries. The data structures that are supported in the implementation
+of the protocol are integer, unsigned integer and float arrays. The
+protocol has no missing data support meaning PyArrow arrays with
+validity mask can not be used to transfer data through the DLPack
Review Comment:
```suggestion
protocol has no missing data support meaning PyArrow arrays with
missing values cannot be used to transfer data through the DLPack
```
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,82 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+Producing side of the DLPack Protocol is implemented for ``pa.Array``
+and can be used to interchange data between PyArrow and other tensor
+libraries. The data structures that are supported in the implementation
+of the protocol are integer, unsigned integer and float arrays. The
+protocol has no missing data support meaning PyArrow arrays with
+validity mask can not be used to transfer data through the DLPack
+protocol. Currently Arrow implementation of the protocol only supports
+data on a CPU device.
+
+The DLPack Protocol is
+`selected as the Python array API standard <https://data-apis.org/array-api/latest/design_topics/data_interchange.html?highlight=dlpack#dlpack-an-in-memory-tensor-structure>`_
Review Comment:
```suggestion
`selected as the Python array API standard <https://data-apis.org/array-api/latest/design_topics/data_interchange.html#dlpack-an-in-memory-tensor-structure>`_
```
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,82 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+Producing side of the DLPack Protocol is implemented for ``pa.Array``
+and can be used to interchange data between PyArrow and other tensor
+libraries. The data structures that are supported in the implementation
+of the protocol are integer, unsigned integer and float arrays. The
+protocol has no missing data support meaning PyArrow arrays with
+validity mask can not be used to transfer data through the DLPack
+protocol. Currently Arrow implementation of the protocol only supports
+data on a CPU device.
+
+The DLPack Protocol is
Review Comment:
Some text suggestions: I would maybe first explain DLPack before saying what is implemented by pyarrow (so moving this paragraph first), and then within this paragraph I think you can also explain first what DLPack is and then mention it is selected by the consortium.
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,82 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+Producing side of the DLPack Protocol is implemented for ``pa.Array``
+and can be used to interchange data between PyArrow and other tensor
+libraries. The data structures that are supported in the implementation
+of the protocol are integer, unsigned integer and float arrays. The
+protocol has no missing data support meaning PyArrow arrays with
+validity mask can not be used to transfer data through the DLPack
+protocol. Currently Arrow implementation of the protocol only supports
+data on a CPU device.
+
+The DLPack Protocol is
+`selected as the Python array API standard <https://data-apis.org/array-api/latest/design_topics/data_interchange.html?highlight=dlpack#dlpack-an-in-memory-tensor-structure>`_
+by the
+`Consortium for Python Data API Standards <https://data-apis.org/>`_
+in order to enable device aware data interchange between array/tensor
+libraries in the Python ecosystem. Being device aware allows exchange
+of data on devices other than the CPU (e.g. GPU). See more about the standard
+in the
+`protocol documentation <https://data-apis.org/array-api/latest/index.html>`_
+and more about the DLPack in the
+`Python Specification for DLPack <https://dmlc.github.io/dlpack/latest/python_spec.html#python-spec>`_.
+
+Data interchange syntax of the protocol includes
+
+1. ``from_dlpack(x)``: consuming an array object that implements a ``__dlpack__`å` method
Review Comment:
```suggestion
1. ``from_dlpack(x)``: consuming an array object that implements a ``__dlpack__`` method
```
##########
docs/source/python/dlpack.rst:
##########
@@ -0,0 +1,82 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _pyarrow-dlpack:
+
+The DLPack Protocol
+===================
+
+Producing side of the DLPack Protocol is implemented for ``pa.Array``
+and can be used to interchange data between PyArrow and other tensor
+libraries. The data structures that are supported in the implementation
+of the protocol are integer, unsigned integer and float arrays. The
+protocol has no missing data support meaning PyArrow arrays with
+validity mask can not be used to transfer data through the DLPack
+protocol. Currently Arrow implementation of the protocol only supports
+data on a CPU device.
+
+The DLPack Protocol is
+`selected as the Python array API standard <https://data-apis.org/array-api/latest/design_topics/data_interchange.html?highlight=dlpack#dlpack-an-in-memory-tensor-structure>`_
+by the
+`Consortium for Python Data API Standards <https://data-apis.org/>`_
+in order to enable device aware data interchange between array/tensor
+libraries in the Python ecosystem. Being device aware allows exchange
+of data on devices other than the CPU (e.g. GPU). See more about the standard
+in the
+`protocol documentation <https://data-apis.org/array-api/latest/index.html>`_
+and more about the DLPack in the
+`Python Specification for DLPack <https://dmlc.github.io/dlpack/latest/python_spec.html#python-spec>`_.
+
+Data interchange syntax of the protocol includes
+
+1. ``from_dlpack(x)``: consuming an array object that implements a ``__dlpack__`å` method
+ and creating a new array while sharing the memory.
+
+2. ``__dlpack__(self, stream=None)`` and ``__dlpack_device__``: producing a PyCapsule with
+ the DLPack struct which is called from within ``from_dlpack(x)``.
+
+PyArrow implements the second part of the protocol (``__dlpack__(self, stream=None)`` and
+``__dlpack_device__``).
Review Comment:
```suggestion
``__dlpack_device__``), and can thus be consumed by libraries implementing ``from_dlpack``.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1373504459
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
+
+
+cdef void deleter(DLManagedTensor* tensor) with gil:
+ if tensor.manager_ctx is NULL:
+ return
+ free(tensor.dl_tensor.shape)
+ cpython.Py_DECREF(<Array>tensor.manager_ctx)
+ tensor.manager_ctx = NULL
+ free(tensor)
+
+
+cpdef object to_dlpack(Array arr) except *:
+ cdef DLManagedTensor* dlm_tensor = <DLManagedTensor*> malloc(sizeof(DLManagedTensor))
+
+ cdef DLTensor* dl_tensor = &dlm_tensor.dl_tensor
+ cdef intptr_t data_ptr = arr.buffers()[1].address
+ dl_tensor.data = <void*> data_ptr
+ dl_tensor.ndim = 1
+ cdef int64_t* shape = <int64_t*> malloc(sizeof(int64_t))
+ shape[0] = len(arr)
+ dl_tensor.shape = shape
+
+ dl_tensor.strides = NULL
+ dl_tensor.byte_offset = 0
+
+ cdef DLDevice* device = &dl_tensor.device
+ device.device_type = kDLCPU
+ device.device_id = 0
+
+ cdef DLDataType* dtype = &dl_tensor.dtype
+ if arr.type in [uint8(), uint16(), uint32(), uint64()]:
+ dtype.code = kDLUInt
+ elif arr.type in [int8(), int16(), int32(), int64()]:
+ dtype.code = kDLInt
+ elif arr.type in [float16(), float32(), float64()]:
+ dtype.code = kDLFloat
+ elif arr.type == bool_():
+ dtype.code = kDLBool
+
+ dtype.lanes = <uint16_t>1
+ dtype.bits = <uint8_t>arr.type.bit_width
+
+ dlm_tensor.manager_ctx = <void*>arr
+ cpython.Py_INCREF(arr)
Review Comment:
Right. Basically everything related to `DLManagedTensor` moved to Arrow C++. The capsule can be either in Cython or PyArrow C++, as you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1835608699
> I assume this is because we don't yet have a consumer, and in those C++ tests you are exporting the array without consuming it. In that case you will have to call the deleter explicitly (in the python tests, by consuming it by numpy, it's numpy that will ensure to call the deleter when the numpy array goes out of scope)
Ah of course, thank you Joris!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417063057
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
Will try it out, thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417149146
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
Well, it looks like [this PR](https://github.com/dmlc/dlpack/pull/93) was quickly approved :-D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1417576828
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
> setting it to 0 universally was the safest thing to align all libraries and tell the consumer "do not use the pointer."
That sounds reasonable, thanks. In any case, shouldn't this be mentioned in https://dmlc.github.io/dlpack/latest/c_api.html#_CPPv4N8DLTensor4dataE rather than in a passing mention at the end of the Python spec? Otherwise, non-Python implementors of the DLPack spec won't see it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1416893060
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
Any pointers how to fix a segfault with `.release()` call on the ctx? 😬
```cpp
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x40)
frame #0: 0x00000001085d0694 libarrow.1500.dylib`arrow::dlpack::ExportArray(arr=std::__1::shared_ptr<arrow::Array>::element_type @ 0x000000011de17d88 strong=1 weak=1) at dlpack.cc:125:27
122 ctx->tensor.dl_tensor.byte_offset = 0;
123
124 // return dlm_tensor;
-> 125 ctx->tensor.manager_ctx = ctx.release();
126 ctx->tensor.deleter = [](struct DLManagedTensor *self) {
127 delete reinterpret_cast<ManagerCtx*>(self->manager_ctx);
128 };
```
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
Review Comment:
Any pointers how to fix a segfault with `.release()` call on the ctx? 😬
```cpp
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x40)
frame #0: 0x00000001085d0694 libarrow.1500.dylib`arrow::dlpack::ExportArray(arr=std::__1::shared_ptr<arrow::Array>::element_type @ 0x000000011de17d88 strong=1 weak=1) at dlpack.cc:125:27
122 ctx->tensor.dl_tensor.byte_offset = 0;
123
124 // return dlm_tensor;
-> 125 ctx->tensor.manager_ctx = ctx.release();
126 ctx->tensor.deleter = [](struct DLManagedTensor *self) {
127 delete reinterpret_cast<ManagerCtx*>(self->manager_ctx);
128 };
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1418824668
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
+ if (arr->length() == 0) {
+ dlm_tensor->dl_tensor.data = NULL;
+ } else if (arr->offset() > 0) {
+ const auto byte_width = arr->type()->byte_width();
+ const auto start = arr->offset() * byte_width;
+ ARROW_ASSIGN_OR_RAISE(auto sliced_buffer,
+ SliceBufferSafe(array_ref->buffers[1], start));
+ dlm_tensor->dl_tensor.data =
+ const_cast<void*>(reinterpret_cast<const void*>(sliced_buffer->address()));
+ } else {
+ dlm_tensor->dl_tensor.data = const_cast<void*>(
+ reinterpret_cast<const void*>(array_ref->buffers[1]->address()));
+ }
+
+ // Define DLDevice struct
+ DLDevice ctx;
+ if (array_ref->buffers[1]->device_type() == DeviceAllocationType::kCPU) {
+ ctx.device_id = 0;
+ ctx.device_type = DLDeviceType::kDLCPU;
+ } else {
+ return Status::NotImplemented(
+ "DLPack support is implemented only for buffers on CPU device.");
+ }
+ dlm_tensor->dl_tensor.device = ctx;
+
+ dlm_tensor->dl_tensor.ndim = 1;
+ dlm_tensor->dl_tensor.dtype = arr_type;
+ std::vector<int64_t>* shape_arr = &DLMTensor->shape;
+ shape_arr->resize(1);
+ (*shape_arr)[0] = arr->length();
+ dlm_tensor->dl_tensor.shape = shape_arr->data();
+ dlm_tensor->dl_tensor.strides = NULL;
+ dlm_tensor->dl_tensor.byte_offset = 0;
+
+ *out = dlm_tensor;
+ return Status::OK();
+}
+
+Status ExportDevice(const std::shared_ptr<Array>& arr, DLDevice* out) {
+ DLDevice device;
+ if (arr->data()->buffers[1]->device_type() == DeviceAllocationType::kCPU) {
Review Comment:
https://github.com/apache/arrow/pull/38472/commits/d9b3182123a0178d56468c50199941d94d01a406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1424988997
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+#include "arrow/type_traits.h"
+
+namespace arrow::dlpack {
+
+namespace {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type.bit_width();
+ switch (type.id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ return dtype;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ return dtype;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ return dtype;
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ type.ToString());
+ }
+}
+
+} // namespace
+struct ManagerCtx {
+ std::shared_ptr<ArrayData> ref;
+ DLManagedTensor tensor;
+};
+
+Result<DLManagedTensor*> ExportArray(const std::shared_ptr<Array>& arr) {
+ // Define DLDevice struct nad check if array type is supported
+ // by the DLPack protocol at the same time. Raise TypeError if not.
+ // Supported data types: int, uint, float with no validity buffer.
+ ARROW_ASSIGN_OR_RAISE(auto device, ExportDevice(arr))
+
+ // Define the DLDataType struct
+ const DataType* arrow_type = arr->type().get();
+ ARROW_ASSIGN_OR_RAISE(auto dlpack_type, GetDLDataType(*arrow_type));
+
+ // Create ManagerCtx with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ std::unique_ptr<ManagerCtx> ctx(new ManagerCtx);
Review Comment:
Happy to change to `make_unique`. @pitrou any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1425027655
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+#include "arrow/type_traits.h"
+
+namespace arrow::dlpack {
+
+namespace {
+
+Result<DLDataType> GetDLDataType(const DataType& type) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type.bit_width();
+ switch (type.id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ return dtype;
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ return dtype;
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ return dtype;
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError("DataType is not compatible with DLPack spec: ",
+ type.ToString());
+ }
+}
+
+} // namespace
+struct ManagerCtx {
+ std::shared_ptr<ArrayData> ref;
+ DLManagedTensor tensor;
+};
+
+Result<DLManagedTensor*> ExportArray(const std::shared_ptr<Array>& arr) {
+ // Define DLDevice struct nad check if array type is supported
+ // by the DLPack protocol at the same time. Raise TypeError if not.
+ // Supported data types: int, uint, float with no validity buffer.
+ ARROW_ASSIGN_OR_RAISE(auto device, ExportDevice(arr))
+
+ // Define the DLDataType struct
+ const DataType* arrow_type = arr->type().get();
+ ARROW_ASSIGN_OR_RAISE(auto dlpack_type, GetDLDataType(*arrow_type));
+
+ // Create ManagerCtx with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ std::unique_ptr<ManagerCtx> ctx(new ManagerCtx);
Review Comment:
Given that we have the explicit `delete` as well, I personally like seeing the `new`+`delete` combo explicitly, even though `make_unique` is exactly equivalent AFAIU
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1853513984
@kkraus14 I am double-checking if I am correct in assuming DLpack only supports byte-packed booleans. Or are bit-packed booleans also supported but not really used?
I am not sure what came out of the discussion in https://github.com/dmlc/dlpack/issues/75 and I am finding the docs unclear:
[example in the docs suggest underlying storage size of bool is 8 bits](https://dmlc.github.io/dlpack/latest/c_api.html#_CPPv410DLDataType), but there is no specific info about `bool1`.
Will be happy to hear your thoughts on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1406080124
##########
cpp/src/arrow/dlpack_structure.h:
##########
Review Comment:
And https://github.com/apache/arrow/pull/38472/commits/c798874fe1bbdb4b405b05dd7559f2216443240b 😊
##########
cpp/src/arrow/dlpack_structure.h:
##########
Review Comment:
And https://github.com/apache/arrow/pull/38472/commits/c798874fe1bbdb4b405b05dd7559f2216443240b 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1781158554
cc @rok: this is an initial draft for the dlpack support. Will still need to create a separate method for fixed shape tensor array plus add more tests (inspecting the produced PyCapsule with cffi).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1373480164
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
Review Comment:
Yes, I was mainly looking at CuPy and mpi4py. As we will move the code to C++ I will keep in mind the example from the dlpack docs 👍
##########
python/pyarrow/_dlpack.pxi:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from libc.stdlib cimport malloc, free
+
+cimport cpython
+from cpython.pycapsule cimport PyCapsule_New
+from cython import sizeof
+
+
+ctypedef enum DLDeviceType:
+ kDLCPU = 1
+ kDLCUDA = 2
+ kDLCUDAHost = 3
+ kDLOpenCL = 4
+ kDLVulkan = 7
+ kDLMetal = 8
+ kDLVPI = 9
+ kDLROCM = 10
+ kDLROCMHost = 11
+ kDLExtDev = 12
+ kDLCUDAManaged = 13
+ kDLOneAPI = 14
+ kDLWebGPU = 15
+ kDLHexagon = 16
+
+ctypedef struct DLDevice:
+ DLDeviceType device_type
+ int32_t device_id
+
+ctypedef enum DLDataTypeCode:
+ kDLInt = 0
+ kDLUInt = 1
+ kDLFloat = 2
+ kDLOpaqueHandle = 3
+ kDLBfloat = 4
+ kDLComplex = 5
+ kDLBool = 6
+
+ctypedef struct DLDataType:
+ uint8_t code
+ uint8_t bits
+ uint16_t lanes
+
+ctypedef struct DLTensor:
+ void *data
+ DLDevice device
+ int32_t ndim
+ DLDataType dtype
+ int64_t *shape
+ int64_t *strides
+ uint64_t byte_offset
+
+ctypedef struct DLManagedTensor:
+ DLTensor dl_tensor
+ void *manager_ctx
+ void (*deleter)(DLManagedTensor *)
+
+
+cdef void pycapsule_deleter(object dltensor) noexcept:
+ cdef DLManagedTensor* dlm_tensor
+ if cpython.PyCapsule_IsValid(dltensor, 'dltensor'):
+ dlm_tensor = <DLManagedTensor*>cpython.PyCapsule_GetPointer(
+ dltensor, 'dltensor')
+ dlm_tensor.deleter(dlm_tensor)
Review Comment:
Yes, I was mainly looking at CuPy and mpi4py. As we will move the code to C++ I will keep in mind the example from the dlpack docs 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #38472:
URL: https://github.com/apache/arrow/pull/38472#issuecomment-1831501646
@github-actions crossbow submit test-cuda-python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-33984: [Python] __dlpack__ implementation (producer) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #38472:
URL: https://github.com/apache/arrow/pull/38472#discussion_r1415635359
##########
cpp/src/arrow/c/dlpack.cc:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/dlpack.h"
+
+#include "arrow/array/array_base.h"
+#include "arrow/c/dlpack_abi.h"
+#include "arrow/device.h"
+#include "arrow/type.h"
+
+namespace arrow {
+
+namespace dlpack {
+
+Status getDLDataType(const std::shared_ptr<DataType>& type, DLDataType* out) {
+ DLDataType dtype;
+ dtype.lanes = 1;
+ dtype.bits = type->bit_width();
+ switch (type->id()) {
+ case Type::INT8:
+ case Type::INT16:
+ case Type::INT32:
+ case Type::INT64:
+ dtype.code = DLDataTypeCode::kDLInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::UINT8:
+ case Type::UINT16:
+ case Type::UINT32:
+ case Type::UINT64:
+ dtype.code = DLDataTypeCode::kDLUInt;
+ *out = dtype;
+ return Status::OK();
+ case Type::HALF_FLOAT:
+ case Type::FLOAT:
+ case Type::DOUBLE:
+ dtype.code = DLDataTypeCode::kDLFloat;
+ *out = dtype;
+ return Status::OK();
+ case Type::BOOL:
+ // DLPack supports byte-packed boolean values
+ return Status::TypeError("Bit-packed boolean data type not supported by DLPack.");
+ default:
+ return Status::TypeError(
+ "Can only use __dlpack__ on primitive arrays without NullType and Decimal "
+ "types.");
+ }
+}
+
+struct DLMTensorCtx {
+ std::shared_ptr<ArrayData> ref;
+ std::vector<int64_t> shape;
+ DLManagedTensor tensor;
+};
+
+static void deleter(DLManagedTensor* arg) {
+ delete static_cast<DLMTensorCtx*>(arg->manager_ctx);
+}
+
+Status ExportArray(const std::shared_ptr<Array>& arr, DLManagedTensor** out) {
+ if (arr->null_count() > 0) {
+ return Status::TypeError("Can only use __dlpack__ on arrays with no nulls.");
+ }
+
+ // Define the DLDataType struct
+ // Supported data types: int, uint, float
+ DLDataType arr_type;
+ RETURN_NOT_OK(getDLDataType(arr->type(), &arr_type));
+
+ // Create DLMTensorCtx struct with the reference to
+ // the data of the array
+ std::shared_ptr<ArrayData> array_ref = arr->data();
+ DLMTensorCtx* DLMTensor = new DLMTensorCtx;
+ DLMTensor->ref = array_ref;
+
+ // Define DLManagedTensor struct defined by
+ // DLPack (dlpack_structure.h)
+ DLManagedTensor* dlm_tensor = &DLMTensor->tensor;
+ dlm_tensor->manager_ctx = DLMTensor;
+ dlm_tensor->deleter = &deleter;
+
+ // Define the data pointer to the DLTensor
+ // If array is of length 0, data pointer should be NULL
Review Comment:
Hmm, that is weird. @kkraus14 Do you know the rationale for this? Allowing the data to be NULL for 0-sized arrays is one thing, but mandating that it is NULL seems silly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org