You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "amol- (via GitHub)" <gi...@apache.org> on 2024/10/16 21:58:32 UTC
[PR] [GH-44446] Add mask creation helper [arrow]
amol- opened a new pull request, #44447:
URL: https://github.com/apache/arrow/pull/44447
### Rationale for this change
Implement a convenience method to create boolean masks where only some rows are set to ``true``
### What changes are included in this PR?
C++ ``MakeMaskArray`` factory function and ``pyarrow.mask`` Python factory function
### Are these changes tested?
Yes both C++ and Python
### Are there any user-facing changes?
Yes, new API has been added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1809559720
##########
cpp/src/arrow/array/util.cc:
##########
@@ -915,6 +916,29 @@ Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
return builder->Finish();
}
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
Review Comment:
This `std::vector` could be a `span<int64_t>` for more flexibility. The selection vector could come from anywhere.
##########
python/pyarrow/array.pxi:
##########
@@ -538,6 +538,51 @@ def repeat(value, size, MemoryPool memory_pool=None):
return pyarrow_wrap_array(c_array)
+def mask(indices, length, MemoryPool memory_pool=None):
+ """
+ Create a boolean Array instance where specific indices are marked as True.
+
+ Parameters
+ ----------
+ indices : list[int]
Review Comment:
If the utility is templated on integer type, it could be used to implement kernels, but implementing the kernel involves a lot more code due to dispatching logic. So I would be in favor of having the utility as it is and leaving the kernel for later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1807812463
##########
python/pyarrow/array.pxi:
##########
@@ -538,6 +538,51 @@ def repeat(value, size, MemoryPool memory_pool=None):
return pyarrow_wrap_array(c_array)
+def mask(indices, length, MemoryPool memory_pool=None):
+ """
+ Create a boolean Array instance where specific indices are marked as True.
+
+ Parameters
+ ----------
+ indices : list[int]
Review Comment:
If I remember correctly compute functions are expected to work _only_ on arrow data (arrays), which would make less convenient this function as a factory as its purpose is mostly to create a mask from literals.
I think it would make sense to make it accept arrays too btw
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1804971066
##########
python/pyarrow/array.pxi:
##########
@@ -538,6 +538,51 @@ def repeat(value, size, MemoryPool memory_pool=None):
return pyarrow_wrap_array(c_array)
+def mask(indices, length, MemoryPool memory_pool=None):
Review Comment:
@jorisvandenbossche Does this API look ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-44446] Add mask creation helper [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #44447:
URL: https://github.com/apache/arrow/pull/44447#issuecomment-2418039522
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1875841135
##########
cpp/src/arrow/array/util.cc:
##########
@@ -915,6 +917,71 @@ Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
return builder->Finish();
}
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
+ int64_t length, MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
+ for (int64_t index : indices) {
+ if (index < 0 || index >= length) {
+ return Status::IndexError("Index out of bounds: ", index);
+ }
+ bit_util::SetBit(buffer->mutable_data(), index);
+ }
+ return std::make_shared<BooleanArray>(length, buffer);
+}
+
+template <typename IndexType>
+Result<std::shared_ptr<Array>> MakeMaskArrayImpl(
+ const std::shared_ptr<NumericArray<IndexType>>& indices, int64_t length,
+ MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
+ for (int64_t i = 0; i < indices->length(); ++i) {
+ int64_t index = indices->Value(i);
Review Comment:
The outer MakeMaskArray function already prevents values from being null, that's why it's not done in the Impl again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1804974475
##########
python/pyarrow/array.pxi:
##########
@@ -538,6 +538,51 @@ def repeat(value, size, MemoryPool memory_pool=None):
return pyarrow_wrap_array(c_array)
+def mask(indices, length, MemoryPool memory_pool=None):
+ """
+ Create a boolean Array instance where specific indices are marked as True.
+
+ Parameters
+ ----------
+ indices : list[int]
Review Comment:
It would be nice if `indices` could also be passed as a Arrow array... This means that perhaps the C++ mask function should be a compute kernel?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1804970343
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -869,6 +869,31 @@ TEST_F(TestArray, TestConcurrentFillFromScalar) {
}
}
+TEST_F(TestArray, TestMakeMaskArray) {
+ ASSERT_OK_AND_ASSIGN(auto array, MakeMaskArray({5, 8}, 10));
+ ASSERT_OK(array->ValidateFull());
+ ASSERT_EQ(array->length(), 10);
+
+ ASSERT_OK_AND_ASSIGN(auto true_scalar, MakeScalar(boolean(), true));
Review Comment:
You could instead create a comparison array using `ArrayFromJSON`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1875843784
##########
cpp/src/arrow/array/util.h:
##########
@@ -69,6 +69,33 @@ ARROW_EXPORT
Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
MemoryPool* pool = default_memory_pool());
+/// \brief Create an Array representing a boolean mask
+///
+/// The mask will have all elements set to false except for those
+/// indices specified in the indices vector.
+///
+/// \param[in] indices Which indices in the mask should be set to true
+/// \param[in] length The total length of the mask
+/// \param[in] pool the memory pool to allocate memory from
+/// \return the resulting Array
+ARROW_EXPORT
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
Review Comment:
I thought Arrow was constrained to C++17, isn't span a C++20 addition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1875954163
##########
cpp/src/arrow/array/util.cc:
##########
@@ -915,6 +917,71 @@ Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
return builder->Finish();
}
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
+ int64_t length, MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
+ for (int64_t index : indices) {
+ if (index < 0 || index >= length) {
+ return Status::IndexError("Index out of bounds: ", index);
+ }
+ bit_util::SetBit(buffer->mutable_data(), index);
+ }
+ return std::make_shared<BooleanArray>(length, buffer);
+}
+
+template <typename IndexType>
+Result<std::shared_ptr<Array>> MakeMaskArrayImpl(
+ const std::shared_ptr<NumericArray<IndexType>>& indices, int64_t length,
+ MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
+ for (int64_t i = 0; i < indices->length(); ++i) {
+ int64_t index = indices->Value(i);
+ if (index < 0 || index >= length) {
+ return Status::IndexError("Index out of bounds: ", index);
+ }
+ bit_util::SetBit(buffer->mutable_data(), index);
+ }
+ return std::make_shared<BooleanArray>(length, buffer);
+}
+
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::shared_ptr<Array>& indices,
+ int64_t length, MemoryPool* pool) {
+ if (indices->null_count() > 0) {
+ return Status::Invalid("Indices array must not contain null values");
+ }
Review Comment:
I don't see a reason why it would make sense to accept null values, it verifies if there are null values and rejects the array in case there are because there doesn't seem to be a case where it would make sense to have nulls in an array of indices.
If the user has an array containing nulls it can be used by passing it to `compute.drop_null` before using `MakeMaskArray`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: Add mask creation helper [arrow]
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on PR #44447:
URL: https://github.com/apache/arrow/pull/44447#issuecomment-2418084462
I'm a bit confused by the linter complaining about unrelated code
```
+++ fixed//arrow/python/pyarrow/lib.pyx
@@ -87,9 +87,9 @@
def is_threading_enabled() -> bool:
"""
- Returns True if threading is enabled in libarrow.
-
- If it isn't enabled, then python shouldn't create any
+ Returns True if threading is enabled in libarrow.
+
+ If it isn't enabled, then python shouldn't create any
threads either, because we're probably on a system where
threading doesn't work (e.g. Emscripten).
"""
@@ -175,7 +175,7 @@
if __cuda_loaded is not True:
raise ImportError(
"Trying to import data on a CUDA device, but PyArrow is not built with "
- f"CUDA support.\n(importing 'pyarrow.cuda' resulted in \"{
+ f"CUDA support.\n(importing 'pyarrow.cuda' resulted in "{
__cuda_loaded}\")."
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1804969824
##########
cpp/src/arrow/array/util.cc:
##########
@@ -915,6 +916,29 @@ Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
return builder->Finish();
}
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
+ int64_t length, MemoryPool* pool) {
+ BooleanBuilder builder(pool);
+ RETURN_NOT_OK(builder.Resize(length));
+
+ auto indices_end = indices.end();
+ auto i = indices.begin();
+ for (int64_t builder_i = 0; builder_i < length; builder_i++) {
Review Comment:
This make `length` append calls and also assumes that the indices are ordered.
Instead, how about start from an all-false bitmap and use `SetBit` calls to set the mask bits?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1873689442
##########
cpp/src/arrow/array/util.cc:
##########
@@ -915,6 +917,71 @@ Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
return builder->Finish();
}
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
+ int64_t length, MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
+ for (int64_t index : indices) {
+ if (index < 0 || index >= length) {
+ return Status::IndexError("Index out of bounds: ", index);
+ }
+ bit_util::SetBit(buffer->mutable_data(), index);
+ }
+ return std::make_shared<BooleanArray>(length, buffer);
+}
+
+template <typename IndexType>
+Result<std::shared_ptr<Array>> MakeMaskArrayImpl(
+ const std::shared_ptr<NumericArray<IndexType>>& indices, int64_t length,
+ MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
+ for (int64_t i = 0; i < indices->length(); ++i) {
+ int64_t index = indices->Value(i);
Review Comment:
The value could be null and nulls must be skipped.
##########
cpp/src/arrow/array/util.cc:
##########
@@ -915,6 +917,71 @@ Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
return builder->Finish();
}
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
+ int64_t length, MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
+ for (int64_t index : indices) {
+ if (index < 0 || index >= length) {
+ return Status::IndexError("Index out of bounds: ", index);
+ }
+ bit_util::SetBit(buffer->mutable_data(), index);
+ }
+ return std::make_shared<BooleanArray>(length, buffer);
+}
+
+template <typename IndexType>
+Result<std::shared_ptr<Array>> MakeMaskArrayImpl(
+ const std::shared_ptr<NumericArray<IndexType>>& indices, int64_t length,
+ MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
+ for (int64_t i = 0; i < indices->length(); ++i) {
+ int64_t index = indices->Value(i);
+ if (index < 0 || index >= length) {
+ return Status::IndexError("Index out of bounds: ", index);
+ }
+ bit_util::SetBit(buffer->mutable_data(), index);
+ }
+ return std::make_shared<BooleanArray>(length, buffer);
+}
+
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::shared_ptr<Array>& indices,
+ int64_t length, MemoryPool* pool) {
+ if (indices->null_count() > 0) {
+ return Status::Invalid("Indices array must not contain null values");
+ }
Review Comment:
Hmm. If it takes an Arrow array of indices it should be able to handle nulls. The loop can be specialized based on the result of `indices->MayHaveNulls()` so the common case doesn't have to check the validity bitmap of every iteration.
##########
cpp/src/arrow/array/util.cc:
##########
@@ -915,6 +917,71 @@ Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
return builder->Finish();
}
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
+ int64_t length, MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
Review Comment:
```suggestion
ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateEmptyBitmap(length, pool));
```
##########
cpp/src/arrow/array/util.h:
##########
@@ -69,6 +69,33 @@ ARROW_EXPORT
Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
MemoryPool* pool = default_memory_pool());
+/// \brief Create an Array representing a boolean mask
+///
+/// The mask will have all elements set to false except for those
+/// indices specified in the indices vector.
+///
+/// \param[in] indices Which indices in the mask should be set to true
+/// \param[in] length The total length of the mask
+/// \param[in] pool the memory pool to allocate memory from
+/// \return the resulting Array
+ARROW_EXPORT
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
Review Comment:
As this function doesn't need to take ownership of `indices`, it can accept a `span<int64_t>` instead of a `std::vector<int64_t> &` so both vectors or any contiguous buffer if int64_t can be passed.
##########
cpp/src/arrow/array/util.cc:
##########
@@ -915,6 +917,71 @@ Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
return builder->Finish();
}
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
+ int64_t length, MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
+ for (int64_t index : indices) {
+ if (index < 0 || index >= length) {
+ return Status::IndexError("Index out of bounds: ", index);
+ }
+ bit_util::SetBit(buffer->mutable_data(), index);
+ }
+ return std::make_shared<BooleanArray>(length, buffer);
+}
+
+template <typename IndexType>
+Result<std::shared_ptr<Array>> MakeMaskArrayImpl(
+ const std::shared_ptr<NumericArray<IndexType>>& indices, int64_t length,
+ MemoryPool* pool) {
+ ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBitmap(length, pool));
+ bit_util::SetBitsTo(buffer->mutable_data(), 0, length, false);
Review Comment:
```suggestion
ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateEmptyBitmap(length, pool));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1875849315
##########
cpp/src/arrow/array/util.h:
##########
@@ -69,6 +69,33 @@ ARROW_EXPORT
Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
MemoryPool* pool = default_memory_pool());
+/// \brief Create an Array representing a boolean mask
+///
+/// The mask will have all elements set to false except for those
+/// indices specified in the indices vector.
+///
+/// \param[in] indices Which indices in the mask should be set to true
+/// \param[in] length The total length of the mask
+/// \param[in] pool the memory pool to allocate memory from
+/// \return the resulting Array
+ARROW_EXPORT
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
Review Comment:
Yes, that's why we have a `util::span` backport.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: Add mask creation helper [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #44447:
URL: https://github.com/apache/arrow/pull/44447#issuecomment-2418060641
:warning: GitHub issue #44446 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1807817010
##########
python/pyarrow/array.pxi:
##########
@@ -538,6 +538,51 @@ def repeat(value, size, MemoryPool memory_pool=None):
return pyarrow_wrap_array(c_array)
+def mask(indices, length, MemoryPool memory_pool=None):
+ """
+ Create a boolean Array instance where specific indices are marked as True.
+
+ Parameters
+ ----------
+ indices : list[int]
Review Comment:
At the C++ level you're right, at the Python level we allow implicit conversion if it's non-ambiguous:
```python
>>> pc.list_flatten([[1, 2, 3], None, [4, None, 5]])
<pyarrow.lib.Int64Array object at 0x7f37d6454340>
[
1,
2,
3,
4,
null,
5
]
```
@felipecrv What do you think about this? Should it be a compute kernel or rather a general facility?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1876056726
##########
python/pyarrow/array.pxi:
##########
@@ -538,6 +538,51 @@ def repeat(value, size, MemoryPool memory_pool=None):
return pyarrow_wrap_array(c_array)
+def mask(indices, length, MemoryPool memory_pool=None):
Review Comment:
Late response, and the general API looks good, my one suggestion would be to use a bit more descriptive name. `mask` could also be interpret in the active sense ("mask those values"), and eg pandas has a method with that name that work in that sense. Some more explicit options: `create_mask`, `make_mask`, `mask_from_indices`, ..
This is essentially the counterpart for `indices_nonzero` I think? (which converts a mask into indices)
Could maybe mention that in a "See Also" section in the docstring
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on PR #44447:
URL: https://github.com/apache/arrow/pull/44447#issuecomment-2515436244
When you have the chance I'd appreciate if you could re-review this one @pitrou ๐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on PR #44447:
URL: https://github.com/apache/arrow/pull/44447#issuecomment-2419796913
@pitrou Do you have time to review? I think this is more or less done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #44447:
URL: https://github.com/apache/arrow/pull/44447#issuecomment-3901716598
Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1874474892
##########
cpp/src/arrow/array/util.h:
##########
@@ -69,6 +69,33 @@ ARROW_EXPORT
Result<std::shared_ptr<Array>> MakeEmptyArray(std::shared_ptr<DataType> type,
MemoryPool* pool = default_memory_pool());
+/// \brief Create an Array representing a boolean mask
+///
+/// The mask will have all elements set to false except for those
+/// indices specified in the indices vector.
+///
+/// \param[in] indices Which indices in the mask should be set to true
+/// \param[in] length The total length of the mask
+/// \param[in] pool the memory pool to allocate memory from
+/// \return the resulting Array
+ARROW_EXPORT
+Result<std::shared_ptr<Array>> MakeMaskArray(const std::vector<int64_t>& indices,
Review Comment:
Note that it would be a `span<const int64_t>` then :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1864680817
##########
python/pyarrow/array.pxi:
##########
@@ -538,6 +538,51 @@ def repeat(value, size, MemoryPool memory_pool=None):
return pyarrow_wrap_array(c_array)
+def mask(indices, length, MemoryPool memory_pool=None):
+ """
+ Create a boolean Array instance where specific indices are marked as True.
+
+ Parameters
+ ----------
+ indices : list[int]
Review Comment:
Added an overloaded version that accepts Array
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on code in PR #44447:
URL: https://github.com/apache/arrow/pull/44447#discussion_r1864680817
##########
python/pyarrow/array.pxi:
##########
@@ -538,6 +538,51 @@ def repeat(value, size, MemoryPool memory_pool=None):
return pyarrow_wrap_array(c_array)
+def mask(indices, length, MemoryPool memory_pool=None):
+ """
+ Create a boolean Array instance where specific indices are marked as True.
+
+ Parameters
+ ----------
+ indices : list[int]
Review Comment:
Added an overloaded version that accepts Array. The Python wrapper accepts everything supported by `asarray`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on PR #44447:
URL: https://github.com/apache/arrow/pull/44447#issuecomment-2513141273
I think the two failures are unrelated to this branch.
For Java it seems to be a memory leak in FlightSQL
```
Error: TestFlightSqlStreams.tearDown:238 ยป IllegalState Memory was leaked by query. Memory leaked: (81920)
```
And for R it seems related to datetimes
```
โโ Failure ('test-dplyr-funcs-datetime.R:302:5'): timestamp round trip correctly via strftime and strptime โโ
```
This pull request introduces a new feature for creating boolean mask arrays in the Arrow library and includes corresponding tests and documentation updates. The most important changes are summarized below:
### New Feature: Boolean Mask Arrays
* [`cpp/src/arrow/array/util.cc`](diffhunk://#diff-c3b0484ad8586ff46fa035d446a7d1c3a30cd35d13cd05678c99814938e07d5bR920-R984): Added `MakeMaskArray` functions to create boolean mask arrays from either a vector of indices or an Arrow array of indices. These functions set specified indices to true and all other indices to false.
* [`cpp/src/arrow/array/array_test.cc`](diffhunk://#diff-0d86cf45e6dbc2db7c716f28ac6f7c6a1d938f5dc91611a2394f8b250c178026R897-R917): Added a new test case `TestMakeMaskArray` to verify the correctness of the `MakeMaskArray` functions, including tests for out-of-bounds indices.
### Header and Implementation Updates
* [`cpp/src/arrow/array/util.h`](diffhunk://#diff-61040a2cf5bfc48c4e7edf5b64494ca173a471c6ca1e4e318270e99366015163R72-R98): Declared the `MakeMaskArray` functions with detailed documentation on their usage and parameters.
* [`cpp/src/arrow/array/util.cc`](diffhunk://#diff-c3b0484ad8586ff46fa035d446a7d1c3a30cd35d13cd05678c99814938e07d5bR32): Included necessary headers and using declarations for the new functions. [[1]](diffhunk://#diff-c3b0484ad8586ff46fa035d446a7d1c3a30cd35d13cd05678c99814938e07d5bR32) [[2]](diffhunk://#diff-c3b0484ad8586ff46fa035d446a7d1c3a30cd35d13cd05678c99814938e07d5bR55)
### Python Bindings and Tests
* [`python/pyarrow/array.pxi`](diffhunk://#diff-8511a20d3b6b9b098c1ac0dfc286e8127fdc60f96172adac4d281697527db8c2R541-R585): Added a `mask` function to create a boolean mask array in Python using the new `MakeMaskArray` functionality.
* [`python/pyarrow/tests/test_array.py`](diffhunk://#diff-626a9af939cc3db72d09d8db062e114297ac25b45f9057f864191a98c40e2d34R4231-R4239): Added a test case `test_mask_array` to verify the Python `mask` function.
### Documentation
* [`docs/source/python/api/arrays.rst`](diffhunk://#diff-47cf574ae5cbb4ec39df97e7472728aba21c60c8d0c39cfb4f80dcc7e05cf128R34): Updated the documentation to include the new `mask` function.
* [`python/pyarrow/__init__.py`](diffhunk://#diff-d4875eaed78803ab6f81979ac5cdc15bf0cc756d94fd12ef1a3d61bea2fe773bL196-R196): Added `mask` to the list of imported functions in the `pyarrow` module.
* [`python/pyarrow/includes/libarrow.pxd`](diffhunk://#diff-48e87fef366d65330100bf1703a542b8cb447575b6f8c3a3617741ce57854d4aR249-R254): Declared the `MakeMaskArray` functions in the Cython header file for Arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-44446: [C++][Python] Add mask creation helper [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #44447: GH-44446: [C++][Python] Add mask creation helper
URL: https://github.com/apache/arrow/pull/44447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org