You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/06/27 09:10:41 UTC
[GitHub] [arrow] ZMZ91 opened a new pull request #10606: add support for take implementation on dense union type
ZMZ91 opened a new pull request #10606:
URL: https://github.com/apache/arrow/pull/10606
https://issues.apache.org/jira/browse/ARROW-13005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10606: add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#issuecomment-869129636
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ZMZ91 commented on pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
ZMZ91 commented on pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#issuecomment-870416932
ARROW-13005: [C++] Implementation of take function for dense union data type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ZMZ91 commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
ZMZ91 commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r663586860
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,66 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ typename TypeTraits<DenseUnionType>::ValueOffsetBuilderType value_offset_builder;
+ typename TypeTraits<DenseUnionType>::ChildIdBuilderType child_id_builder;
+ std::vector<int8_t> type_codes;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_builder(ctx->memory_pool()),
+ child_id_builder(ctx->memory_pool()) {
+ DenseUnionArray typed_values(this->values);
+ type_codes = typed_values.union_type()->type_codes();
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ auto child_id = typed_values.child_id(index);
+ child_id_builder.UnsafeAppend(type_codes[child_id]);
+ auto value_offset = typed_values.value_offset(index);
+ value_offset_builder.UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ // TODO: not able to handle null case
+ VisitNoop));
+ return Status::OK();
+ }
+
+ Status Init() override {
+ RETURN_NOT_OK(child_id_builder.Reserve(output_length));
+ RETURN_NOT_OK(value_offset_builder.Reserve(output_length));
+ return Status::OK();
+ }
+
+ Status Finish() override {
+ std::shared_ptr<Array> child_ids;
+ std::shared_ptr<Array> value_offsets;
+ RETURN_NOT_OK(child_id_builder.Finish(&child_ids));
+ RETURN_NOT_OK(value_offset_builder.Finish(&value_offsets));
+
+ DenseUnionArray typed_values(this->values);
+ auto num_fields = typed_values.num_fields();
+ ArrayVector child_arrays;
+ child_arrays.reserve(num_fields);
+ BufferVector buffers = {nullptr, checked_cast<const Int8Array&>(*child_ids).values(),
+ checked_cast<const Int32Array&>(*value_offsets).values()};
+ *out = ArrayData(typed_values.type(), child_ids->length(), std::move(buffers), 0);
+ for (int i = 0; i < typed_values.num_fields(); i++) {
+ out->child_data.push_back(typed_values.field(i)->data());
Review comment:
But for the null case, it still needs to a null in the first child, right? And there may be no nulls in the first child.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#issuecomment-869609896
https://issues.apache.org/jira/browse/ARROW-13005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bkietz closed pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
bkietz closed pull request #10606:
URL: https://github.com/apache/arrow/pull/10606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ZMZ91 commented on pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
ZMZ91 commented on pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#issuecomment-876060454
Thank you @bkietz. All comments resolved. Please help check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ZMZ91 commented on pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
ZMZ91 commented on pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#issuecomment-870416932
ARROW-13005: [C++] Implementation of take function for dense union data type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bkietz commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
bkietz commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r665454642
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,82 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ Int32Builder value_offset_builder;
+ Int8Builder child_id_builder;
+ std::vector<int8_t> type_codes;
+ std::vector<std::shared_ptr<Int32Builder>> child_indices_builders;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_builder(ctx->memory_pool()),
+ child_id_builder(ctx->memory_pool()) {
+ DenseUnionArray typed_values(this->values);
+ type_codes = typed_values.union_type()->type_codes();
+ child_indices_builders.reserve(type_codes.size());
+ for (size_t i = 0; i < type_codes.size(); i++) {
+ child_indices_builders.push_back(
+ std::make_shared<Int32Builder>(ctx->memory_pool()));
+ }
Review comment:
```suggestion
value_offset_builder_(ctx->memory_pool()),
child_id_builder_(ctx->memory_pool()),
type_codes_(checked_cast<const UnionType&>(*this->values->type).type_codes()),
child_indices_builders_(type_codes_.size()) {
for (auto& child_indices_builder : child_indices_builders_) {
child_indices_builder = Int32Builder(ctx->memory_pool()));
}
```
class ArrayBuilder doesn't currently allow assignment like this but it should: please also add `ARROW_DEFAULT_MOVE_AND_ASSIGN(ArrayBuilder);` to class ArrayBuilder
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,82 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ Int32Builder value_offset_builder;
+ Int8Builder child_id_builder;
+ std::vector<int8_t> type_codes;
+ std::vector<std::shared_ptr<Int32Builder>> child_indices_builders;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_builder(ctx->memory_pool()),
+ child_id_builder(ctx->memory_pool()) {
+ DenseUnionArray typed_values(this->values);
+ type_codes = typed_values.union_type()->type_codes();
+ child_indices_builders.reserve(type_codes.size());
+ for (size_t i = 0; i < type_codes.size(); i++) {
+ child_indices_builders.push_back(
+ std::make_shared<Int32Builder>(ctx->memory_pool()));
+ }
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ int8_t child_id = typed_values.child_id(index);
+ child_id_builder.UnsafeAppend(type_codes[child_id]);
+ int32_t value_offset = typed_values.value_offset(index);
+ value_offset_builder.UnsafeAppend(child_indices_builders[child_id]->length());
+ RETURN_NOT_OK(child_indices_builders[child_id]->Reserve(1));
+ child_indices_builders[child_id]->UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ [&]() {
+ int8_t child_id = 0;
+ child_id_builder.UnsafeAppend(type_codes[child_id]);
+ value_offset_builder.UnsafeAppend(child_indices_builders[child_id]->length());
+ RETURN_NOT_OK(child_indices_builders[child_id]->Reserve(1));
+ child_indices_builders[child_id]->UnsafeAppendNull();
+ return Status::OK();
+ }));
+ return Status::OK();
+ }
+
+ Status Init() override {
+ RETURN_NOT_OK(child_id_builder.Reserve(output_length));
+ RETURN_NOT_OK(value_offset_builder.Reserve(output_length));
+ return Status::OK();
+ }
+
+ Status Finish() override {
+ std::shared_ptr<Array> child_ids;
+ std::shared_ptr<Array> value_offsets;
+ RETURN_NOT_OK(child_id_builder.Finish(&child_ids));
+ RETURN_NOT_OK(value_offset_builder.Finish(&value_offsets));
+
+ DenseUnionArray typed_values(this->values);
+ auto num_fields = typed_values.num_fields();
+ BufferVector buffers = {nullptr, checked_cast<const Int8Array&>(*child_ids).values(),
+ checked_cast<const Int32Array&>(*value_offsets).values()};
+ *out = ArrayData(typed_values.type(), child_ids->length(), std::move(buffers), 0);
+ for (auto i = 0; i < num_fields; i++) {
+ std::shared_ptr<Int32Array> child_indices_array;
+ RETURN_NOT_OK(child_indices_builders[i]->Finish(&child_indices_array));
Review comment:
Nit: prefer result returning overloads
```suggestion
ARROW_ASSIGN_OR_RAISE(auto child_indices_array,
child_indices_builders[i]->Finish());
```
##########
File path: cpp/src/arrow/compute/kernels/vector_selection_test.cc
##########
@@ -1281,33 +1280,32 @@ TEST_F(TestTakeKernelWithStruct, TakeStruct) {
class TestTakeKernelWithUnion : public TestTakeKernelTyped<UnionType> {};
-// TODO: Restore Union take functionality
-TEST_F(TestTakeKernelWithUnion, DISABLED_TakeUnion) {
- for (auto union_ : UnionTypeFactories()) {
- auto union_type = union_({field("a", int32()), field("b", utf8())}, {2, 5});
- auto union_json = R"([
+// TODO: Restore Sparse Union take functionality
Review comment:
Are you going to address this in this PR? If not, please file a follow up JIRA and remove this comment or include that JIRA's number here
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,82 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ Int32Builder value_offset_builder;
+ Int8Builder child_id_builder;
Review comment:
It's worth noting that ArrayBuilder always produces a null bitmap, which is not necessary for the buffers these are used to produce. See also: TypedBufferBuilder
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,82 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ Int32Builder value_offset_builder;
+ Int8Builder child_id_builder;
+ std::vector<int8_t> type_codes;
+ std::vector<std::shared_ptr<Int32Builder>> child_indices_builders;
Review comment:
Nit: this class is large enough to warrant naming members clearly
```suggestion
Int32Builder value_offset_builder_;
Int8Builder child_id_builder_;
std::vector<int8_t> type_codes_;
std::vector<Int32Builder> child_indices_builders_;
```
##########
File path: cpp/src/arrow/compute/kernels/vector_selection_test.cc
##########
@@ -1281,33 +1280,32 @@ TEST_F(TestTakeKernelWithStruct, TakeStruct) {
class TestTakeKernelWithUnion : public TestTakeKernelTyped<UnionType> {};
-// TODO: Restore Union take functionality
-TEST_F(TestTakeKernelWithUnion, DISABLED_TakeUnion) {
- for (auto union_ : UnionTypeFactories()) {
- auto union_type = union_({field("a", int32()), field("b", utf8())}, {2, 5});
- auto union_json = R"([
+// TODO: Restore Sparse Union take functionality
+TEST_F(TestTakeKernelWithUnion, TakeUnion) {
+ auto union_type = dense_union({field("a", int32()), field("b", utf8())}, {2, 5});
+ auto union_json = R"([
null,
[2, 222],
[5, "hello"],
[5, "eh"],
null,
[2, 111]
Review comment:
Since union doesn't support top level nulls, please be explicit about the type_id associated with a null and test with nulls in each child array
```suggestion
[2, null],
[2, 222],
[5, "hello"],
[5, "eh"],
[2, null],
[2, 111],
[5, null]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ZMZ91 commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
ZMZ91 commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r663461793
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,66 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ typename TypeTraits<DenseUnionType>::ValueOffsetBuilderType value_offset_builder;
+ typename TypeTraits<DenseUnionType>::ChildIdBuilderType child_id_builder;
+ std::vector<int8_t> type_codes;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_builder(ctx->memory_pool()),
+ child_id_builder(ctx->memory_pool()) {
+ DenseUnionArray typed_values(this->values);
+ type_codes = typed_values.union_type()->type_codes();
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ auto child_id = typed_values.child_id(index);
+ child_id_builder.UnsafeAppend(type_codes[child_id]);
+ auto value_offset = typed_values.value_offset(index);
+ value_offset_builder.UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ // TODO: not able to handle null case
+ VisitNoop));
+ return Status::OK();
+ }
+
+ Status Init() override {
+ RETURN_NOT_OK(child_id_builder.Reserve(output_length));
+ RETURN_NOT_OK(value_offset_builder.Reserve(output_length));
+ return Status::OK();
+ }
+
+ Status Finish() override {
+ std::shared_ptr<Array> child_ids;
+ std::shared_ptr<Array> value_offsets;
+ RETURN_NOT_OK(child_id_builder.Finish(&child_ids));
+ RETURN_NOT_OK(value_offset_builder.Finish(&value_offsets));
+
+ DenseUnionArray typed_values(this->values);
+ auto num_fields = typed_values.num_fields();
+ ArrayVector child_arrays;
+ child_arrays.reserve(num_fields);
+ BufferVector buffers = {nullptr, checked_cast<const Int8Array&>(*child_ids).values(),
+ checked_cast<const Int32Array&>(*value_offsets).values()};
+ *out = ArrayData(typed_values.type(), child_ids->length(), std::move(buffers), 0);
+ for (int i = 0; i < typed_values.num_fields(); i++) {
+ out->child_data.push_back(typed_values.field(i)->data());
Review comment:
Really appreciate for both your reviews. This is a quick implementation of take function on dense union data type. But as you see there are some problems.
@pitrou yes, this implementation will copy all child arrays and as you mentioned below, it's hard to handle null case. Because it doesn't maintain child builders and there may be no nulls in the first or any child array.
@bkietz if the indices from the argument not in asc order or the original dense union array not following the protocol, the value offsets for the generated array may not increase.
And I've thought about @pitrou 's solution that maintaining all child builders there. I agree that's a better solution but not very clear how to handle all different kinds of child builders. They could be any type, including complex ones like map, list or event union it self. And the implementation here sounds like a big switch case to me. Maybe some macro helps. Is there some example in existing code that handles various array builders could be referred to here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r663785134
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,66 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ typename TypeTraits<DenseUnionType>::ValueOffsetBuilderType value_offset_builder;
+ typename TypeTraits<DenseUnionType>::ChildIdBuilderType child_id_builder;
+ std::vector<int8_t> type_codes;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_builder(ctx->memory_pool()),
+ child_id_builder(ctx->memory_pool()) {
+ DenseUnionArray typed_values(this->values);
+ type_codes = typed_values.union_type()->type_codes();
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ auto child_id = typed_values.child_id(index);
+ child_id_builder.UnsafeAppend(type_codes[child_id]);
+ auto value_offset = typed_values.value_offset(index);
+ value_offset_builder.UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ // TODO: not able to handle null case
+ VisitNoop));
+ return Status::OK();
+ }
+
+ Status Init() override {
+ RETURN_NOT_OK(child_id_builder.Reserve(output_length));
+ RETURN_NOT_OK(value_offset_builder.Reserve(output_length));
+ return Status::OK();
+ }
+
+ Status Finish() override {
+ std::shared_ptr<Array> child_ids;
+ std::shared_ptr<Array> value_offsets;
+ RETURN_NOT_OK(child_id_builder.Finish(&child_ids));
+ RETURN_NOT_OK(value_offset_builder.Finish(&value_offsets));
+
+ DenseUnionArray typed_values(this->values);
+ auto num_fields = typed_values.num_fields();
+ ArrayVector child_arrays;
+ child_arrays.reserve(num_fields);
+ BufferVector buffers = {nullptr, checked_cast<const Int8Array&>(*child_ids).values(),
+ checked_cast<const Int32Array&>(*value_offsets).values()};
+ *out = ArrayData(typed_values.type(), child_ids->length(), std::move(buffers), 0);
+ for (int i = 0; i < typed_values.num_fields(); i++) {
+ out->child_data.push_back(typed_values.field(i)->data());
Review comment:
You can put a null in the indices.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ZMZ91 commented on pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
ZMZ91 commented on pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#issuecomment-875660411
Hi @pitrou and @bkietz, just update the pr due to your comments. Please help check the latest code. Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ZMZ91 commented on pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
ZMZ91 commented on pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#issuecomment-876854392
Hi @bkietz and @pitrou, just a question not quite related to this pr and about dense union array. I've seen the limitation for array data is under 2GB. Does it apply to nested arrays or just the sub arrays inside? For example, dense union array, it has child id array, value offset array and a bunch of child arrays. Each of them should be under 2GB. And is it restricted that the sum under 2GB?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r660750124
##########
File path: cpp/src/arrow/compute/kernels/vector_selection_test.cc
##########
@@ -1279,6 +1279,58 @@ TEST_F(TestTakeKernelWithStruct, TakeStruct) {
struct_type, R"([{"a": 1}, {"a": 2, "b": "hello"}])", "[0, 1, 0]");
}
+class TestTakeKernelWithDenseUnion : public TestTakeKernelTyped<DenseUnionType> {
+ public:
+ void AssertTake(const std::shared_ptr<DataType>& type, const std::string& values,
+ const std::string& indices, const std::string& type_codes,
+ const std::string& value_offsets) {
+ auto union_array =
+ std::static_pointer_cast<DenseUnionArray>(ArrayFromJSON(type, values));
+ ArrayVector children;
+ children.reserve(type->num_fields());
+ std::vector<std::string> field_names;
+ field_names.reserve(type->num_fields());
+ for (int i = 0; i < type->num_fields(); i++) {
+ children.push_back(union_array->field(i));
+ field_names.push_back(type->field(i)->name());
+ }
+ ASSIGN_OR_ABORT(
+ std::shared_ptr<Array> expected,
+ DenseUnionArray::Make(*ArrayFromJSON(int8(), type_codes),
+ *ArrayFromJSON(int32(), value_offsets), children,
+ field_names, union_array->union_type()->type_codes()));
+
+ std::shared_ptr<Array> actual;
+
+ for (auto index_type : {int8(), uint32()}) {
+ ASSERT_OK(TakeJSON(type, values, index_type, indices, &actual));
+ ASSERT_OK(actual->Validate());
+ AssertArraysEqual(*expected, *actual, /*verbose=*/true);
+ }
+ }
+};
+
+TEST_F(TestTakeKernelWithDenseUnion, TakeDenseUnion) {
+ auto union_type = dense_union({field("a", int32()), field("b", utf8())}, {2, 5});
Review comment:
Not sure why you're not reusing the test code from `DISABLED_TakeUnion` below? The example results seem different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r663785134
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,66 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ typename TypeTraits<DenseUnionType>::ValueOffsetBuilderType value_offset_builder;
+ typename TypeTraits<DenseUnionType>::ChildIdBuilderType child_id_builder;
+ std::vector<int8_t> type_codes;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_builder(ctx->memory_pool()),
+ child_id_builder(ctx->memory_pool()) {
+ DenseUnionArray typed_values(this->values);
+ type_codes = typed_values.union_type()->type_codes();
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ auto child_id = typed_values.child_id(index);
+ child_id_builder.UnsafeAppend(type_codes[child_id]);
+ auto value_offset = typed_values.value_offset(index);
+ value_offset_builder.UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ // TODO: not able to handle null case
+ VisitNoop));
+ return Status::OK();
+ }
+
+ Status Init() override {
+ RETURN_NOT_OK(child_id_builder.Reserve(output_length));
+ RETURN_NOT_OK(value_offset_builder.Reserve(output_length));
+ return Status::OK();
+ }
+
+ Status Finish() override {
+ std::shared_ptr<Array> child_ids;
+ std::shared_ptr<Array> value_offsets;
+ RETURN_NOT_OK(child_id_builder.Finish(&child_ids));
+ RETURN_NOT_OK(value_offset_builder.Finish(&value_offsets));
+
+ DenseUnionArray typed_values(this->values);
+ auto num_fields = typed_values.num_fields();
+ ArrayVector child_arrays;
+ child_arrays.reserve(num_fields);
+ BufferVector buffers = {nullptr, checked_cast<const Int8Array&>(*child_ids).values(),
+ checked_cast<const Int32Array&>(*value_offsets).values()};
+ *out = ArrayData(typed_values.type(), child_ids->length(), std::move(buffers), 0);
+ for (int i = 0; i < typed_values.num_fields(); i++) {
+ out->child_data.push_back(typed_values.field(i)->data());
Review comment:
You can put a null in the indices.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bkietz commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
bkietz commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r666344417
##########
File path: cpp/src/arrow/compute/kernels/vector_selection_test.cc
##########
@@ -607,31 +607,29 @@ TEST_F(TestFilterKernelWithStruct, FilterStruct) {
class TestFilterKernelWithUnion : public TestFilterKernel<UnionType> {};
-TEST_F(TestFilterKernelWithUnion, DISABLED_FilterUnion) {
- for (auto union_ : UnionTypeFactories()) {
- auto union_type = union_({field("a", int32()), field("b", utf8())}, {2, 5});
- auto union_json = R"([
- null,
+TEST_F(TestFilterKernelWithUnion, FilterUnion) {
+ auto union_type = dense_union({field("a", int32()), field("b", utf8())}, {2, 5});
+ auto union_json = R"([
+ [2, null],
[2, 222],
[5, "hello"],
[5, "eh"],
- null,
+ [2, null],
[2, 111]
Review comment:
To test that nulls in other than the first field work, please add:
```suggestion
[2, 111],
[5, null]
```
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,80 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ TypedBufferBuilder<int32_t> value_offset_buffer_builder_;
+ TypedBufferBuilder<int8_t> child_id_buffer_builder_;
+ std::vector<int8_t> type_codes_;
+ std::vector<Int32Builder> child_indices_builders_;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_buffer_builder_(ctx->memory_pool()),
+ child_id_buffer_builder_(ctx->memory_pool()),
+ type_codes_(checked_cast<const UnionType&>(*this->values->type).type_codes()),
+ child_indices_builders_(type_codes_.size()) {
+ for (auto& child_indices_builder : child_indices_builders_) {
+ child_indices_builder = Int32Builder(ctx->memory_pool());
+ }
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ int8_t child_id = typed_values.child_id(index);
+ child_id_buffer_builder_.UnsafeAppend(type_codes_[child_id]);
+ int32_t value_offset = typed_values.value_offset(index);
+ value_offset_buffer_builder_.UnsafeAppend(
+ child_indices_builders_[child_id].length());
+ RETURN_NOT_OK(child_indices_builders_[child_id].Reserve(1));
+ child_indices_builders_[child_id].UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ [&]() {
+ int8_t child_id = 0;
+ child_id_buffer_builder_.UnsafeAppend(type_codes_[child_id]);
+ value_offset_buffer_builder_.UnsafeAppend(
+ child_indices_builders_[child_id].length());
+ RETURN_NOT_OK(child_indices_builders_[child_id].Reserve(1));
+ child_indices_builders_[child_id].UnsafeAppendNull();
+ return Status::OK();
+ }));
+ return Status::OK();
+ }
+
+ Status Init() override {
+ RETURN_NOT_OK(child_id_buffer_builder_.Reserve(output_length));
+ RETURN_NOT_OK(value_offset_buffer_builder_.Reserve(output_length));
+ return Status::OK();
+ }
+
+ Status Finish() override {
+ ARROW_ASSIGN_OR_RAISE(auto child_ids_buffer, child_id_buffer_builder_.Finish());
+ ARROW_ASSIGN_OR_RAISE(auto value_offsets_buffer,
+ value_offset_buffer_builder_.Finish());
+ DenseUnionArray typed_values(this->values);
+ auto num_fields = typed_values.num_fields();
+ BufferVector buffers = {nullptr, child_ids_buffer, value_offsets_buffer};
Review comment:
```suggestion
BufferVector buffers{nullptr, std::move(child_ids_buffer), std::move(value_offsets_buffer)};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ZMZ91 commented on pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
ZMZ91 commented on pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#issuecomment-879501530
Hi @bkietz and @pitrou, could you help check what else to update in this pr? Or we may merge it? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#issuecomment-870715790
@ZMZ91 Thanks for tackling this slightly delicate issue. I've posted two comments about the chosen approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r663466643
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,66 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ typename TypeTraits<DenseUnionType>::ValueOffsetBuilderType value_offset_builder;
+ typename TypeTraits<DenseUnionType>::ChildIdBuilderType child_id_builder;
+ std::vector<int8_t> type_codes;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_builder(ctx->memory_pool()),
+ child_id_builder(ctx->memory_pool()) {
+ DenseUnionArray typed_values(this->values);
+ type_codes = typed_values.union_type()->type_codes();
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ auto child_id = typed_values.child_id(index);
+ child_id_builder.UnsafeAppend(type_codes[child_id]);
+ auto value_offset = typed_values.value_offset(index);
+ value_offset_builder.UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ // TODO: not able to handle null case
+ VisitNoop));
+ return Status::OK();
+ }
+
+ Status Init() override {
+ RETURN_NOT_OK(child_id_builder.Reserve(output_length));
+ RETURN_NOT_OK(value_offset_builder.Reserve(output_length));
+ return Status::OK();
+ }
+
+ Status Finish() override {
+ std::shared_ptr<Array> child_ids;
+ std::shared_ptr<Array> value_offsets;
+ RETURN_NOT_OK(child_id_builder.Finish(&child_ids));
+ RETURN_NOT_OK(value_offset_builder.Finish(&value_offsets));
+
+ DenseUnionArray typed_values(this->values);
+ auto num_fields = typed_values.num_fields();
+ ArrayVector child_arrays;
+ child_arrays.reserve(num_fields);
+ BufferVector buffers = {nullptr, checked_cast<const Int8Array&>(*child_ids).values(),
+ checked_cast<const Int32Array&>(*value_offsets).values()};
+ *out = ArrayData(typed_values.type(), child_ids->length(), std::move(buffers), 0);
+ for (int i = 0; i < typed_values.num_fields(); i++) {
+ out->child_data.push_back(typed_values.field(i)->data());
Review comment:
I don't think you need to use any child builders. Instead, you should be able to call `Take` on each of the child arrays; you just have to construct the indices carefully for each child array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bkietz commented on pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
bkietz commented on pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#issuecomment-877229283
The 2GB limitation applies to buffers and is type specific. It derives from usage of signed 32 bit integers to encode offsets, so for example a StringArray cannot have more than 2GB total character data because otherwise we could not express the final offset. IIUC it's allowed for a child of the union to be an Int8Array with `2**33` elements but a dense union's offsets could not point to most of those elements.
In short, this is not a constraint you need to enforce in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r660748979
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,66 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ typename TypeTraits<DenseUnionType>::ValueOffsetBuilderType value_offset_builder;
+ typename TypeTraits<DenseUnionType>::ChildIdBuilderType child_id_builder;
+ std::vector<int8_t> type_codes;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_builder(ctx->memory_pool()),
+ child_id_builder(ctx->memory_pool()) {
+ DenseUnionArray typed_values(this->values);
+ type_codes = typed_values.union_type()->type_codes();
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ auto child_id = typed_values.child_id(index);
+ child_id_builder.UnsafeAppend(type_codes[child_id]);
+ auto value_offset = typed_values.value_offset(index);
+ value_offset_builder.UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ // TODO: not able to handle null case
+ VisitNoop));
+ return Status::OK();
+ }
+
+ Status Init() override {
+ RETURN_NOT_OK(child_id_builder.Reserve(output_length));
+ RETURN_NOT_OK(value_offset_builder.Reserve(output_length));
+ return Status::OK();
+ }
+
+ Status Finish() override {
+ std::shared_ptr<Array> child_ids;
+ std::shared_ptr<Array> value_offsets;
+ RETURN_NOT_OK(child_id_builder.Finish(&child_ids));
+ RETURN_NOT_OK(value_offset_builder.Finish(&value_offsets));
+
+ DenseUnionArray typed_values(this->values);
+ auto num_fields = typed_values.num_fields();
+ ArrayVector child_arrays;
+ child_arrays.reserve(num_fields);
+ BufferVector buffers = {nullptr, checked_cast<const Int8Array&>(*child_ids).values(),
+ checked_cast<const Int32Array&>(*value_offsets).values()};
+ *out = ArrayData(typed_values.type(), child_ids->length(), std::move(buffers), 0);
+ for (int i = 0; i < typed_values.num_fields(); i++) {
+ out->child_data.push_back(typed_values.field(i)->data());
Review comment:
Hmm, if I understand correctly, you're reusing the full child arrays even if only a small fraction of them was taken? If so, this seems wasteful.
I think this can be done otherwise: maintain maps of the original offsets to the taken offsets, per child. This will admittedly use a lot of temporary storage (perhaps this can be done one child at a time?), but at least the final union array will be reasonably compact.
@bkietz What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r660749699
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,66 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ typename TypeTraits<DenseUnionType>::ValueOffsetBuilderType value_offset_builder;
+ typename TypeTraits<DenseUnionType>::ChildIdBuilderType child_id_builder;
+ std::vector<int8_t> type_codes;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_builder(ctx->memory_pool()),
+ child_id_builder(ctx->memory_pool()) {
+ DenseUnionArray typed_values(this->values);
+ type_codes = typed_values.union_type()->type_codes();
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ auto child_id = typed_values.child_id(index);
+ child_id_builder.UnsafeAppend(type_codes[child_id]);
+ auto value_offset = typed_values.value_offset(index);
+ value_offset_builder.UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ // TODO: not able to handle null case
+ VisitNoop));
Review comment:
Well, either a null in the indices should raise `Status::NotImplemented`, or it should append a null in one of the child arrays (any of them, for example the first one by convention).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bkietz commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
bkietz commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r666993306
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,81 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ TypedBufferBuilder<int32_t> value_offset_buffer_builder_;
+ TypedBufferBuilder<int8_t> child_id_buffer_builder_;
+ std::vector<int8_t> type_codes_;
+ std::vector<Int32Builder> child_indices_builders_;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_buffer_builder_(ctx->memory_pool()),
+ child_id_buffer_builder_(ctx->memory_pool()),
+ type_codes_(checked_cast<const UnionType&>(*this->values->type).type_codes()),
+ child_indices_builders_(type_codes_.size()) {
+ for (auto& child_indices_builder : child_indices_builders_) {
+ child_indices_builder = Int32Builder(ctx->memory_pool());
+ }
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ int8_t child_id = typed_values.child_id(index);
+ child_id_buffer_builder_.UnsafeAppend(type_codes_[child_id]);
+ int32_t value_offset = typed_values.value_offset(index);
+ value_offset_buffer_builder_.UnsafeAppend(
+ child_indices_builders_[child_id].length());
Review comment:
```suggestion
static_cast<int32_t>(child_indices_builders_[child_id].length()));
```
This is implicit conversion from `int64_t -> int32_t`. Unfortunately it needs to be explicit: https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/39924781/job/kkm24qqycxd29y5f#L1096
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,81 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ TypedBufferBuilder<int32_t> value_offset_buffer_builder_;
+ TypedBufferBuilder<int8_t> child_id_buffer_builder_;
+ std::vector<int8_t> type_codes_;
+ std::vector<Int32Builder> child_indices_builders_;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_buffer_builder_(ctx->memory_pool()),
+ child_id_buffer_builder_(ctx->memory_pool()),
+ type_codes_(checked_cast<const UnionType&>(*this->values->type).type_codes()),
+ child_indices_builders_(type_codes_.size()) {
+ for (auto& child_indices_builder : child_indices_builders_) {
+ child_indices_builder = Int32Builder(ctx->memory_pool());
+ }
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ int8_t child_id = typed_values.child_id(index);
+ child_id_buffer_builder_.UnsafeAppend(type_codes_[child_id]);
+ int32_t value_offset = typed_values.value_offset(index);
+ value_offset_buffer_builder_.UnsafeAppend(
+ child_indices_builders_[child_id].length());
+ RETURN_NOT_OK(child_indices_builders_[child_id].Reserve(1));
+ child_indices_builders_[child_id].UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ [&]() {
+ int8_t child_id = 0;
+ child_id_buffer_builder_.UnsafeAppend(type_codes_[child_id]);
+ value_offset_buffer_builder_.UnsafeAppend(
+ child_indices_builders_[child_id].length());
Review comment:
```suggestion
value_offset_buffer_builder_.UnsafeAppend(
static_cast<int32_t>(child_indices_builders_[child_id].length()));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] bkietz commented on a change in pull request #10606: ARROW-13005: [C++] Add support for take implementation on dense union type
Posted by GitBox <gi...@apache.org>.
bkietz commented on a change in pull request #10606:
URL: https://github.com/apache/arrow/pull/10606#discussion_r660855712
##########
File path: cpp/src/arrow/type_traits.h
##########
@@ -435,6 +435,8 @@ struct TypeTraits<DenseUnionType> {
using ArrayType = DenseUnionArray;
using BuilderType = DenseUnionBuilder;
using ScalarType = DenseUnionScalar;
+ using ValueOffsetBuilderType = Int32Builder;
+ using ChildIdBuilderType = Int8Builder;
Review comment:
These are not generally useful traits since they're not shared by other types. Please revert them
##########
File path: cpp/src/arrow/compute/kernels/vector_selection_test.cc
##########
@@ -1279,6 +1279,58 @@ TEST_F(TestTakeKernelWithStruct, TakeStruct) {
struct_type, R"([{"a": 1}, {"a": 2, "b": "hello"}])", "[0, 1, 0]");
}
+class TestTakeKernelWithDenseUnion : public TestTakeKernelTyped<DenseUnionType> {
+ public:
+ void AssertTake(const std::shared_ptr<DataType>& type, const std::string& values,
+ const std::string& indices, const std::string& type_codes,
+ const std::string& value_offsets) {
+ auto union_array =
+ std::static_pointer_cast<DenseUnionArray>(ArrayFromJSON(type, values));
+ ArrayVector children;
+ children.reserve(type->num_fields());
+ std::vector<std::string> field_names;
+ field_names.reserve(type->num_fields());
+ for (int i = 0; i < type->num_fields(); i++) {
+ children.push_back(union_array->field(i));
+ field_names.push_back(type->field(i)->name());
+ }
+ ASSIGN_OR_ABORT(
+ std::shared_ptr<Array> expected,
+ DenseUnionArray::Make(*ArrayFromJSON(int8(), type_codes),
+ *ArrayFromJSON(int32(), value_offsets), children,
+ field_names, union_array->union_type()->type_codes()));
+
Review comment:
```suggestion
ASSERT_OK(expected->ValidateFull());
```
##########
File path: cpp/src/arrow/compute/kernels/vector_selection_test.cc
##########
@@ -1279,6 +1279,58 @@ TEST_F(TestTakeKernelWithStruct, TakeStruct) {
struct_type, R"([{"a": 1}, {"a": 2, "b": "hello"}])", "[0, 1, 0]");
}
+class TestTakeKernelWithDenseUnion : public TestTakeKernelTyped<DenseUnionType> {
+ public:
+ void AssertTake(const std::shared_ptr<DataType>& type, const std::string& values,
+ const std::string& indices, const std::string& type_codes,
+ const std::string& value_offsets) {
+ auto union_array =
+ std::static_pointer_cast<DenseUnionArray>(ArrayFromJSON(type, values));
+ ArrayVector children;
+ children.reserve(type->num_fields());
+ std::vector<std::string> field_names;
+ field_names.reserve(type->num_fields());
+ for (int i = 0; i < type->num_fields(); i++) {
+ children.push_back(union_array->field(i));
+ field_names.push_back(type->field(i)->name());
+ }
+ ASSIGN_OR_ABORT(
Review comment:
```suggestion
ASSERT_OK_AND_ASSIGN(
```
##########
File path: cpp/src/arrow/compute/kernels/vector_selection.cc
##########
@@ -1668,6 +1668,66 @@ struct ListImpl : public Selection<ListImpl<Type>, Type> {
}
};
+struct DenseUnionImpl : public Selection<DenseUnionImpl, DenseUnionType> {
+ using Base = Selection<DenseUnionImpl, DenseUnionType>;
+ LIFT_BASE_MEMBERS();
+
+ typename TypeTraits<DenseUnionType>::ValueOffsetBuilderType value_offset_builder;
+ typename TypeTraits<DenseUnionType>::ChildIdBuilderType child_id_builder;
+ std::vector<int8_t> type_codes;
+
+ DenseUnionImpl(KernelContext* ctx, const ExecBatch& batch, int64_t output_length,
+ Datum* out)
+ : Base(ctx, batch, output_length, out),
+ value_offset_builder(ctx->memory_pool()),
+ child_id_builder(ctx->memory_pool()) {
+ DenseUnionArray typed_values(this->values);
+ type_codes = typed_values.union_type()->type_codes();
+ }
+
+ template <typename Adapter>
+ Status GenerateOutput() {
+ DenseUnionArray typed_values(this->values);
+ Adapter adapter(this);
+ RETURN_NOT_OK(adapter.Generate(
+ [&](int64_t index) {
+ auto child_id = typed_values.child_id(index);
+ child_id_builder.UnsafeAppend(type_codes[child_id]);
+ auto value_offset = typed_values.value_offset(index);
+ value_offset_builder.UnsafeAppend(value_offset);
+ return Status::OK();
+ },
+ // TODO: not able to handle null case
+ VisitNoop));
+ return Status::OK();
+ }
+
+ Status Init() override {
+ RETURN_NOT_OK(child_id_builder.Reserve(output_length));
+ RETURN_NOT_OK(value_offset_builder.Reserve(output_length));
+ return Status::OK();
+ }
+
+ Status Finish() override {
+ std::shared_ptr<Array> child_ids;
+ std::shared_ptr<Array> value_offsets;
+ RETURN_NOT_OK(child_id_builder.Finish(&child_ids));
+ RETURN_NOT_OK(value_offset_builder.Finish(&value_offsets));
+
+ DenseUnionArray typed_values(this->values);
+ auto num_fields = typed_values.num_fields();
+ ArrayVector child_arrays;
+ child_arrays.reserve(num_fields);
+ BufferVector buffers = {nullptr, checked_cast<const Int8Array&>(*child_ids).values(),
+ checked_cast<const Int32Array&>(*value_offsets).values()};
+ *out = ArrayData(typed_values.type(), child_ids->length(), std::move(buffers), 0);
+ for (int i = 0; i < typed_values.num_fields(); i++) {
+ out->child_data.push_back(typed_values.field(i)->data());
Review comment:
I think the larger problem here is that we may produce a dense union array whose offsets are not increasing, which is a violation of [the protocol](https://github.com/bkietz/arrow/blob/3c994458183a4585063a4925bad7bf02f29ab93c/docs/source/format/Columnar.rst#L561)
##########
File path: cpp/src/arrow/compute/kernels/vector_selection_test.cc
##########
@@ -1279,6 +1279,58 @@ TEST_F(TestTakeKernelWithStruct, TakeStruct) {
struct_type, R"([{"a": 1}, {"a": 2, "b": "hello"}])", "[0, 1, 0]");
}
+class TestTakeKernelWithDenseUnion : public TestTakeKernelTyped<DenseUnionType> {
+ public:
+ void AssertTake(const std::shared_ptr<DataType>& type, const std::string& values,
+ const std::string& indices, const std::string& type_codes,
+ const std::string& value_offsets) {
+ auto union_array =
+ std::static_pointer_cast<DenseUnionArray>(ArrayFromJSON(type, values));
+ ArrayVector children;
+ children.reserve(type->num_fields());
+ std::vector<std::string> field_names;
+ field_names.reserve(type->num_fields());
+ for (int i = 0; i < type->num_fields(); i++) {
+ children.push_back(union_array->field(i));
+ field_names.push_back(type->field(i)->name());
+ }
+ ASSIGN_OR_ABORT(
+ std::shared_ptr<Array> expected,
+ DenseUnionArray::Make(*ArrayFromJSON(int8(), type_codes),
+ *ArrayFromJSON(int32(), value_offsets), children,
+ field_names, union_array->union_type()->type_codes()));
+
+ std::shared_ptr<Array> actual;
+
+ for (auto index_type : {int8(), uint32()}) {
+ ASSERT_OK(TakeJSON(type, values, index_type, indices, &actual));
+ ASSERT_OK(actual->Validate());
Review comment:
```suggestion
ASSERT_OK(actual->ValidateFull());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org