You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "felipecrv (via GitHub)" <gi...@apache.org> on 2024/04/13 18:36:36 UTC
[PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
felipecrv opened a new pull request, #41189:
URL: https://github.com/apache/arrow/pull/41189
### Rationale for this change
Old behavior is not required by the Arrow spec and is wasteful.
### What changes are included in this PR?
Just the change in how `FixedSizeListBuilder` deals with nulls being appends.
### Are these changes tested?
By existing tests.
### Are there any user-facing changes?
`FixedSizeListBuilder::AppendNull()` and `AppendNulls()` don't append nulls to the underlying value builder and call `AppendEmptyValues(fixed_list_size)` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41189:
URL: https://github.com/apache/arrow/pull/41189#discussion_r1565888338
##########
cpp/src/arrow/array/builder_nested.cc:
##########
@@ -213,13 +213,13 @@ Status FixedSizeListBuilder::AppendValues(int64_t length, const uint8_t* valid_b
Status FixedSizeListBuilder::AppendNull() {
RETURN_NOT_OK(Reserve(1));
UnsafeAppendToBitmap(false);
- return value_builder_->AppendNulls(list_size_);
+ return value_builder_->AppendEmptyValues(list_size_);
Review Comment:
So both it ok and this might help performance for deep nested thing like `FixLengthArray<struct<str, str, str>, ...>`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #41189:
URL: https://github.com/apache/arrow/pull/41189#discussion_r1565883792
##########
cpp/src/arrow/array/builder_nested.cc:
##########
@@ -213,13 +213,13 @@ Status FixedSizeListBuilder::AppendValues(int64_t length, const uint8_t* valid_b
Status FixedSizeListBuilder::AppendNull() {
RETURN_NOT_OK(Reserve(1));
UnsafeAppendToBitmap(false);
- return value_builder_->AppendNulls(list_size_);
+ return value_builder_->AppendEmptyValues(list_size_);
Review Comment:
More details in the issue https://github.com/apache/arrow/issues/41188
Summary:
When you append nulls you force the creation of a null bitmap for the entire array.
`[[1, 2], null, [3, 4]]` currently needs a `[1, 0 1]` validity map at the top-level and a `[1, 1, 0, 0, 1, 1]` map in the child values array when it could have all values as valid and an empty bitmap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on PR #41189:
URL: https://github.com/apache/arrow/pull/41189#issuecomment-3547434674
Thank you for your contribution. Unfortunately, this
pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label
or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you
do not have repository permissions to reopen the PR, please tag a maintainer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on PR #41189:
URL: https://github.com/apache/arrow/pull/41189#issuecomment-2056950761
> Oops, it looks like the Python CI failures are related @felipecrv .
They are. I will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #41189:
URL: https://github.com/apache/arrow/pull/41189#discussion_r1565885932
##########
cpp/src/arrow/array/builder_nested.cc:
##########
@@ -213,13 +213,13 @@ Status FixedSizeListBuilder::AppendValues(int64_t length, const uint8_t* valid_b
Status FixedSizeListBuilder::AppendNull() {
RETURN_NOT_OK(Reserve(1));
UnsafeAppendToBitmap(false);
- return value_builder_->AppendNulls(list_size_);
+ return value_builder_->AppendEmptyValues(list_size_);
Review Comment:
But when you have an array like `[[1, null], null, [3, 4]]`, then you cannot run away from having a validity bitmap for the child values array —— `[1, 0, , 1, 1, 1, 1]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #41189:
URL: https://github.com/apache/arrow/pull/41189#issuecomment-2053727318
So the child validity would be undefined here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #41189:
URL: https://github.com/apache/arrow/pull/41189#discussion_r1566441618
##########
cpp/src/arrow/array/builder_nested.cc:
##########
@@ -213,13 +213,13 @@ Status FixedSizeListBuilder::AppendValues(int64_t length, const uint8_t* valid_b
Status FixedSizeListBuilder::AppendNull() {
RETURN_NOT_OK(Reserve(1));
UnsafeAppendToBitmap(false);
- return value_builder_->AppendNulls(list_size_);
+ return value_builder_->AppendEmptyValues(list_size_);
Review Comment:
It reduces the number of scattered bit reads and writes when dealing with fixed-size-lists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41189:
URL: https://github.com/apache/arrow/pull/41189#issuecomment-2056935963
Oops, it looks like the Python CI failures are related @felipecrv .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41189:
URL: https://github.com/apache/arrow/pull/41189#discussion_r1565873027
##########
cpp/src/arrow/array/builder_nested.cc:
##########
@@ -213,13 +213,13 @@ Status FixedSizeListBuilder::AppendValues(int64_t length, const uint8_t* valid_b
Status FixedSizeListBuilder::AppendNull() {
RETURN_NOT_OK(Reserve(1));
UnsafeAppendToBitmap(false);
- return value_builder_->AppendNulls(list_size_);
+ return value_builder_->AppendEmptyValues(list_size_);
Review Comment:
So, when child is non-nullable, would it matter previously?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #41189:
URL: https://github.com/apache/arrow/pull/41189#issuecomment-2058995220
+1!
I wonder would this approach have a threshold of null percentage above which it would make sense to keep current approach (assuming everything is nullable)? (I didn't look into `AppendEmptyValues` so this question might not even make sense)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41189:
URL: https://github.com/apache/arrow/pull/41189#issuecomment-2053727088
:warning: GitHub issue #41188 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #41189:
URL: https://github.com/apache/arrow/pull/41189#discussion_r1565885932
##########
cpp/src/arrow/array/builder_nested.cc:
##########
@@ -213,13 +213,13 @@ Status FixedSizeListBuilder::AppendValues(int64_t length, const uint8_t* valid_b
Status FixedSizeListBuilder::AppendNull() {
RETURN_NOT_OK(Reserve(1));
UnsafeAppendToBitmap(false);
- return value_builder_->AppendNulls(list_size_);
+ return value_builder_->AppendEmptyValues(list_size_);
Review Comment:
But when you have an array like `[[1, null], null, [3, 4]]`, then you cannot run away from having a validity bitmap for the child values array —— `[1, 0, 1, 1, 1, 1]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on PR #41189:
URL: https://github.com/apache/arrow/pull/41189#issuecomment-2057811021
Fixing these bugs require fixing the `Cast` kernel when the source is a fixed-size list because it currently relies on the values array having nulls on these empty values.
Example: casting a fixed-size list of strings to integer fails due to empty strings not casting successfully to int.
I’m working on the fix. It’s something that would be necessary even without this change in the builder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41189:
URL: https://github.com/apache/arrow/pull/41189#discussion_r1565888338
##########
cpp/src/arrow/array/builder_nested.cc:
##########
@@ -213,13 +213,13 @@ Status FixedSizeListBuilder::AppendValues(int64_t length, const uint8_t* valid_b
Status FixedSizeListBuilder::AppendNull() {
RETURN_NOT_OK(Reserve(1));
UnsafeAppendToBitmap(false);
- return value_builder_->AppendNulls(list_size_);
+ return value_builder_->AppendEmptyValues(list_size_);
Review Comment:
So might be a performance for deep nested thing like `FixLengthArray<struct<str, str, str>, ...>`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on PR #41189:
URL: https://github.com/apache/arrow/pull/41189#issuecomment-2054055091
> So the child validity would be undefined for null parent here?
It will de defined as valid + empty value. What empty means depends on the type. For integers it's `0`, for strings it's `""`...
Example from the CI logs:
```
___________ [doctest] pyarrow.lib.FixedSizeListArray.values.__get__ ____________
3323 FixedSizeListArray.flatten : ...
3324
3325 Examples
3326 --------
3327 >>> import pyarrow as pa
3328 >>> array = pa.array(
3329 ... [[1, 2], None, [3, None]],
3330 ... type=pa.list_(pa.int32(), 2)
3331 ... )
3332 >>> array.values
Differences (unified diff with -expected +actual):
@@ -1,8 +1,8 @@
-<pyarrow.lib.Int32Array object at ...>
+<pyarrow.lib.Int32Array object at 0x7f9ee8e430c0>
[
1,
2,
- null,
- null,
+ 0,
+ 0,
3,
null
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on PR #41189:
URL: https://github.com/apache/arrow/pull/41189#issuecomment-2059968343
> +1! I wonder would this approach have a threshold of null percentage above which it would make sense to keep current approach (assuming everything is nullable)? (I didn't look into `AppendEmptyValues` so this question might not even make sense)
This is not about optimizing the building of FSL arrays. Just minimizing the existence of unnecessary validity bitmaps. If at least one valid list's value is null, there will be a bitmap. The difference is that a null list doesn't imply nulls in the children.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41188: [C++] Don't recursively produce nulls when appending nulls to a FixedSizeListBuilder [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #41189:
URL: https://github.com/apache/arrow/pull/41189#issuecomment-2060003715
Oh, got it, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org