You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "fenfeng9 (via GitHub)" <gi...@apache.org> on 2026/05/02 10:34:43 UTC
[PR] GH-49888: [C++][Compute] Count logical nulls in run-end encoded arrays [arrow]
fenfeng9 opened a new pull request, #49908:
URL: https://github.com/apache/arrow/pull/49908
### Rationale for this change
The `count` kernel used `GetNullCount()`, which reports the physical null count. For run-end encoded arrays, this ignored nulls in the encoded values child.
### What changes are included in this PR?
Use `ComputeLogicalNullCount()` in the `count` kernel so run-end encoded arrays are counted correctly. Add C++ and Python tests for this case.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49888: [C++][Compute] Fix count for run-end encoded arrays with nulls [arrow]
Posted by "tadeja (via GitHub)" <gi...@apache.org>.
tadeja commented on code in PR #49908:
URL: https://github.com/apache/arrow/pull/49908#discussion_r3183467975
##########
python/pyarrow/tests/test_compute.py:
##########
@@ -2849,6 +2849,14 @@ def test_count():
pc.count(arr, 'something else')
+def test_count_run_end_encoded_nulls():
+ arr = pc.run_end_encode(pa.array([1, None]))
+
+ assert pc.count(arr, mode="only_valid").as_py() == 1
+ assert pc.count(arr, mode="only_null").as_py() == 1
+ assert pc.count(arr, mode="all").as_py() == 2
Review Comment:
```suggestion
arr = pc.run_end_encode(
pa.array([1, 1, None, None, None, 2, 2, 2, None, 3]))
assert pc.count(arr, mode="only_valid").as_py() == 6
assert pc.count(arr, mode="only_null").as_py() == 4
assert pc.count(arr, mode="all").as_py() == 10
# Slice crosses run boundaries: logical [None, None, 2, 2, 2, None].
assert pc.count(arr.slice(3, 6), mode="only_valid").as_py() == 3
assert pc.count(arr.slice(3, 6), mode="only_null").as_py() == 3
```
Similar suggestion here, so all three modes go over multiple runs and one slice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49888: [C++][Compute] Fix count for run-end encoded arrays with nulls [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #49908:
URL: https://github.com/apache/arrow/pull/49908
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49888: [C++][Compute] Fix count for run-end encoded arrays with nulls [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #49908:
URL: https://github.com/apache/arrow/pull/49908#issuecomment-4389511767
After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 1b3f31300979cd6976a2125a8dccfa7c049706ca.
None of the specified runs were found on the Conbench server.
The [full Conbench report](https://github.com/apache/arrow/runs/74643846756) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49888: [C++][Compute] Fix count for run-end encoded arrays with nulls [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #49908:
URL: https://github.com/apache/arrow/pull/49908#issuecomment-4458401162
After merging your PR, Conbench analyzed the 2 benchmarking runs that have been run so far on merge-commit 1b3f31300979cd6976a2125a8dccfa7c049706ca.
There was 1 benchmark result with an error:
- Commit Run on `amd64-m5-4xlarge-linux` at [2026-05-15 02:41:57Z](https://conbench.arrow-dev.org/compare/runs/9046ed0b98a94f3984397582e86d41cd...07818e956851463f82a6cd7d0d2bf640/)
- [`csv-read` (Python) with compression=uncompressed, dataset=nyctaxi_2010-01, output_format=arrow_table, streaming=streaming](https://conbench.arrow-dev.org/benchmark-results/06a068bb281973638000d372e659d778)
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/76149979154) has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49888: [C++][Compute] Fix count for run-end encoded arrays with nulls [arrow]
Posted by "fenfeng9 (via GitHub)" <gi...@apache.org>.
fenfeng9 commented on PR #49908:
URL: https://github.com/apache/arrow/pull/49908#issuecomment-4380526257
The test failures look unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49888: [C++][Compute] Count logical nulls in run-end encoded arrays [arrow]
Posted by "fenfeng9 (via GitHub)" <gi...@apache.org>.
fenfeng9 commented on PR #49908:
URL: https://github.com/apache/arrow/pull/49908#issuecomment-4363612434
The added tests use a simplified version of the reproducer from the issue.
#### Reproduce
```python
import pyarrow as pa
import pyarrow.compute as pc
array = pa.array([1, None])
encoded = pc.run_end_encode(array)
print(f"plain only_null: ", pc.count(array, mode="only_null"))
print(f"run_end_encode only_null: ", pc.count(encoded, mode="only_null"))
```
#### Result
```
plain only_null: 1
run_end_encode only_null: 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49888: [C++][Compute] Fix count for run-end encoded arrays with nulls [arrow]
Posted by "fenfeng9 (via GitHub)" <gi...@apache.org>.
fenfeng9 commented on PR #49908:
URL: https://github.com/apache/arrow/pull/49908#issuecomment-4379524089
Thanks for the suggestion. I updated the C++ and Python tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49888: [C++][Compute] Fix count for run-end encoded arrays with nulls [arrow]
Posted by "tadeja (via GitHub)" <gi...@apache.org>.
tadeja commented on code in PR #49908:
URL: https://github.com/apache/arrow/pull/49908#discussion_r3183425057
##########
cpp/src/arrow/compute/kernels/aggregate_test.cc:
##########
@@ -941,6 +941,13 @@ TYPED_TEST(TestCountKernel, SimpleCount) {
EXPECT_THAT(Count(*MakeScalar(ty, 1), all), ResultWith(Datum(int64_t(1))));
}
+TEST(TestCountKernel, RunEndEncodedNulls) {
+ auto input = ArrayFromJSON(int32(), "[1, null]");
+ ASSERT_OK_AND_ASSIGN(auto encoded, RunEndEncode(input));
+
+ ValidateCount(*encoded.make_array(), {1, 1});
Review Comment:
```suggestion
auto input = ArrayFromJSON(int32(), "[1, 1, null, null, null, 2, 2, 2, null, 3]");
ASSERT_OK_AND_ASSIGN(auto encoded, RunEndEncode(input));
ValidateCount(*encoded.make_array(), {6, 4});
// Slice crosses run boundaries: logical [null, null, 2, 2, 2, null].
ValidateCount(*encoded.make_array()->Slice(3, 6), {3, 3});
```
Just a small suggestion from my end to make the test harder to break in the future, if you'd like to consider it. Asymmetric `{6, 4}` covers `run_length()` weighting (instead of symmetric `{1,1}`), and `Slice(3, 6)` covers `values.offset` in `ree_util::LogicalNullCount`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49888: [C++][Compute] Fix count for run-end encoded arrays with nulls [arrow]
Posted by "fenfeng9 (via GitHub)" <gi...@apache.org>.
fenfeng9 commented on PR #49908:
URL: https://github.com/apache/arrow/pull/49908#issuecomment-4379552496
The original behavior is:
#### Reproduce
```python
import pyarrow as pa
import pyarrow.compute as pc
array = pa.array([1, 1, None, None, None, 2, 2, 2, None, 3])
encoded = pc.run_end_encode(array)
# Logical slice: [None, None, 2, 2, 2, None].
slice_plain = array.slice(3, 6)
slice_encoded = encoded.slice(3, 6)
print("pyarrow:", pa.__version__)
print()
print(f"{'case':<12} {'only_valid':>10} {'only_null':>10} {'all':>6}")
for name, value in [
("plain", array),
("ree", encoded),
("slice plain", slice_plain),
("slice ree", slice_encoded),
]:
print(
f"{name:<12} "
f"{pc.count(value, mode='only_valid').as_py():>10} "
f"{pc.count(value, mode='only_null').as_py():>10} "
f"{pc.count(value, mode='all').as_py():>6}"
)
```
#### Result
```
pyarrow: 24.0.0
case only_valid only_null all
plain 6 4 10
ree 10 0 10
slice plain 3 3 6
slice ree 6 0 6
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49888: [C++][Compute] Fix count for run-end encoded arrays with nulls [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #49908:
URL: https://github.com/apache/arrow/pull/49908#issuecomment-4389043225
And thanks @tadeja for the useful reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org