You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Jefffrey (via GitHub)" <gi...@apache.org> on 2026/10/14 05:13:25 UTC
[PR] fix: account for offsets in runarrays when using extend for MutableArrayData [arrow-rs]
Jefffrey opened a new pull request, #11546:
URL: https://github.com/apache/arrow-rs/pull/11546
# Which issue does this PR close?
<!--
It is not necessary to raise an issue, but it is encouraged if it is a non-trivial
enhancement/bug.
If this PR relates to an existing issue, but does not fully resolve it, please change
the wording below from "Closes #NNN" to "Relates to #NNN" or "Part of #NNN" to avoid
false closing an issue.
-->
None
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the
issue then this section is not needed. Explaining clearly why changes are proposed
helps reviewers understand your changes and offer better suggestions for fixes.
Please try to explain the problem you are trying to solve in terms of the user-visible
behavior, rather than the implementation.
-->
Correctness edge case when using RunArrays in MutableArrayData
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is
sometimes worth providing a summary of the individual changes in this PR.
-->
- Account for `ArrayData` offset of RunArrays when extending `MutableArrayData`
- Checked conversion of usize length to Run index type to avoid wrapping
# Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they
covered by existing tests, if so identify these tests)?
If this PR claims a performance improvement, please include evidence such as
benchmark results.
DO NOT simply paste a list of commands ran to verify the code; this is what our
CI is for and it pollutes the PR. ONLY explain if testing was included with this
PR, or why it wasn't.
-->
Yes
# Are there any significant user-facing changes?
<!--
Significant user-facing changes include:
- Breaking public API change
- Significant change to existing functionality (e.g. function returns something different)
These need to be explicitly called out to ensure we have proper documentation if
required and to ensure we follow proper semantic versioning.
DO NOT restate the changes of this PR; if a bug was fixed and already explained
above, we don't need this to be restated as a user-facing change.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: account for offsets in runarrays when using extend for MutableArrayData [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11546:
URL: https://github.com/apache/arrow-rs/pull/11546#discussion_r4274596030
##########
arrow-data/src/transform/run.rs:
##########
@@ -712,4 +714,56 @@ mod tests {
"unexpected error: {err}"
);
}
+
+ #[test]
+ fn test_extend_nulls_overflow() {
Review Comment:
I verified this fails as expected without this change
```shell
---- transform::run::tests::test_extend_accounts_for_offset stdout ----
thread 'transform::run::tests::test_extend_accounts_for_offset' (57027272) panicked at arrow-data/src/transform/run.rs:112:53:
called `Option::unwrap()` on a `None` value
---- transform::run::tests::test_extend_nulls_overflow stdout ----
thread 'transform::run::tests::test_extend_nulls_overflow' (57027278) panicked at arrow-data/src/transform/run.rs:722:67:
called `Result::unwrap_err()` on an `Ok` value: ()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: account for offsets in runarrays when using extend for MutableArrayData [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #11546:
URL: https://github.com/apache/arrow-rs/pull/11546#issuecomment-6183880094
thanks @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: account for offsets in runarrays when using extend for MutableArrayData [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #11546:
URL: https://github.com/apache/arrow-rs/pull/11546#issuecomment-6183938597
> @Jefffrey you might want to merge from main before adding this to the queue...all the queued jobs today failed on audit. The fix for this just merged in #11562.
i thought merge queue would merge up from main when running the CI 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: account for offsets in runarrays when using extend for MutableArrayData [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #11546:
URL: https://github.com/apache/arrow-rs/pull/11546#issuecomment-6183951455
Guess we'll see...I wasn't sure so I've been merging before re-queuing. Maybe that's not necessary 🤷
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: account for offsets in runarrays when using extend for MutableArrayData [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #11546:
URL: https://github.com/apache/arrow-rs/pull/11546#discussion_r4267040731
##########
arrow-data/src/transform/run.rs:
##########
@@ -230,6 +230,8 @@ pub fn build_extend(array: &ArrayData) -> Extend<'_> {
// Build run ends and values indices directly for batch processing
macro_rules! build_and_process_impl {
($run_end_type:ty) => {{
+ let source_buffer = &source_buffer.typed_data()
+ [source_buffer_offset..source_buffer_offset + source_buffer_len];
Review Comment:
in practice it would be uncommon to have a runarray hit this case, of having a runs child arraydata with an offset as it requires manual construction
- its similar to https://github.com/apache/arrow-rs/pull/11536
##########
arrow-data/src/transform/run.rs:
##########
@@ -58,8 +58,9 @@ pub fn extend_nulls(mutable: &mut _MutableArrayData, len: usize) -> Result<(), A
macro_rules! extend_nulls_impl {
($run_end_type:ty) => {{
let last_run_end = get_last_run_end::<$run_end_type>(&mutable.child_data[0]);
- let new_value = last_run_end
- .checked_add(<$run_end_type as ArrowNativeType>::usize_as(len))
Review Comment:
`len` is a usize, if it is > `i16::MAX` and the run index type is `i16` the this could wrap to negative
swapping in `from_usize` which can chain into `checked_add` and share the same `ok_or_else` flow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: account for offsets in runarrays when using extend for MutableArrayData [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #11546:
URL: https://github.com/apache/arrow-rs/pull/11546#discussion_r4267040731
##########
arrow-data/src/transform/run.rs:
##########
@@ -230,6 +230,8 @@ pub fn build_extend(array: &ArrayData) -> Extend<'_> {
// Build run ends and values indices directly for batch processing
macro_rules! build_and_process_impl {
($run_end_type:ty) => {{
+ let source_buffer = &source_buffer.typed_data()
+ [source_buffer_offset..source_buffer_offset + source_buffer_len];
Review Comment:
in practice it would be uncommon to have a runarray hit this case, of having a runs child arraydata with an offset as it requires manual construction via arraydata
- its similar to https://github.com/apache/arrow-rs/pull/11536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: account for offsets in runarrays when using extend for MutableArrayData [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #11546:
URL: https://github.com/apache/arrow-rs/pull/11546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: account for offsets in runarrays when using extend for MutableArrayData [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #11546:
URL: https://github.com/apache/arrow-rs/pull/11546#issuecomment-6183910766
@Jefffrey you might want to merge from main before adding this to the queue...all the queued jobs today failed on audit. The fix for this just merged in #11562.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org