You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "unj1m (via GitHub)" <gi...@apache.org> on 2024/04/04 19:37:36 UTC
[I] pyarrow.compute.and_ produces corrupted arrow files [arrow]
unj1m opened a new issue, #41016:
URL: https://github.com/apache/arrow/issues/41016
### Describe the bug, including details regarding any error messages, version, and platform.
```
$ python
Python 3.10.12 (main, Apr 4 2024, 12:45:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow.compute
>>> a1 = pyarrow.array([True]*48 + [False]*48)
>>> a2 = pyarrow.array([True, False] * 48)
>>> pyarrow.compute.and_(a1, a2).true_count
Segmentation fault (core dumped)
```
There segfault started in pyarrow==9.0.0.
It doesn't happen in 7 or 8.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2057212700
This will be released with 16.0.0. The 16.0.0 has been on feature freeze for a week and we are still fixing some CI jobs and bugs. If things go as planned we probably have a Release Candidate by the end of the week. We should have have a release in ~1-2 weeks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2046286847
After working for a while on the Arrow codebase, I can say that there is a lot of code that assumes the following (undocumented) invariant:
> `null_count` can only transition from `-1` to a value different from `0` **if and only if** the bitmap buffer present
another way to put it:
> well-formed arrays that have `null_count != -1` are that way for only one reason: `length > 0` and there is a bitmap that must be scanned to get to the real value of `null_count`
The fact that this invariant is undocumented means there are defensive coding here and there to protect against violations:
https://github.com/apache/arrow/blob/cd607d00b9e3c5f2ac2a2e373ba7571fd75fc725/cpp/src/arrow/array/data.h#L287-L291
...but there is also code that carefully guarantees that it's preserved in constructors:
https://github.com/apache/arrow/blob/cd607d00b9e3c5f2ac2a2e373ba7571fd75fc725/cpp/src/arrow/array/data.cc#L50-L67
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2046311183
> Is #38553 a similar issue? The error in that case is also related to nulls.
Yes. And the immediate fix is similar to what I recommended in @jorisvandenbossche PR:
Replace `data->null_count != 0` checks with `data->MayHaveNulls()` checks. This is the code:
https://github.com/apache/arrow/blob/cd607d00b9e3c5f2ac2a2e373ba7571fd75fc725/cpp/src/arrow/array/array_nested.cc#L835-L843
@jorisvandenbossche there needs to be an issue about investigating why `pyarrow` is producing these malformed arrays. Because it's unlikely that this was the case for too long and not one hit these crashes before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "rohanjain101 (via GitHub)" <gi...@apache.org>.
rohanjain101 commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2045812492
Is https://github.com/apache/arrow/issues/38553 a similar issue? The error in that case is also related to nulls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2057142287
Issue resolved by pull request 41070
https://github.com/apache/arrow/pull/41070
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "unj1m (via GitHub)" <gi...@apache.org>.
unj1m commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2042973313
Thank you for getting to the bottom of this so quickly.
Are changes also needed for the simple binary kernals?
I wonder why this isn't affecting my Arrow Rust code, which I assume is calling the same C++ code. I guess arrow-rs re-implements some functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2042524924
@unj1m Thanks for the report!
The direct cause of this crash is because inside the `null_count` method, we are incorrectly checking the array's `null_count` attribute to take a different code path, but when this attribute is still set to "unknown" (-1), this leads us to a code path that actually accesses the non-existing validity bitmap buffer.
Now, an additional underlying issue is that the binary "and" kernel (and I assume this is true for all simple binary kernels) should not produce a result with the `null_count` set to -1 if it didn't need to allocate any validity bitmap. I seem to recall a previous issue where this also came up, but don't directly find it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "unj1m (via GitHub)" <gi...@apache.org>.
unj1m commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2057237518
Cool, Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2046291210
This discussion triggered me to create this issue: https://github.com/apache/arrow/issues/41113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "unj1m (via GitHub)" <gi...@apache.org>.
unj1m commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2057201787
Any idea when this will be released? :smiley:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "kyle-hex (via GitHub)" <gi...@apache.org>.
kyle-hex commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2108867811
Unfortunately, my coding environment won't allow me to upgrade to pyarrow v 16. Is there any workaround for this that doesn't involve iterating over the entire array and testing each value individually?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2109467539
> Is there any workaround for this that doesn't involve iterating over the entire array and testing each value individually?
A workaround to avoid the segfault is adding a call to `null_count` before calling `true_count`. Editing the original example, the below does not crash on pyarrow 15.0:
```python
import pyarrow.compute
a1 = pyarrow.array([True]*48 + [False]*48)
a2 = pyarrow.array([True, False] * 48)
res = pyarrow.compute.and_(a1, a2)
# call null_count to populate it before calling true_count
res.null_count
true_count = res.true_count
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #41016: [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer
URL: https://github.com/apache/arrow/issues/41016
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] BooleanArray true_count crashing in case of unknown null count (null_count = -1) without validity buffer [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #41016:
URL: https://github.com/apache/arrow/issues/41016#issuecomment-2044251261
> I wonder why this isn't affecting my Arrow Rust code, which I assume is calling the same C++ code. I guess arrow-rs re-implements some functions.
Arrow C++ (on which pyarrow is based) and Arrow Rust are two completely independent implementations, so that is expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org