You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "jorisvandenbossche (via GitHub)" <gi...@apache.org> on 2024/01/22 15:40:46 UTC
[I] [C++] Filter kernel for MonthDayNanoInterval array returning garbage values [arrow]
jorisvandenbossche opened a new issue, #39740:
URL: https://github.com/apache/arrow/issues/39740
Example with latest pyarrow:
```
>>> import pyarrow as pa
>>> pa.__version__
Out[30]: '15.0.0.dev436+gac50918a6.d20240117'
>>> import pyarrow.compute as pc
>>> arr = pa.array([pa.MonthDayNano((1, 1, 1)), pa.MonthDayNano((2, 2, 2))])
>>> arr
<pyarrow.lib.MonthDayNanoIntervalArray object at 0x7f8e89147220>
[
1M1d1ns,
2M2d2ns
]
>>> pc.filter(arr, pa.array([True, False]))
Out[36]:
<pyarrow.lib.MonthDayNanoIntervalArray object at 0x7f8e89e24580>
[
3M0d0ns # <-- this value didn't even exist in the original array
]
>>> arr = pa.array([pa.MonthDayNano((1, 1, 1)), None])
>>> arr
<pyarrow.lib.MonthDayNanoIntervalArray object at 0x7f8e89e24820>
[
1M1d1ns,
null
]
>>> pc.filter(arr, pa.array([True, False]))
<pyarrow.lib.MonthDayNanoIntervalArray object at 0x7f8e88e4eda0>
[
null # <-- this should have selected the first element of the array, but returning null instead
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Filter kernel for MonthDayNanoInterval array returning garbage values [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #39740:
URL: https://github.com/apache/arrow/issues/39740#issuecomment-1906387875
At least, the DCHECK should be turned into an actual error return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Filter kernel for MonthDayNanoInterval array returning garbage values [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #39740:
URL: https://github.com/apache/arrow/issues/39740#issuecomment-1906389981
> We should maybe also consider to stop treating MonthDayNano interval automatically as primitive type
Well, it _is_ a primitive type. This is just a bug in the take and filter code that it doesn't properly handle more primitive type widths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Filter kernel for MonthDayNanoInterval array returning garbage values [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #39740:
URL: https://github.com/apache/arrow/issues/39740#issuecomment-1904351359
We should maybe also consider to stop treating MonthDayNano interval automatically as primitive type (`is_primitive` in type_traits.h returns True for all interval types)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Filter kernel for MonthDayNanoInterval array returning garbage values [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #39740:
URL: https://github.com/apache/arrow/issues/39740#issuecomment-1904285784
Note that also take is failing (in the end filter is translated into an equivalent take, I suppose, or at least sharing code):
```
>>> arr.take([0])
<pyarrow.lib.MonthDayNanoIntervalArray object at 0x7f8e88f93520>
[
null
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Filter kernel for MonthDayNanoInterval array returning garbage values [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #39740: [C++] Filter kernel for MonthDayNanoInterval array returning garbage values
URL: https://github.com/apache/arrow/issues/39740
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Filter kernel for MonthDayNanoInterval array returning garbage values [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #39740:
URL: https://github.com/apache/arrow/issues/39740#issuecomment-1904308478
The interval types are considered as a primitive type, so that's the reason this kernel is not raising a not-implemented error (while there is no explicit support for intervals).
However, it seems this PrimitiveTakeExec isn't set up to handle the larger bit width of internvals (128 for MonthDayNano):
https://github.com/apache/arrow/blob/3b73f438bf0a1abc76009f017c1df9c244854aca/cpp/src/arrow/compute/kernels/vector_selection_take_internal.cc#L575-L595
So we are just allocating some values for the output, but then not setting anything (and also not raising an error for this, because it's only a debug check ..)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Filter kernel for MonthDayNanoInterval array returning garbage values [arrow]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #39740:
URL: https://github.com/apache/arrow/issues/39740#issuecomment-1906932633
Out of curiosity how does Arrow define "primitive"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Filter kernel for MonthDayNanoInterval array returning garbage values [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #39740:
URL: https://github.com/apache/arrow/issues/39740#issuecomment-1907847260
Unfortunately there's no rigorous conceptual definition, it's more of an adhoc category. Roughly, everything that's a number, a boolean or a temporal is considered primitive, except decimals for some reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org