You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2017/05/09 22:49:04 UTC
[jira] [Updated] (ARROW-991) [Python] PyArray_SimpleNew should not
be used with NPY_DATETIME
[ https://issues.apache.org/jira/browse/ARROW-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wes McKinney updated ARROW-991:
-------------------------------
Description:
From a non-deterministic test failure in pandas reported to me by [~jreback], I see that {{PyArray_SimpleNew}} calls {{PyArray_DescrFromType}}, which returns a reference to a cached dtype: https://github.com/numpy/numpy/blob/1d592c12ca7f9c7f471aa8d20b538c5cb4f2cdce/numpy/core/src/multiarray/arraytypes.c.src#L4403. Since we expect to be able to mutate a datetime64 dtype in libarrow_python, this can cause failures in other code when uses a dtype that we have accidentally mutated internally.
Luckily we are only modifying dtype metadata in a couple of places so this should not be too invasive to fix: https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/pandas_convert.cc#L1995
was:
From a non-deterministic test failure in pandas reported to me by [~jreback], I see that {{PyArray_SimpleNew}} calls {{PyArray_DescrFromType}}, which return a reference to a cached dtype: https://github.com/numpy/numpy/blob/1d592c12ca7f9c7f471aa8d20b538c5cb4f2cdce/numpy/core/src/multiarray/arraytypes.c.src#L4403. Since we expect to be able to mutate a datetime64 dtype in libarrow_python, this can cause failures in other code when uses a dtype that we have accidentally mutated internally.
Luckily we are only modifying dtype metadata in a couple of places so this should not be too invasive to fix: https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/pandas_convert.cc#L1995
> [Python] PyArray_SimpleNew should not be used with NPY_DATETIME
> ---------------------------------------------------------------
>
> Key: ARROW-991
> URL: https://issues.apache.org/jira/browse/ARROW-991
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.3.0
> Reporter: Wes McKinney
> Assignee: Wes McKinney
> Priority: Critical
> Fix For: 0.4.0
>
>
> From a non-deterministic test failure in pandas reported to me by [~jreback], I see that {{PyArray_SimpleNew}} calls {{PyArray_DescrFromType}}, which returns a reference to a cached dtype: https://github.com/numpy/numpy/blob/1d592c12ca7f9c7f471aa8d20b538c5cb4f2cdce/numpy/core/src/multiarray/arraytypes.c.src#L4403. Since we expect to be able to mutate a datetime64 dtype in libarrow_python, this can cause failures in other code when uses a dtype that we have accidentally mutated internally.
> Luckily we are only modifying dtype metadata in a couple of places so this should not be too invasive to fix: https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/pandas_convert.cc#L1995
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)