[jira] [Commented] (ARROW-8684) [Packaging][Python] "SystemError: Bad call flags in _PyMethodDef_RawFastCallDict" in Python 3.7.7 on macOS when using pyarrow wheel

["Wes McKinney (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098623#comment-17098623 ] 

Wes McKinney commented on ARROW-8684:
-------------------------------------

I briefly tried creating a debug build on macOS but couldn't get the NumPy or pyarrow wheels to install into it, so I'm pretty out of my depth to begin debugging this. 

Here's the backtrace I see with lldb

{code}
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "pyarrow/types.pxi", line 2638, in pyarrow.lib._unregister_py_extension_types
SystemError: Bad call flags in _PyMethodDef_RawFastCallDict. METH_OLDARGS is no longer supported!
Process 86225 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xa00)
    frame #0: 0x00000001001b689a python`new_threadstate + 410
python`new_threadstate:
->  0x1001b689a <+410>: movq   0xa00(%r14), %rax
    0x1001b68a1 <+417>: incq   %rax
    0x1001b68a4 <+420>: movq   %rax, 0xa00(%r14)
    0x1001b68ab <+427>: movq   %rax, 0x110(%rbx)
Target 0: (python) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xa00)
  * frame #0: 0x00000001001b689a python`new_threadstate + 410
    frame #1: 0x00000001001b79b9 python`PyGILState_Ensure + 185
    frame #2: 0x000000012bbb7562 libarrow_python.17.dylib`arrow::py::OwnedRefNoGIL::~OwnedRefNoGIL() + 18
    frame #3: 0x000000012bbea74f libarrow_python.17.dylib`arrow::py::PyExtensionType::~PyExtensionType() + 47
    frame #4: 0x000000012aa4cec9 libarrow.17.dylib`arrow::ExtensionTypeRegistryImpl::~ExtensionTypeRegistryImpl() + 89
    frame #5: 0x000000012a903291 libarrow.17.dylib`std::__1::shared_ptr<arrow::Device>::~shared_ptr() + 49
    frame #6: 0x00007fff63df53cf libsystem_c.dylib`__cxa_finalize_ranges + 319
    frame #7: 0x00007fff63df56b3 libsystem_c.dylib`exit + 55
    frame #8: 0x00007fff63d4f3dc libdyld.dylib`start + 8
    frame #9: 0x00007fff63d4f3d5 libdyld.dylib`start + 1
{code}

So the segfault is happening it seems just because the call to pyarrow.lib._unregister_py_extension_types is failing. We could try moving the {{atexit}} business into pure Python land to see if that makes the problem go away

> [Packaging][Python] "SystemError: Bad call flags in _PyMethodDef_RawFastCallDict" in Python 3.7.7 on macOS when using pyarrow wheel
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-8684
>                 URL: https://issues.apache.org/jira/browse/ARROW-8684
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Wes McKinney
>            Priority: Blocker
>             Fix For: 1.0.0, 0.17.1
>
>
> [~npr] reported this on the 0.17.0 RC0 vote thread but I have confirmed it independently. It was also reported at
> https://github.com/apache/arrow/issues/7082
> Here are steps to reproduce on macOS:
> {code}
> conda create -yn py-3.7-defaults python=3.7 -c defaults
> conda activate py-3.7-defaults
> pip install pyarrow
> {code}
> Now open the Python interpreter, run {{import pyarrow}}, then exit the interpreter ({{python -c "import pyarrow"}} didn't trigger it for me):
> {code}
> $ python
> Python 3.7.7 (default, Mar 26 2020, 10:32:53) 
> [Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import pyarrow
> >>> 
> Error in atexit._run_exitfuncs:
> Traceback (most recent call last):
>   File "pyarrow/types.pxi", line 2638, in pyarrow.lib._unregister_py_extension_types
> SystemError: Bad call flags in _PyMethodDef_RawFastCallDict. METH_OLDARGS is no longer supported!
> Segmentation fault: 11
> {code}
> It fails with Python 3.7.6 when using {{-c conda-forge}} also, so it is not particular to defaults.
> Frustratingly, the problem doesn't exist in Python 3.7.4 but occurs for me with 3.7.5, 3.7.6, and 3.7.7. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)