You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "llama90 (via GitHub)" <gi...@apache.org> on 2024/04/20 04:24:11 UTC

[I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error on M1 Macbook [arrow]

llama90 opened a new issue, #41312:
URL: https://github.com/apache/arrow/issues/41312

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   
   I've received the latest code and encountered an error while running the Python unit tests.
   
   <details><summary>error message</summary>
   
   ```
   python -m pytest pyarrow/tests/test_dataset.py
   ============================================================================================================ test session starts =============================================================================================================
   platform darwin -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
   rootdir: /Users/lama/workspace/arrow-new/python
   configfile: setup.cfg
   plugins: hypothesis-6.99.9
   collected 351 items
   
   pyarrow/tests/test_dataset.py ...............................................................................................................s........................................................................................ [ 56%]
   ...........ssssss....................ssss.s.................................................................................ss........................Fatal Python error: Segmentation fault
   
   Current thread 0x000000020330bac0 (most recent call first):
     File "/Users/lama/workspace/arrow-new/python/pyarrow/tests/test_dataset.py", line 5645 in test_make_write_options_error
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pytest/__main__.py", line 5 in <module>
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/runpy.py", line 86 in _run_code
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/runpy.py", line 196 in _run_module_as_main
   
   Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pyarrow._fs, cython.cimports.libc.math, pyarrow._compute, pyarrow._acero, pyarrow._csv, pyarrow._json, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_
 dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._dataset, pyarrow._parquet, pyarrow._dataset_parquet, pyarrow._feather (total: 64)
   [1]    41728 segmentation fault  python -m pytest pyarrow/tests/test_dataset.py
   ```
   
   </details>
   
   There was an error in `test_dataset.py`, so I checked whether the submodule was up to date.
   
   ```bash
   git submodule status
    74278bc4a1122d74945969e6dec405abd1533ec3 ../cpp/submodules/parquet-testing (heads/master)
    735ae7128d571398dd798d7ff004adebeb342883 ../testing (remotes/origin/HEAD)
   ```
   
   Upon further inspection, it turns out that the error occurs in the tests for the `test_make_write_options_error` function.
   
   <details><summary>run test without test_make_write_options_error</summary>
   
   ```bash
   python -m pytest pyarrow/tests/test_dataset.py -k "not test_make_write_options_error"
   
   ============================================================================================================ test session starts =============================================================================================================
   platform darwin -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
   rootdir: /Users/lama/workspace/arrow-new/python
   configfile: setup.cfg
   plugins: hypothesis-6.99.9
   collected 351 items / 1 deselected / 350 selected
   
   pyarrow/tests/test_dataset.py ...............................................................................................................s........................................................................................ [ 57%]
   ...........ssssss....................ssss.s.................................................................................ss........................                                                                                 [100%]
   
   =============================================================================================== 336 passed, 14 skipped, 1 deselected in 17.20s ===============================================================================================
   ```
   
   </details>
   
   If I exclude that test, all other tests pass successfully.
   
   I use an M1 Macbook.
   
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2067778929

   Hmm. Does the https://github.com/apache/arrow/pull/40976/files#diff-4e3747424164e360600eab8972114148de49d999bff7b61157c0e535752d87aeR202-R204 check not working on the environment?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2067704305

   It is related to the [PR 40976](https://github.com/apache/arrow/pull/40976), and in the case of CI, the unit test seems to [pass for macOS 12](https://github.com/apache/arrow/actions/runs/8558433247/job/23452987925?pr=40976#step:6:137), but I'm using 14.4, so I guess it fails...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2067990519

   I tested only the `test_make_write_options_error` as a test case in CI, and there were no issues.
   
   - https://github.com/apache/arrow/actions/runs/8771799647/job/24070017215?pr=41313#step:6:81
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2118573569

   Please take a look at this [part](https://arrow.apache.org/cookbook/py/io.html#writing-partitioned-datasets). Upon examining example codes, you can see that the `dataset` is used as follows, and this syntax works correctly: `import pyarrow.dataset as ds`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2118545631

   In my case, I had set up the environment to build PyArrow using cpp code. However, I found that using the code you provided still resulted in the same symptoms. 🤔


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2067685938

   I attached lldb backtrace result.
   
   <details><summary>lldb result</summary>
   
   ```
   lldb -- python -m pytest pyarrow/tests/test_dataset.py -k "test_make_write_options_error"
   (lldb) target create "python"
   Current executable set to '/Users/lama/anaconda3/envs/pyarrow-dev-310/bin/python' (arm64).
   (lldb) settings set -- target.run-args  "-m" "pytest" "pyarrow/tests/test_dataset.py" "-k" "test_make_write_options_error"
   (lldb) run
   Process 72156 launched: '/Users/lama/anaconda3/envs/pyarrow-dev-310/bin/python' (arm64)
   ============================= test session starts ==============================
   platform darwin -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
   rootdir: /Users/lama/workspace/arrow-new/python
   configfile: setup.cfg
   plugins: hypothesis-6.99.9
   collected 351 items / 350 deselected / 1 selected
   
   Process 72156 stopped
   * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x1)
       frame #0: 0x0000000106533b78 _dataset.cpython-310-darwin.so`__pyx_pw_7pyarrow_8_dataset_10FileFormat_7make_write_options(_object*, _object* const*, long, _object*) + 56
   _dataset.cpython-310-darwin.so`__pyx_pw_7pyarrow_8_dataset_10FileFormat_7make_write_options:
   ->  0x106533b78 <+56>: ldr    x8, [x0]
       0x106533b7c <+60>: ldr    x9, [x8, #0x60]
       0x106533b80 <+64>: add    x8, sp, #0x30
       0x106533b84 <+68>: blr    x9
   Target 0: (python) stopped.
   (lldb) bt
   * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x1)
     * frame #0: 0x0000000106533b78 _dataset.cpython-310-darwin.so`__pyx_pw_7pyarrow_8_dataset_10FileFormat_7make_write_options(_object*, _object* const*, long, _object*) + 56
       frame #1: 0x0000000126b6e4d8 _dataset_parquet.cpython-310-darwin.so`__pyx_pw_7pyarrow_16_dataset_parquet_17ParquetFileFormat_3make_write_options(_object*, _object* const*, long, _object*) + 260
       frame #2: 0x0000000100149540 python`call_function + 524
       frame #3: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #4: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #5: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
       frame #6: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #7: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
       frame #8: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #9: 0x0000000100149540 python`call_function + 524
       frame #10: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #11: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #12: 0x000000010005ac64 python`method_vectorcall + 172
       frame #13: 0x0000000100149540 python`call_function + 524
       frame #14: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #15: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #16: 0x00000001000577ac python`_PyObject_FastCallDictTstate + 156
       frame #17: 0x0000000100058454 python`_PyObject_Call_Prepend + 176
       frame #18: 0x00000001000cdee0 python`slot_tp_call + 116
       frame #19: 0x000000010005759c python`_PyObject_MakeTpCall + 612
       frame #20: 0x00000001001495d8 python`call_function + 676
       frame #21: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
       frame #22: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #23: 0x0000000100149540 python`call_function + 524
       frame #24: 0x0000000100144e38 python`_PyEval_EvalFrameDefault + 24900
       frame #25: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #26: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
       frame #27: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #28: 0x0000000100149540 python`call_function + 524
       frame #29: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #30: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #31: 0x000000010005ac64 python`method_vectorcall + 172
       frame #32: 0x0000000100149540 python`call_function + 524
       frame #33: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #34: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #35: 0x00000001000577ac python`_PyObject_FastCallDictTstate + 156
       frame #36: 0x0000000100058454 python`_PyObject_Call_Prepend + 176
       frame #37: 0x00000001000cdee0 python`slot_tp_call + 116
       frame #38: 0x0000000100057fb0 python`_PyObject_Call + 148
       frame #39: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
       frame #40: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #41: 0x0000000100149540 python`call_function + 524
       frame #42: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
       frame #43: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #44: 0x000000010005ac64 python`method_vectorcall + 172
       frame #45: 0x0000000100149540 python`call_function + 524
       frame #46: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
       frame #47: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #48: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
       frame #49: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #50: 0x0000000100149540 python`call_function + 524
       frame #51: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
       frame #52: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #53: 0x0000000100149540 python`call_function + 524
       frame #54: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
       frame #55: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #56: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
       frame #57: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #58: 0x0000000100149540 python`call_function + 524
       frame #59: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #60: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #61: 0x000000010005ac64 python`method_vectorcall + 172
       frame #62: 0x0000000100149540 python`call_function + 524
       frame #63: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #64: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #65: 0x00000001000577ac python`_PyObject_FastCallDictTstate + 156
       frame #66: 0x0000000100058454 python`_PyObject_Call_Prepend + 176
       frame #67: 0x00000001000cdee0 python`slot_tp_call + 116
       frame #68: 0x000000010005759c python`_PyObject_MakeTpCall + 612
       frame #69: 0x00000001001495d8 python`call_function + 676
       frame #70: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
       frame #71: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #72: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
       frame #73: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #74: 0x0000000100149540 python`call_function + 524
       frame #75: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #76: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #77: 0x000000010005ac64 python`method_vectorcall + 172
       frame #78: 0x0000000100149540 python`call_function + 524
       frame #79: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #80: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #81: 0x00000001000577ac python`_PyObject_FastCallDictTstate + 156
       frame #82: 0x0000000100058454 python`_PyObject_Call_Prepend + 176
       frame #83: 0x00000001000cdee0 python`slot_tp_call + 116
       frame #84: 0x000000010005759c python`_PyObject_MakeTpCall + 612
       frame #85: 0x00000001001495d8 python`call_function + 676
       frame #86: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
       frame #87: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #88: 0x0000000100149540 python`call_function + 524
       frame #89: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
       frame #90: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #91: 0x0000000100149540 python`call_function + 524
       frame #92: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
       frame #93: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #94: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
       frame #95: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #96: 0x0000000100149540 python`call_function + 524
       frame #97: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #98: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #99: 0x000000010005ac64 python`method_vectorcall + 172
       frame #100: 0x0000000100149540 python`call_function + 524
       frame #101: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #102: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #103: 0x00000001000577ac python`_PyObject_FastCallDictTstate + 156
       frame #104: 0x0000000100058454 python`_PyObject_Call_Prepend + 176
       frame #105: 0x00000001000cdee0 python`slot_tp_call + 116
       frame #106: 0x000000010005759c python`_PyObject_MakeTpCall + 612
       frame #107: 0x00000001001495d8 python`call_function + 676
       frame #108: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
       frame #109: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #110: 0x0000000100149540 python`call_function + 524
       frame #111: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
       frame #112: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #113: 0x0000000100149540 python`call_function + 524
       frame #114: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
       frame #115: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #116: 0x000000010013db40 python`PyEval_EvalCode + 120
       frame #117: 0x0000000100139760 python`builtin_exec + 836
       frame #118: 0x00000001000ab880 python`cfunction_vectorcall_FASTCALL + 200
       frame #119: 0x0000000100149540 python`call_function + 524
       frame #120: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
       frame #121: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #122: 0x0000000100149540 python`call_function + 524
       frame #123: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
       frame #124: 0x000000010013e364 python`_PyEval_Vector + 2036
       frame #125: 0x00000001001bcfc4 python`pymain_run_module + 272
       frame #126: 0x00000001001bc604 python`Py_RunMain + 1584
       frame #127: 0x00000001001bda54 python`pymain_main + 1180
       frame #128: 0x000000010000131c python`main + 56
       frame #129: 0x000000019b2760e0 dyld`start + 2360
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068575753

   I tried reproducing on my M1 with Python 3.11 but no luck. The tests is run, not skipped, and it passes:
   
   ```python
   (pyarrow-dev) alenkafrim@Alenkas-MacBook-Pro arrow % python -m pytest python/pyarrow/tests/test_dataset.py -k test_make_write_options_error
   =================================================================== test session starts ===================================================================
   platform darwin -- Python 3.11.7, pytest-7.4.4, pluggy-1.4.0
   rootdir: /Users/alenkafrim/repos/arrow/python
   configfile: setup.cfg
   plugins: hypothesis-6.99.5, cython-0.2.1
   collected 351 items / 350 deselected / 1 selected                                                                                                         
   
   python/pyarrow/tests/test_dataset.py .                                                                                                              [100%]
   
   ============================================================ 1 passed, 350 deselected in 0.56s ============================================================
   ```
   
   I will also try with Python 3.10 later. I am currently at the conference so might take some time.
   
   I suspect though that something might be off with the setup? Could you try a clean new build (`git clean -Xfd .` in `arrow/python` or maybe even in `arrow/`), also cleaning `dist/` folder (https://arrow.apache.org/docs/dev/developers/python.html#using-system-and-bundled-dependencies)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068312242

   @AlenkaF Could you take a look at this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068024528

   I checked a few things:
   
   I was using Python `3.10` on my environment, but the CI system was using Python `3.11`. So, I set up Python `3.11` with conda and ran the tests again.
   
   I found that some tests in `pyarrow/tests/test_dataset.py` were skipped when I tested. But it can be executed with `export PYARROW_TEST_DATASET=ON`.
   
   Even though I built it by referring to the contents of ci , an error appears...
   
   - https://github.com/apache/arrow/blob/16e20b712ab8dafa9d1ec6154415fe5748fe6135/.github/workflows/python.yml#L137-L157
   
   #### python 3.11
   
   <details><summary>test result</summary>
   
   ```bash
   (pyarrow-dev-311) python -m pytest -k "test_make_write_options_error"
   ========================================================================================================== test session starts ===========================================================================================================
   platform darwin -- Python 3.11.9, pytest-8.1.1, pluggy-1.5.0
   rootdir: /Users/lama/workspace/arrow-new/python
   configfile: setup.cfg
   plugins: hypothesis-6.100.1
   collected 7667 items / 7666 deselected / 4 skipped / 1 selected
   
   pyarrow/tests/test_dataset.py F                                                                                                                                                                                                    [100%]
   
   ================================================================================================================ FAILURES ================================================================================================================
   _____________________________________________________________________________________________________ test_make_write_options_error ______________________________________________________________________________________________________
   
       def test_make_write_options_error():
           # GH-39440: calling make_write_options as a static class method
           msg_1 = ("make_write_options() should be called on an "
                    "instance of ParquetFileFormat")
           # GH-41043: In Cython2 all Cython methods were "regular" C extension methods
           # see: https://github.com/cython/cython/issues/6127#issuecomment-2038153359
           msg_2 = ("descriptor 'make_write_options' for "
                    "'pyarrow._dataset_parquet.ParquetFileFormat' objects "
                    "doesn't apply to a 'int'")
           with pytest.raises(TypeError) as excinfo:
   >           pa.dataset.ParquetFileFormat.make_write_options(43)
   E           AttributeError: module 'pyarrow' has no attribute 'dataset'
   
   pyarrow/tests/test_dataset.py:5645: AttributeError
   ======================================================================================================== short test summary info =========================================================================================================
   FAILED pyarrow/tests/test_dataset.py::test_make_write_options_error - AttributeError: module 'pyarrow' has no attribute 'dataset'
   ============================================================================================= 1 failed, 4 skipped, 7666 deselected in 1.05s ==============================================================================================
   ```
   
   </details>
   
   #### Python 3.10
   
   <details><summary>test result</summary>
   
   ```
   (pyarrow-dev-310) python -m pytest -k "test_make_write_options_error"
   ============================================================================================================ test session starts =============================================================================================================
   platform darwin -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
   rootdir: /Users/lama/workspace/arrow-new/python
   configfile: setup.cfg
   plugins: hypothesis-6.99.9
   collected 7667 items / 7666 deselected / 4 skipped / 1 selected
   
   pyarrow/tests/test_dataset.py Fatal Python error: Segmentation fault
   
   Current thread 0x000000020330bac0 (most recent call first):
     File "/Users/lama/workspace/arrow-new/python/pyarrow/tests/test_dataset.py", line 5645 in test_make_write_options_error
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pytest/__main__.py", line 5 in <module>
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/runpy.py", line 86 in _run_code
     File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/runpy.py", line 196 in _run_module_as_main
   
   Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pyarrow._fs, cython.cimports.libc.math, pyarrow._compute, pyarrow._acero, pyarrow._csv, pyarrow._json, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_
 dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._dataset, pyarrow._parquet, pyarrow._dataset_parquet, _cffi_backend, pyarrow._pyarrow_cpp_tests, pyarrow._feather (total: 66)
   [1]    77852 segmentation fault  python -m pytest -k "test_make_write_options_error"
   ```
   
   </details>
   
   Additionally, by enabling detailed output for the tests, it has been confirmed that the test is being passed when triggered.
   
   - `tests/test_dataset.py::test_make_write_options_error <- ../../../../../Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyarrow/tests/test_dataset.py PASSED
    `
     - https://github.com/apache/arrow/actions/runs/8772308641/job/24071088984?pr=41313#step:6:90
     - https://github.com/apache/arrow/actions/runs/8772308641/job/24071089064?pr=41313#step:6:90
   
   ## To summarize
   
   - The test appears not to have been run even though `PYARROW_TEST_DATASET` is `ON`.
     - Why are tests treated as passed in CI? 
   - If running the test locally, different errors appear depending on the python version.
     - 3.10: segmentation fault
     - 3.11: AttributeError: module 'pyarrow' has no attribute 'dataset'
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 closed issue #41312: [Python] Broken unit test: Segmentation fault in test_make_write_options_error
URL: https://github.com/apache/arrow/issues/41312


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "martibosch (via GitHub)" <gi...@apache.org>.
martibosch commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2117762625

   I cannot manage to use pyarrow on my ubuntu computer using conda. I have tried creating conda-forge environments with pyarrow (16.0.0) and also creating the environment with only python 3.10 and 3.11 and then installing pyarrow (16.1.0) via pip, and I always get `AttributeError: module 'pyarrow' has no attribute 'dataset'`. This can be reproduced as in:
   
   ```
   conda env create -n pyarrow -c conda-forge python=3.11 pyarrow=16
   conda activate pyarrow
   python -c 'import pyarrow as pa;pa.dataset'
   ```
   
   or:
   
   ```
   conda env create -n pyarrow -c conda-forge python=3.11
   conda activate pyarrow
   pip install pyarrow==16.1.0
   python -c 'import pyarrow as pa;pa.dataset'
   ```
   
   How can I fix this? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2067818479

   Yes, it looks like it was supposed to check there, but it's not working as expected.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068859092

   Thanks for trying it out and confirming! Maybe it would be worth testing with new conda env also.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]

Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068658290

   Hello @AlenkaF 
   
   I followed the suggested documentation and cleaned all builds, then created a Python virtual environment using `python3 -m venv` instead of conda. I confirmed this resolved the issue.
   
   ```bash
   python -m pytest python/pyarrow/tests/test_dataset.py -k test_make_write_options_error
   
   ========================================================================================================== test session starts ===========================================================================================================
   platform darwin -- Python 3.11.9, pytest-8.1.1, pluggy-1.5.0
   rootdir: /Users/lama/workspace/arrow-new/python
   configfile: setup.cfg
   plugins: hypothesis-6.100.1
   collected 351 items / 350 deselected / 1 selected
   
   python/pyarrow/tests/test_dataset.py .                                                                                                                                                                                             [100%]
   
   =================================================================================================== 1 passed, 350 deselected in 0.64s ====================================================================================================
   ```
   
   I still don't fully understand why it doesn't work with conda, but I gather it's due to an environmental issue. 
   
   Thank you for helping me check!
   
   cc @kou 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org