You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "llama90 (via GitHub)" <gi...@apache.org> on 2024/04/20 04:24:11 UTC
[I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error on M1 Macbook [arrow]
llama90 opened a new issue, #41312:
URL: https://github.com/apache/arrow/issues/41312
### Describe the bug, including details regarding any error messages, version, and platform.
I've received the latest code and encountered an error while running the Python unit tests.
<details><summary>error message</summary>
```
python -m pytest pyarrow/tests/test_dataset.py
============================================================================================================ test session starts =============================================================================================================
platform darwin -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /Users/lama/workspace/arrow-new/python
configfile: setup.cfg
plugins: hypothesis-6.99.9
collected 351 items
pyarrow/tests/test_dataset.py ...............................................................................................................s........................................................................................ [ 56%]
...........ssssss....................ssss.s.................................................................................ss........................Fatal Python error: Segmentation fault
Current thread 0x000000020330bac0 (most recent call first):
File "/Users/lama/workspace/arrow-new/python/pyarrow/tests/test_dataset.py", line 5645 in test_make_write_options_error
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pytest/__main__.py", line 5 in <module>
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/runpy.py", line 86 in _run_code
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/runpy.py", line 196 in _run_module_as_main
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pyarrow._fs, cython.cimports.libc.math, pyarrow._compute, pyarrow._acero, pyarrow._csv, pyarrow._json, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_
dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._dataset, pyarrow._parquet, pyarrow._dataset_parquet, pyarrow._feather (total: 64)
[1] 41728 segmentation fault python -m pytest pyarrow/tests/test_dataset.py
```
</details>
There was an error in `test_dataset.py`, so I checked whether the submodule was up to date.
```bash
git submodule status
74278bc4a1122d74945969e6dec405abd1533ec3 ../cpp/submodules/parquet-testing (heads/master)
735ae7128d571398dd798d7ff004adebeb342883 ../testing (remotes/origin/HEAD)
```
Upon further inspection, it turns out that the error occurs in the tests for the `test_make_write_options_error` function.
<details><summary>run test without test_make_write_options_error</summary>
```bash
python -m pytest pyarrow/tests/test_dataset.py -k "not test_make_write_options_error"
============================================================================================================ test session starts =============================================================================================================
platform darwin -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /Users/lama/workspace/arrow-new/python
configfile: setup.cfg
plugins: hypothesis-6.99.9
collected 351 items / 1 deselected / 350 selected
pyarrow/tests/test_dataset.py ...............................................................................................................s........................................................................................ [ 57%]
...........ssssss....................ssss.s.................................................................................ss........................ [100%]
=============================================================================================== 336 passed, 14 skipped, 1 deselected in 17.20s ===============================================================================================
```
</details>
If I exclude that test, all other tests pass successfully.
I use an M1 Macbook.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2067778929
Hmm. Does the https://github.com/apache/arrow/pull/40976/files#diff-4e3747424164e360600eab8972114148de49d999bff7b61157c0e535752d87aeR202-R204 check not working on the environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2067704305
It is related to the [PR 40976](https://github.com/apache/arrow/pull/40976), and in the case of CI, the unit test seems to [pass for macOS 12](https://github.com/apache/arrow/actions/runs/8558433247/job/23452987925?pr=40976#step:6:137), but I'm using 14.4, so I guess it fails...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2067990519
I tested only the `test_make_write_options_error` as a test case in CI, and there were no issues.
- https://github.com/apache/arrow/actions/runs/8771799647/job/24070017215?pr=41313#step:6:81
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2118573569
Please take a look at this [part](https://arrow.apache.org/cookbook/py/io.html#writing-partitioned-datasets). Upon examining example codes, you can see that the `dataset` is used as follows, and this syntax works correctly: `import pyarrow.dataset as ds`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2118545631
In my case, I had set up the environment to build PyArrow using cpp code. However, I found that using the code you provided still resulted in the same symptoms. 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2067685938
I attached lldb backtrace result.
<details><summary>lldb result</summary>
```
lldb -- python -m pytest pyarrow/tests/test_dataset.py -k "test_make_write_options_error"
(lldb) target create "python"
Current executable set to '/Users/lama/anaconda3/envs/pyarrow-dev-310/bin/python' (arm64).
(lldb) settings set -- target.run-args "-m" "pytest" "pyarrow/tests/test_dataset.py" "-k" "test_make_write_options_error"
(lldb) run
Process 72156 launched: '/Users/lama/anaconda3/envs/pyarrow-dev-310/bin/python' (arm64)
============================= test session starts ==============================
platform darwin -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /Users/lama/workspace/arrow-new/python
configfile: setup.cfg
plugins: hypothesis-6.99.9
collected 351 items / 350 deselected / 1 selected
Process 72156 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x1)
frame #0: 0x0000000106533b78 _dataset.cpython-310-darwin.so`__pyx_pw_7pyarrow_8_dataset_10FileFormat_7make_write_options(_object*, _object* const*, long, _object*) + 56
_dataset.cpython-310-darwin.so`__pyx_pw_7pyarrow_8_dataset_10FileFormat_7make_write_options:
-> 0x106533b78 <+56>: ldr x8, [x0]
0x106533b7c <+60>: ldr x9, [x8, #0x60]
0x106533b80 <+64>: add x8, sp, #0x30
0x106533b84 <+68>: blr x9
Target 0: (python) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x1)
* frame #0: 0x0000000106533b78 _dataset.cpython-310-darwin.so`__pyx_pw_7pyarrow_8_dataset_10FileFormat_7make_write_options(_object*, _object* const*, long, _object*) + 56
frame #1: 0x0000000126b6e4d8 _dataset_parquet.cpython-310-darwin.so`__pyx_pw_7pyarrow_16_dataset_parquet_17ParquetFileFormat_3make_write_options(_object*, _object* const*, long, _object*) + 260
frame #2: 0x0000000100149540 python`call_function + 524
frame #3: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #4: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #5: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
frame #6: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #7: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
frame #8: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #9: 0x0000000100149540 python`call_function + 524
frame #10: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #11: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #12: 0x000000010005ac64 python`method_vectorcall + 172
frame #13: 0x0000000100149540 python`call_function + 524
frame #14: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #15: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #16: 0x00000001000577ac python`_PyObject_FastCallDictTstate + 156
frame #17: 0x0000000100058454 python`_PyObject_Call_Prepend + 176
frame #18: 0x00000001000cdee0 python`slot_tp_call + 116
frame #19: 0x000000010005759c python`_PyObject_MakeTpCall + 612
frame #20: 0x00000001001495d8 python`call_function + 676
frame #21: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
frame #22: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #23: 0x0000000100149540 python`call_function + 524
frame #24: 0x0000000100144e38 python`_PyEval_EvalFrameDefault + 24900
frame #25: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #26: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
frame #27: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #28: 0x0000000100149540 python`call_function + 524
frame #29: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #30: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #31: 0x000000010005ac64 python`method_vectorcall + 172
frame #32: 0x0000000100149540 python`call_function + 524
frame #33: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #34: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #35: 0x00000001000577ac python`_PyObject_FastCallDictTstate + 156
frame #36: 0x0000000100058454 python`_PyObject_Call_Prepend + 176
frame #37: 0x00000001000cdee0 python`slot_tp_call + 116
frame #38: 0x0000000100057fb0 python`_PyObject_Call + 148
frame #39: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
frame #40: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #41: 0x0000000100149540 python`call_function + 524
frame #42: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
frame #43: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #44: 0x000000010005ac64 python`method_vectorcall + 172
frame #45: 0x0000000100149540 python`call_function + 524
frame #46: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
frame #47: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #48: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
frame #49: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #50: 0x0000000100149540 python`call_function + 524
frame #51: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
frame #52: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #53: 0x0000000100149540 python`call_function + 524
frame #54: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
frame #55: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #56: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
frame #57: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #58: 0x0000000100149540 python`call_function + 524
frame #59: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #60: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #61: 0x000000010005ac64 python`method_vectorcall + 172
frame #62: 0x0000000100149540 python`call_function + 524
frame #63: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #64: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #65: 0x00000001000577ac python`_PyObject_FastCallDictTstate + 156
frame #66: 0x0000000100058454 python`_PyObject_Call_Prepend + 176
frame #67: 0x00000001000cdee0 python`slot_tp_call + 116
frame #68: 0x000000010005759c python`_PyObject_MakeTpCall + 612
frame #69: 0x00000001001495d8 python`call_function + 676
frame #70: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
frame #71: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #72: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
frame #73: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #74: 0x0000000100149540 python`call_function + 524
frame #75: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #76: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #77: 0x000000010005ac64 python`method_vectorcall + 172
frame #78: 0x0000000100149540 python`call_function + 524
frame #79: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #80: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #81: 0x00000001000577ac python`_PyObject_FastCallDictTstate + 156
frame #82: 0x0000000100058454 python`_PyObject_Call_Prepend + 176
frame #83: 0x00000001000cdee0 python`slot_tp_call + 116
frame #84: 0x000000010005759c python`_PyObject_MakeTpCall + 612
frame #85: 0x00000001001495d8 python`call_function + 676
frame #86: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
frame #87: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #88: 0x0000000100149540 python`call_function + 524
frame #89: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
frame #90: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #91: 0x0000000100149540 python`call_function + 524
frame #92: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
frame #93: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #94: 0x0000000100145658 python`_PyEval_EvalFrameDefault + 26980
frame #95: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #96: 0x0000000100149540 python`call_function + 524
frame #97: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #98: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #99: 0x000000010005ac64 python`method_vectorcall + 172
frame #100: 0x0000000100149540 python`call_function + 524
frame #101: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #102: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #103: 0x00000001000577ac python`_PyObject_FastCallDictTstate + 156
frame #104: 0x0000000100058454 python`_PyObject_Call_Prepend + 176
frame #105: 0x00000001000cdee0 python`slot_tp_call + 116
frame #106: 0x000000010005759c python`_PyObject_MakeTpCall + 612
frame #107: 0x00000001001495d8 python`call_function + 676
frame #108: 0x0000000100145430 python`_PyEval_EvalFrameDefault + 26428
frame #109: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #110: 0x0000000100149540 python`call_function + 524
frame #111: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
frame #112: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #113: 0x0000000100149540 python`call_function + 524
frame #114: 0x0000000100144e60 python`_PyEval_EvalFrameDefault + 24940
frame #115: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #116: 0x000000010013db40 python`PyEval_EvalCode + 120
frame #117: 0x0000000100139760 python`builtin_exec + 836
frame #118: 0x00000001000ab880 python`cfunction_vectorcall_FASTCALL + 200
frame #119: 0x0000000100149540 python`call_function + 524
frame #120: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
frame #121: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #122: 0x0000000100149540 python`call_function + 524
frame #123: 0x00000001001453bc python`_PyEval_EvalFrameDefault + 26312
frame #124: 0x000000010013e364 python`_PyEval_Vector + 2036
frame #125: 0x00000001001bcfc4 python`pymain_run_module + 272
frame #126: 0x00000001001bc604 python`Py_RunMain + 1584
frame #127: 0x00000001001bda54 python`pymain_main + 1180
frame #128: 0x000000010000131c python`main + 56
frame #129: 0x000000019b2760e0 dyld`start + 2360
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068575753
I tried reproducing on my M1 with Python 3.11 but no luck. The tests is run, not skipped, and it passes:
```python
(pyarrow-dev) alenkafrim@Alenkas-MacBook-Pro arrow % python -m pytest python/pyarrow/tests/test_dataset.py -k test_make_write_options_error
=================================================================== test session starts ===================================================================
platform darwin -- Python 3.11.7, pytest-7.4.4, pluggy-1.4.0
rootdir: /Users/alenkafrim/repos/arrow/python
configfile: setup.cfg
plugins: hypothesis-6.99.5, cython-0.2.1
collected 351 items / 350 deselected / 1 selected
python/pyarrow/tests/test_dataset.py . [100%]
============================================================ 1 passed, 350 deselected in 0.56s ============================================================
```
I will also try with Python 3.10 later. I am currently at the conference so might take some time.
I suspect though that something might be off with the setup? Could you try a clean new build (`git clean -Xfd .` in `arrow/python` or maybe even in `arrow/`), also cleaning `dist/` folder (https://arrow.apache.org/docs/dev/developers/python.html#using-system-and-bundled-dependencies)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068312242
@AlenkaF Could you take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068024528
I checked a few things:
I was using Python `3.10` on my environment, but the CI system was using Python `3.11`. So, I set up Python `3.11` with conda and ran the tests again.
I found that some tests in `pyarrow/tests/test_dataset.py` were skipped when I tested. But it can be executed with `export PYARROW_TEST_DATASET=ON`.
Even though I built it by referring to the contents of ci , an error appears...
- https://github.com/apache/arrow/blob/16e20b712ab8dafa9d1ec6154415fe5748fe6135/.github/workflows/python.yml#L137-L157
#### python 3.11
<details><summary>test result</summary>
```bash
(pyarrow-dev-311) python -m pytest -k "test_make_write_options_error"
========================================================================================================== test session starts ===========================================================================================================
platform darwin -- Python 3.11.9, pytest-8.1.1, pluggy-1.5.0
rootdir: /Users/lama/workspace/arrow-new/python
configfile: setup.cfg
plugins: hypothesis-6.100.1
collected 7667 items / 7666 deselected / 4 skipped / 1 selected
pyarrow/tests/test_dataset.py F [100%]
================================================================================================================ FAILURES ================================================================================================================
_____________________________________________________________________________________________________ test_make_write_options_error ______________________________________________________________________________________________________
def test_make_write_options_error():
# GH-39440: calling make_write_options as a static class method
msg_1 = ("make_write_options() should be called on an "
"instance of ParquetFileFormat")
# GH-41043: In Cython2 all Cython methods were "regular" C extension methods
# see: https://github.com/cython/cython/issues/6127#issuecomment-2038153359
msg_2 = ("descriptor 'make_write_options' for "
"'pyarrow._dataset_parquet.ParquetFileFormat' objects "
"doesn't apply to a 'int'")
with pytest.raises(TypeError) as excinfo:
> pa.dataset.ParquetFileFormat.make_write_options(43)
E AttributeError: module 'pyarrow' has no attribute 'dataset'
pyarrow/tests/test_dataset.py:5645: AttributeError
======================================================================================================== short test summary info =========================================================================================================
FAILED pyarrow/tests/test_dataset.py::test_make_write_options_error - AttributeError: module 'pyarrow' has no attribute 'dataset'
============================================================================================= 1 failed, 4 skipped, 7666 deselected in 1.05s ==============================================================================================
```
</details>
#### Python 3.10
<details><summary>test result</summary>
```
(pyarrow-dev-310) python -m pytest -k "test_make_write_options_error"
============================================================================================================ test session starts =============================================================================================================
platform darwin -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0
rootdir: /Users/lama/workspace/arrow-new/python
configfile: setup.cfg
plugins: hypothesis-6.99.9
collected 7667 items / 7666 deselected / 4 skipped / 1 selected
pyarrow/tests/test_dataset.py Fatal Python error: Segmentation fault
Current thread 0x000000020330bac0 (most recent call first):
File "/Users/lama/workspace/arrow-new/python/pyarrow/tests/test_dataset.py", line 5645 in test_make_write_options_error
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pytest/__main__.py", line 5 in <module>
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/runpy.py", line 86 in _run_code
File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/runpy.py", line 196 in _run_module_as_main
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pyarrow._fs, cython.cimports.libc.math, pyarrow._compute, pyarrow._acero, pyarrow._csv, pyarrow._json, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_
dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._dataset, pyarrow._parquet, pyarrow._dataset_parquet, _cffi_backend, pyarrow._pyarrow_cpp_tests, pyarrow._feather (total: 66)
[1] 77852 segmentation fault python -m pytest -k "test_make_write_options_error"
```
</details>
Additionally, by enabling detailed output for the tests, it has been confirmed that the test is being passed when triggered.
- `tests/test_dataset.py::test_make_write_options_error <- ../../../../../Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyarrow/tests/test_dataset.py PASSED
`
- https://github.com/apache/arrow/actions/runs/8772308641/job/24071088984?pr=41313#step:6:90
- https://github.com/apache/arrow/actions/runs/8772308641/job/24071089064?pr=41313#step:6:90
## To summarize
- The test appears not to have been run even though `PYARROW_TEST_DATASET` is `ON`.
- Why are tests treated as passed in CI?
- If running the test locally, different errors appear depending on the python version.
- 3.10: segmentation fault
- 3.11: AttributeError: module 'pyarrow' has no attribute 'dataset'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 closed issue #41312: [Python] Broken unit test: Segmentation fault in test_make_write_options_error
URL: https://github.com/apache/arrow/issues/41312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "martibosch (via GitHub)" <gi...@apache.org>.
martibosch commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2117762625
I cannot manage to use pyarrow on my ubuntu computer using conda. I have tried creating conda-forge environments with pyarrow (16.0.0) and also creating the environment with only python 3.10 and 3.11 and then installing pyarrow (16.1.0) via pip, and I always get `AttributeError: module 'pyarrow' has no attribute 'dataset'`. This can be reproduced as in:
```
conda env create -n pyarrow -c conda-forge python=3.11 pyarrow=16
conda activate pyarrow
python -c 'import pyarrow as pa;pa.dataset'
```
or:
```
conda env create -n pyarrow -c conda-forge python=3.11
conda activate pyarrow
pip install pyarrow==16.1.0
python -c 'import pyarrow as pa;pa.dataset'
```
How can I fix this? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2067818479
Yes, it looks like it was supposed to check there, but it's not working as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068859092
Thanks for trying it out and confirming! Maybe it would be worth testing with new conda env also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Broken unit test: Segmentation fault in test_make_write_options_error [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #41312:
URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068658290
Hello @AlenkaF
I followed the suggested documentation and cleaned all builds, then created a Python virtual environment using `python3 -m venv` instead of conda. I confirmed this resolved the issue.
```bash
python -m pytest python/pyarrow/tests/test_dataset.py -k test_make_write_options_error
========================================================================================================== test session starts ===========================================================================================================
platform darwin -- Python 3.11.9, pytest-8.1.1, pluggy-1.5.0
rootdir: /Users/lama/workspace/arrow-new/python
configfile: setup.cfg
plugins: hypothesis-6.100.1
collected 351 items / 350 deselected / 1 selected
python/pyarrow/tests/test_dataset.py . [100%]
=================================================================================================== 1 passed, 350 deselected in 0.64s ====================================================================================================
```
I still don't fully understand why it doesn't work with conda, but I gather it's due to an environmental issue.
Thank you for helping me check!
cc @kou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org