You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "h-vetinari (via GitHub)" <gi...@apache.org> on 2023/06/11 01:52:00 UTC
[GitHub] [arrow] h-vetinari opened a new issue, #36026: BUG: test_orc_format segfaults on windows
h-vetinari opened a new issue, #36026:
URL: https://github.com/apache/arrow/issues/36026
### Describe the bug, including details regarding any error messages, version, and platform.
orc in conda-forge doesn't exist for a long time yet, and we've only recently enabled it for the C++ portion of arrow. I tried to switch it on for pyarrow now as well in https://github.com/conda-forge/arrow-cpp-feedstock/pull/1086, and the test suite segfaults as soon as it gets to [`test_dataset.py::test_orc_format`](https://github.com/apache/arrow/blob/apache-arrow-12.0.0/python/pyarrow/tests/test_dataset.py#L2980)
<details>
<summary>stacktrace</summary>
```
[...]
test_dataset.py::test_ipc_format[threaded] PASSED [ 20%]
test_dataset.py::test_ipc_format[serial] PASSED [ 20%]
Fatal Python error: Aborted
Thread 0x00000ea0 (most recent call first):
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pyarrow\tests\test_dataset.py", line 261 in to_table
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pyarrow\tests\test_dataset.py", line 2991 in test_orc_format
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\python.py", line 194 in pytest_pyfunc_call
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\python.py", line 1799 in runtest
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\runner.py", line 169 in pytest_runtest_call
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\runner.py", line 262 in <lambda>
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\runner.py", line 341 in from_call
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\runner.py", line 261 in call_runtest_hook
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\runner.py", line 222 in call_and_report
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\runner.py", line 133 in runtestprotocol
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\runner.py", line 114 in pytest_runtest_protocol
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\main.py", line 348 in pytest_runtestloop
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\main.py", line 323 in _main
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\main.py", line 269 in wrap_session
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\config\__init__.py", line 166 in main
File "D:\bld\apache-arrow_1686428319811\_test_env\Lib\site-packages\_pytest\config\__init__.py", line 189 in console_main
File "D:\bld\apache-arrow_1686428319811\_test_env\Scripts\pytest-script.py", line 9 in <module>
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pyarrow._hdfsio, pyarrow._fs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.miss
ing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.hashing, pandas._libs.tslib, pandas._libs.ops, pyarrow._compute, pandas._libs.arrays, pandas._libs.sparse, pandas._libs.reduction, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.testing, pandas._libs.parsers, pandas._libs.json, fastparquet.cencoding, fastparquet.speedups, pyarrow.gandiva, pyarrow._acero, pyarrow._csv, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet, pyarrow._dataset_parquet, pyarrow._orc, pyarrow._parquet_encryption, pyarrow._flight, pyarrow._substrait, _cffi_backend, pyarrow._pyarrow_cpp_tests, pyarrow._feather, pyarrow._json, numpy.linalg.lapack_lite, scipy._lib._ccallback_c, scipy.sparse._sparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.
linalg._fblas, scipy.linalg._flapack, scipy.linalg._cythonized_array_utils, scipy.linalg._flinalg, scipy.linalg._solve_toeplitz, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_lapack, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, pyarrow_cython_example, bound_function_visit_strings (total: 104)
Tests failed for pyarrow-tests-12.0.0-py311h385a57a_8_cpu.conda - moving package to D:\bld\broken
WARNING:conda_build.build:Tests failed for pyarrow-tests-12.0.0-py311h385a57a_8_cpu.conda - moving package to D:\bld\broken
TESTS FAILED: pyarrow-tests-12.0.0-py311h385a57a_8_cpu.conda
```
</details>
### Component(s)
Format, Packaging, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2006107287
For reference, the previous PR enabling ORC on Windows in some builds also setting that env variable to get it working .. https://github.com/apache/arrow/pull/14208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1998784075
Could you try `set CMAKE_PREFIX_PATH=%ARROW_HOME%` or `set PYARROW_CMAKE_OPTIONS=-DCMAKE_PREFIX_PATH=%ARROW_HOME%`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2009552173
Issue resolved by pull request 40609
https://github.com/apache/arrow/pull/40609
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2009691129
> Right, can the crash be averted? If a C++ exception is raised, then it can be caught and turned into a proper error.
If we can upgrade to ORC 2.0.0, then it will throw if the file path does not exist: https://github.com/apache/orc/blob/main/c%2B%2B/src/Timezone.cc#L678-L682
However, it seems that the current version (1.9.2) simply crashes when open is called: https://github.com/apache/orc/blob/0b4631bc343ed559e284ea6f824f31890f798e31/c%2B%2B/src/OrcFile.cc#L58-L68
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows (conda-forge)
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1697519754
Sorry, it seems I was missing the following to test orc on the Wheels job: https://github.com/apache/arrow/pull/37382/commits/ce4fe1cdcc4f688275e08ba640c1a77f0ba30cb2
I've been able to reproduce the failure on the wheels jobs on [this PR](https://github.com/apache/arrow/pull/37382), the failed jobs here:
* [Python 3.8 wheel](https://github.com/ursacomputing/crossbow/actions/runs/6010525146/job/16302119137)
* [Python 3.9 wheel](https://github.com/ursacomputing/crossbow/actions/runs/6010526204/job/16302121771)
* [Python 3.10 wheel](https://github.com/ursacomputing/crossbow/actions/runs/6010526976/job/16302123921)
* [Python 3.11 wheel](https://github.com/ursacomputing/crossbow/actions/runs/6010525477/job/16302119626)
<details>
<summary>stacktrace</summary>
```
Python311\Lib\site-packages\pyarrow\tests\test_dataset.py .............. [ 24%]
........................................................................ [ 24%]
Fatal Python error: Aborted
Thread 0x00001ab0 (most recent call first):
File "C:\Python311\Lib\site-packages\pyarrow\tests\test_dataset.py", line 261 in to_table
File "C:\Python311\Lib\site-packages\pyarrow\tests\test_dataset.py", line 3072 in test_orc_format
File "C:\Python311\Lib\site-packages\_pytest\python.py", line 194 in pytest_pyfunc_call
File "C:\Python311\Lib\site-packages\pluggy\_callers.py", line 77 in _multicall
File "C:\Python311\Lib\site-packages\pluggy\_manager.py", line 115 in _hookexec
File "C:\Python311\Lib\site-packages\pluggy\_hooks.py", line 493 in __call__
File "C:\Python311\Lib\site-packages\_pytest\python.py", line 1788 in runtest
File "C:\Python311\Lib\site-packages\_pytest\runner.py", line 169 in pytest_runtest_call
File "C:\Python311\Lib\site-packages\pluggy\_callers.py", line 77 in _multicall
File "C:\Python311\Lib\site-packages\pluggy\_manager.py", line 115 in _hookexec
File "C:\Python311\Lib\site-packages\pluggy\_hooks.py", line 493 in __call__
File "C:\Python311\Lib\site-packages\_pytest\runner.py", line 262 in <lambda>
File "C:\Python311\Lib\site-packages\_pytest\runner.py", line 341 in from_call
File "C:\Python311\Lib\site-packages\_pytest\runner.py", line 261 in call_runtest_hook
File "C:\Python311\Lib\site-packages\_pytest\runner.py", line 222 in call_and_report
File "C:\Python311\Lib\site-packages\_pytest\runner.py", line 133 in runtestprotocol
File "C:\Python311\Lib\site-packages\_pytest\runner.py", line 114 in pytest_runtest_protocol
File "C:\Python311\Lib\site-packages\pluggy\_callers.py", line 77 in _multicall
File "C:\Python311\Lib\site-packages\pluggy\_manager.py", line 115 in _hookexec
File "C:\Python311\Lib\site-packages\pluggy\_hooks.py", line 493 in __call__
File "C:\Python311\Lib\site-packages\_pytest\main.py", line 349 in pytest_runtestloop
File "C:\Python311\Lib\site-packages\pluggy\_callers.py", line 77 in _multicall
File "C:\Python311\Lib\site-packages\pluggy\_manager.py", line 115 in _hookexec
File "C:\Python311\Lib\site-packages\pluggy\_hooks.py", line 493 in __call__
File "C:\Python311\Lib\site-packages\_pytest\main.py", line 324 in _main
File "C:\Python311\Lib\site-packages\_pytest\main.py", line 270 in wrap_session
File "C:\Python311\Lib\site-packages\_pytest\main.py", line 317 in pytest_cmdline_main
File "C:\Python311\Lib\site-packages\pluggy\_callers.py", line 77 in _multicall
File "C:\Python311\Lib\site-packages\pluggy\_manager.py", line 115 in _hookexec
File "C:\Python311\Lib\site-packages\pluggy\_hooks.py", line 493 in __call__
File "C:\Python311\Lib\site-packages\_pytest\config\__init__.py", line 166 in main
File "C:\Python311\Lib\site-packages\_pytest\config\__init__.py", line 189 in console_main
File "C:\Python311\Scripts\pytest.exe\__main__.py", line 7 in <module>
File "<frozen runpy>", line 88 in _run_code
File "<frozen runpy>", line 198 in _run_module_as_main
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pyarrow._hdfsio, pyarrow._fs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, cython.cimports.libc.math, pyarrow._compute, pyarrow._acero, pyarrow._csv, pyarrow._json, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.t
slibs.period, pandas._libs.ts...
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1616695811
> @wgtmac may this related to C++ Orc release?
I don't think so. It seems that this issue exists in the old 1.8.x releases as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed issue #36026: [Python] BUG: test_orc_format segfaults on windows
URL: https://github.com/apache/arrow/issues/36026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1997904079
I tried to reproduce it on my Windows PC following the guide https://arrow.apache.org/docs/developers/python.html#building-on-windows
Following commands were executed successfully and arrow cpp library was installed.
```
"E:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\VsDevCmd.bat"
conda create -y -n pyarrow-dev -c conda-forge ^
--file arrow\ci\conda_env_cpp.txt ^
--file arrow\ci\conda_env_python.txt ^
--file arrow\ci\conda_env_gandiva.txt ^
python=3.12
conda activate pyarrow-dev
pushd arrow\cpp\build
set ARROW_HOME=%CONDA_PREFIX%\Library
cmake -G "Visual Studio 17 2022" -A x64 ^
-DCMAKE_INSTALL_PREFIX=%ARROW_HOME% ^
-DCMAKE_UNITY_BUILD=ON ^
-DARROW_CXXFLAGS="/WX /MP" ^
-DARROW_ORC=ON ^
..
cmake --build . --target install --config Release
popd
```
However, when I tried to build pyarrow as below,
```
pushd arrow\python
set CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1
set PYARROW_CMAKE_GENERATOR=Visual Studio 17 2022
set PYARROW_CXXFLAGS="-A x64"
set PYARROW_WITH_ORC=1
python setup.py build_ext --inplace
popd
```
it complained for missing LZ4:
```
-- Found Protobuf: C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/lib/libprotobuf.lib (found version "3.21.12")
-- Found ZLIB: C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/lib/z.lib (found version "1.2.13")
CMake Warning at C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/lib/cmake/Arrow/Findlz4Alt.cmake:29 (find_package):
By not providing "Findlz4.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "lz4", but
CMake did not find one.
Could not find a package configuration file provided by "lz4" with any of
the following names:
lz4Config.cmake
lz4-config.cmake
Add the installation prefix of "lz4" to CMAKE_PREFIX_PATH or set "lz4_DIR"
to a directory containing one of the above files. If "lz4" provides a
separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/share/cmake-3.28/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/lib/cmake/Arrow/ArrowConfig.cmake:99 (find_dependency)
C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/lib/cmake/Arrow/ArrowConfig.cmake:122 (arrow_find_dependencies)
CMakeLists.txt:271 (find_package)
-- Checking for module 'liblz4'
-- No package 'liblz4' found
CMake Error at C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find lz4Alt (missing: LZ4_LIB)
Call Stack (most recent call first):
C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/lib/cmake/Arrow/Findlz4Alt.cmake:97 (find_package_handle_standard_args)
C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/share/cmake-3.28/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/lib/cmake/Arrow/ArrowConfig.cmake:99 (find_dependency)
C:/Users/wgtmac/.conda/envs/pyarrow-dev/Library/lib/cmake/Arrow/ArrowConfig.cmake:122 (arrow_find_dependencies)
CMakeLists.txt:271 (find_package)
-- Configuring incomplete, errors occurred!
```
I can confirm that the file `C:\Users\wgtmac\.conda\envs\pyarrow-dev\Library\lib\liblz4.lib` exists. And I tried to explicitly disable ARROW_WITH_LZ4 but it seems to be turned on anyway.
Do you have any idea? @h-vetinari @raulcd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2001883527
Hmm. Does `C:\Users\wgtmac\.conda\envs\pyarrow-dev\Library\lib\cmake\Arrow\Findlz4Alt.cmake` exist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2005841790
> set `TZDIR=%CONDA_PREFIX%\share\zoneinfo`
I can confirm that setting `TZDIR` makes pyarrow built with `PYARROW_WITH_ORC=1` pass the test suite also on windows. 🥳
Thanks a lot for debugging this, great to have this finally sorted out!
How should we fix this though? I see that in #40609 you're downloading the tzdb, but that's not really viable for us in conda-forge. It would be good if pyarrow could automatically check `%CONDA_PREFIX%\share\zoneinfo` when looking for the tzdb (relative to the site-packages directory on windows, the path would be `../../share/zoneinfo`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2006340788
Yes, I think we can regard ORC java and C++ independently here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2006355564
Any suggestion for the TZDB search path candidates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1834722595
Segfault on windows remains as of v14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1998805686
> Could you try `set CMAKE_PREFIX_PATH=%ARROW_HOME%` or `set PYARROW_CMAKE_OPTIONS=-DCMAKE_PREFIX_PATH=%ARROW_HOME%`?
Thanks! I will try that after work when I have access to my Windows PC.
BTW, do you know why the pure cpp build/test could succeed with `ARROW_WITH_LZ4=ON` but pyarrow failed? @kou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2006344233
For conda, I agree with @h-vetinari that it would indeed be nice if ORC could update the search path to find this automatically. In that case, we could add `tzdata` as a dependency to ensure those files are always available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2006087340
Setting the env variable in our testing is not really a "proper" fix, though? While that fixes it on our side, IMO we can't expect every user to have to set this env variable to avoid getting segfaults?
Do we know why it is segfaulting when that env variable is not set (when the timezone database is not set)? Is that something in the ORC library itself that would need to be fixed? (I would expect that we can turn that into an informative error instead of a segfault)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2009694654
> However, it seems that the current version (1.9.2) simply crashes when open is called: https://github.com/apache/orc/blob/0b4631bc343ed559e284ea6f824f31890f798e31/c%2B%2B/src/OrcFile.cc#L58-L68
Well, it throws a C++ exception (`ParseError`), right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1627377249
I am not familiar with python or pyarrow. Does it mean I can simply run the failed test on Windows to reproduce this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1627634353
Hey @wgtmac, thanks for taking a look! It's from a pretty "standard" build, though arrow & pyarrow are quite a handful, especially considering all the dependencies. I could publish the failing builds in a way that they're (opt-in) installable using conda/mamba/miniforge, would that help?
Otherwise, you could just try building arrow & pyarrow with `-DARROW_ORC=ON` and `export PYARROW_WITH_ORC=1` on windows (and then run the python test suite), and see whether the problem is reproducible like that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1691519378
Still segfaulting as of 13.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1693270412
I've enabled it for Appveyor and I was **not** able to reproduce the issue, see the successful build here: https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/47879511
I am trying to enable it on the Windows wheels too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2046523806
> Like I said, the right approach IMO is to search a couple of reasonable candidate paths for the TZDB, including `$PREFIX/share/zoneinfo` (also on windows).
I tried to do this, but with recent changes that guard the check on `ARROW_ORC_NEED_TIME_ZONE_DATABASE_CHECK` (only for orc <2), this cannot really be done in arrow. So I've opened a PR to orc instead: https://github.com/apache/orc/pull/1882
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2009147537
Thanks for the suggestion!
> allow to specify a path at run time
This sounds like a good alternative to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1997975386
I don't recall seeing a report about such a build issue.
We do have a `Findlz4Alt.cmake` file
> And I tried to explicitly disable ARROW_WITH_LZ4 but it seems to be turned on anyway.
Yeah, if `ARROW_ORC` is enabled, it will always turn on `ARROW_WITH_LZ4` as well:
https://github.com/apache/arrow/blob/e0f2bd2fd46002267293217b8c8db61c8cddd558/cpp/cmake_modules/DefineOptions.cmake#L381-L388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1996340077
Still segfaulting also with orc 2.0.0 (arrow 15.0.1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2006118871
I've improved the error message on the ORC side via this PR: https://github.com/apache/orc/pull/1587. It would be good to use TZDB on the local host when available. As an alternative, the ORC library could ship IANA TZDB as a bundled third-party library and uses it as a fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2006333274
> Unfortunately, the ORC 2.0.0 upgrade is blocked by dropping Java 8 support on the ORC side: #40508.
But for the Python wheels for Windows, if we want to enable ORC in those packages (https://github.com/apache/arrow/issues/37381), that's a separate build toolchain and there we could already update ORC? (as long as it is updated in vcpkg)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2006148606
Ah, that's good to hear! And I also see that ORC just released last week, although I assume that is a bit too recent to already upgrade our dependency here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2020716980
Issue resolved by pull request 40697
https://github.com/apache/arrow/pull/40697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #36026: [Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set)
URL: https://github.com/apache/arrow/issues/36026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2009560673
As I mentioned above (https://github.com/apache/arrow/issues/36026#issuecomment-2006087340), the PR didn't really fix this issue, it just hides it in our own tests, but users still get a segfault.
For 16.0 (and until ORC fixes this), we could also add a check on our side? Eg, if on windows, check that the env variable is set / the path is available, and if not raise an informative error to the user instead of segfaulting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2009673313
Right, can the crash be averted? If a C++ exception is raised, then it can be caught and turned into a proper error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1999921928
@kou I tried all your suggestions with additional guesses, but all failed with missing lz4. I checked `C:\Users\wgtmac\.conda\envs\pyarrow-dev\Library\lib\cmake\Arrow\ArrowConfig.cmake` which contains lines below
```
set(ARROW_BUNDLED_STATIC_LIBS "")
set(ARROW_INCLUDE_PATH_SUFFIXES "include;Library;Library/include")
set(ARROW_LIBRARY_PATH_SUFFIXES ";lib/;lib64;lib32;lib;bin;Library;Library/lib;Library/bin")
set(ARROW_SYSTEM_DEPENDENCIES "SnappyAlt;ProtobufAlt;ZLIB;lz4Alt;zstdAlt;orcAlt")
```
After removing `lz4Alt` from `ARROW_SYSTEM_DEPENDENCIES`, the pyarrow build proceeded to compiling and linking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2002322269
I'm now stuck at linking (same result at the master branch and 13.0.0 release tag, which I chose randomly)
```
Creating library E:/projects/github/arrow/python/build/temp.win32-cpython-312/Release/arrow_python.lib and object E:/projects/github/arrow/python/build/temp.win32-cpython-312/Release/arrow_python.exp
gdb.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) class arrow::Result<class std::shared_ptr<class arrow::Array> >
__cdecl arrow::ipc::internal::json::ArrayFromJSON(class std::shared_ptr<class arrow::DataType> const &,class std::basic_string_view<char ,struct std::char_traits<char> >)" (__imp_?ArrayFromJSON@json@internal@ipc@arrow@@YA?AV?$Result@V?$shared_ptr@VArray@arrow@@@std@ @@4@AEBV?$shared_ptr@VDataType@arrow@@@std@@V?$basic_string_view@DU?$char_traits@D@std@@@7@@Z)，referenced in function "class std::shared_ptr<classarrow::Array> __cdecl arrow::gdb::`anonymous namespace'::SliceArrayFromJSON(class std::shared_ptr<class arrow::DataType> const &,cl ass std::basic_string_view<char,struct std::char_traits<char> >,__int64,__int64)" (?SliceArrayFromJSON@?A0x9bed347f@gdb@arrow@@YA ?AV?$shared_ptr@VArray@arrow@@@std@@AEBV?$shared_ptr@VDataType@arrow@@@5@V?$basic_string_view@DU?$char_traits@D@std@@@5@_J2@Z) [E:\projects\github\arrow\python\build\temp.win32-cpython-312\arrow_python.vcxproj]
gdb.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) class arrow::Result<class std::shared_ptr<class arrow::Array> >
__cdecl arrow::ipc::internal::json::ArrayFromJSON(class std::shared_ptr<class arrow::DataType> const &,char const *)" (__imp_?ArrayFromJ SON@json@internal@ipc@arrow@@YA?AV?$Result@V?$shared_ptr@VArray@arrow@@@std@@@4@AEBV?$shared_ptr@VDataType@arrow@@@std@@PEBD@Z)，referenced in function "void __cdecl arrow::gdb::TestSession(void)" (?TestSession@gdb@arrow@@YAXXZ) [E:\projects\github\arrow\python\build\temp.win32-cpython-312\arrow_python.vcxproj]
gdb.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) class arrow::Status __cdecl arrow::ipc::internal::json::ChunkedArrayFromJSON(class std::shared_ptr<class arrow::DataType> const &,class std::vector<class std::basic_string<char,struct std::char_traits <char>,class std::allocator<char> >,class std::allocator<class std::basic_string<char,struct std::char_traits<char>,class std::al locator<char> > > > const &,class std::shared_ptr<class arrow::ChunkedArray> *)" (__imp_?ChunkedArrayFromJSON@json@internal@ipc@a rrow@@YA?AVStatus@4@AEBV?$shared_ptr@VDataType@arrow@@@std@@AEBV?$vector@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@ @std@@V?$allocator@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@2@@7@PEAV?$shared_ptr@VChunkedArray@arrow@@@7@@ Z)，referenced in function "void __cdecl arrow::gdb::TestSession(void)" (?TestSession@gdb@arrow@@YAXXZ) [E:\projects\github\arrow\python\build\temp.win32-cpython-312\arrow_python.vcxproj]
gdb.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) class arrow::Status __cdecl arrow::ipc::internal::json::ScalarFromJSON(class std::shared_ptr<class arrow::DataType> const &,class std::basic_string_view<char,struct std::char_traits<char> >,class std: :shared_ptr<struct arrow::Scalar> *)" (__imp_?ScalarFromJSON@json@internal@ipc@arrow@@YA?AVStatus@4@AEBV?$shared_ptr@VDataType@ar row@@@std@@V?$basic_string_view@DU?$char_traits@D@std@@@7@PEAV?$shared_ptr@UScalar@arrow@@@7@@Z)，referenced in function "void __cdecl arrow::gdb::TestSession(void)" (?TestSession@gdb@arrow@@YAXXZ) [E:\projects\github\arrow\python\build\temp.win32-cpython-312\arrow_python.vcxproj]
E:\projects\github\arrow\python\build\temp.win32-cpython-312\Release\arrow_python.dll : fatal error LNK1120: 4 unresolved externals [E:\projects\github\arrow\python\build\temp.win32-cpython-312\arrow_python.vcxproj]
Compiling Cython CXX source for _csv...
Building Custom Rule E:/projects/github/arrow/python/CMakeLists.txt
Compiling Cython CXX source for _feather...
Building Custom Rule E:/projects/github/arrow/python/CMakeLists.txt
Compiling Cython CXX source for _fs...
Building Custom Rule E:/projects/github/arrow/python/CMakeLists.txt
Compiling Cython CXX source for _hdfsio...
Building Custom Rule E:/projects/github/arrow/python/CMakeLists.txt
Compiling Cython CXX source for _json...
Building Custom Rule E:/projects/github/arrow/python/CMakeLists.txt
Compiling Cython CXX source for _orc...
Building Custom Rule E:/projects/github/arrow/python/CMakeLists.txt
Compiling Cython CXX source for _pyarrow_cpp_tests...
Building Custom Rule E:/projects/github/arrow/python/CMakeLists.txt
error: command 'C:\\Users\\wgtmac\\.conda\\envs\\pyarrow-dev\\Library\\bin\\cmake.exe' failed with exit code 1
```
If it is hard to make it work on my Windows PC, I will try to modify the CI script to debug this in a pull request. :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2007566504
Currently it is hardcoded (if the env variable is not set) to "/usr/share/zoneinfo" (https://github.com/apache/orc/blob/0b4631bc343ed559e284ea6f824f31890f798e31/c%2B%2B/src/Timezone.cc#L33-L34)
I am not super familiar about whether there are standard ways to get a "user" or "app" prefix (we use `$PREFIX` in our (conda) CI, but that's because we set that ourselves, not necessarily because that is a standard?).
For conda you can get this as `$CONDA_PREFIX`, but I don't know if you want to include conda specific code in the ORC library itself.
Another option would also be to allow to specify a path at run time (we do that in Arrow for the tz database that is used by the kernels with `arrow::GlobalOptions:;timezone_db_path`), and then in the (py)arrow code we could be more opinionated about which path to set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2005618169
Thanks for the debugging!
I started a CI run that sets `TZDIR`, see https://github.com/conda-forge/arrow-cpp-feedstock/pull/1086/commits/fd6a4f60ba8b78c1696bef2df6e789e82af6b4e2. Given that the build phase passed without issue, I did not set `TZDIR` in the build scripts, but I can do that as well if it helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2005608773
@h-vetinari Could you please try the setting below?
`set TZDIR=%CONDA_PREFIX%\share\zoneinfo`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2005845858
Good to know this works! My fix is for the python wheel build on Windows. BTW, I found this: https://github.com/apache/arrow/blob/0956f3f7c9f8d4f976275cd670744b52ee30cbf3/ci/appveyor-cpp-build.bat#L105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1998824680
`cpp/` build uses its `Findlz4Alt.cmake` (we don't need to set search path) but `python/` build uses `Findlz4Alt.cmake` installed by `cmake --install` of `cpp/` (we need to set search path).
(If my assumption is correct. :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1627645587
Thanks for the info! Let me spend some time to build it on Windows first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1691627072
@kou @jorisvandenbossche do you know if we are testing ORC enabled on Windows for Pyarrow somewhere else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1616835940
Definitely not related to 1.9; the segfault happened in the exact same place before for 1.8.3. I don't have data before that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2001884622
Hmm. I hope that the `Findlz4Alt.cmake` can be found by `set(CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}")` in `ArrowConfig.cmake` but it seems that it doesn't work...
Anyway, this is not related to ORC. We can ignore this by removing `lz4Alt` from `ARROW_SYSTEM_DEPENDENCIES`. `ARROW_SYSTEM_DEPENDENCIES` is only needed for static linking and PyArrow uses shared linking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2001883808
Yes, it does exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2006150451
> once C++20 is enforced.
I think that's still gonna take a while. LLVM's libc++ still doesn't have timezone support, and GCC's libstdcxx only gained support in the upcoming GCC 14 (completely aside from the fact that even the C++ implementation needs to load the TZDB).
The bundling in orc is not an elegant solution either - in conda-forge for example, we already have `tzdata.zi` available through [`tzdata`](https://conda-metadata-app.streamlit.app/?q=conda-forge%2Fnoarch%2Ftzdata-2024a-h0c530f3_0.conda), so it shouldn't have to be revendored elsewhere.
Like I said, the right approach IMO is to search a couple of reasonable candidate paths for the TZDB, including `$PREFIX/share/zoneinfo` (also on windows).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2004323054
Update: I'm able to reproduce this on my Windows PC. Actually all pyarrow orc test fail when trying to read data at https://github.com/apache/arrow/blob/5718a2862b4254d8bf938912d8958837ac7313a5/cpp/src/arrow/adapters/orc/adapter.cc#L455
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1960774835
Still segfaulting as of v15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1691625160
It does seem that ORC is disabled on our Windows wheels:
https://github.com/apache/arrow/blob/main/ci/scripts/python_wheel_windows_build.bat#L39
Maybe this is not tested anywhere as I can't find any Windows Pyarrow test with ORC enabled because on Appveyor we use `ARROW_ORC=ON` https://github.com/apache/arrow/blob/main/ci/appveyor-cpp-build.bat#L80 but we don't enable it for `PYARROW`: https://github.com/apache/arrow/blob/main/ci/appveyor-cpp-build.bat#L117-L129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1697080726
I haven't been able to reproduce this issue neither on the Windows wheels nor the Appveyor build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1616185399
The same segfault still appears with orc 1.9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1616447791
@wgtmac may this related to C++ Orc release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1691665398
Do we have any other Windows testing at all?
I suppose not testing ORC for pyarrow on appveyor is just an oversight, since it needs to be enabled manually? (and we forgot to do that) So we could start with enabling it there, and see if we can reproduce those errors?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2053968342
JFYI: https://github.com/apache/orc/pull/1882 was merged (thanks @wgtmac!), so we'll be able to fix this in conda-forge without injecting `TZDIR` environment variables into every user environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2009701724
I need to debug it on Windows. If open() returns -1 and throws the exception, then we should already catch it and see the error message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2006161559
> Ah, that's good to hear! And I also see that ORC just released last week, although I assume that is a bit too recent to already upgrade our dependency here?
Unfortunately, the ORC 2.0.0 upgrade is blocked by dropping Java 8 support on the ORC side: https://github.com/apache/arrow/pull/40508.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2003101234
Yes, I found this on my Mac laptop. Below are the minimal configs required to run pyarrow test. I will try to make it work on my Windows PC once back home.
```
mkdir arrow/cpp/build_conda
pushd arrow/cpp/build_conda
cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_BUILD_TYPE=Debug \
-DARROW_BUILD_TESTS=OFF \
-DARROW_PARQUET=ON \
-DARROW_ORC=ON \
-DARROW_COMPUTE=ON \
-DARROW_CSV=ON \
-DARROW_DATASET=ON \
-DARROW_JSON=ON \
-DPython3_EXECUTABLE=/opt/miniconda3/bin/python3.12 \
..
make -j4
make install
popd
pushd arrow/python
export PYARROW_WITH_PARQUET=1
export PYARROW_WITH_ORC=1
export PYARROW_WITH_DATASET=1
export PYARROW_PARALLEL=4
python setup.py build_ext --inplace
popd
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] BUG: test_orc_format segfaults on windows [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-2003097152
Could you enable JSON by `-DARROW_JSON=ON`?
We also need `-DARROW_CSV=ON`/`-DARROW_DATASET=ON`/`-DARROW_FILESYSTEM=ON`/`-DARROW_HDFS=ON` for PyArrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #36026: [Python] BUG: test_orc_format segfaults on windows
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #36026:
URL: https://github.com/apache/arrow/issues/36026#issuecomment-1692626980
Oh, let's enable it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org