You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "raulcd (via GitHub)" <gi...@apache.org> on 2026/07/07 06:53:49 UTC
[PR] GH-47583: [CI][Python] Enable PyArrow Debug build on Windows CI job [arrow]
raulcd opened a new pull request, #50406:
URL: https://github.com/apache/arrow/pull/50406
# TBD
### Rationale for this change
### What changes are included in this PR?
### Are these changes tested?
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
**This PR contains a "Critical Fix".** (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #50406:
URL: https://github.com/apache/arrow/pull/50406#discussion_r3545118358
##########
ci/scripts/python_build.bat:
##########
@@ -59,8 +59,14 @@ set ARROW_WITH_LZ4=ON
set ARROW_WITH_SNAPPY=ON
set ARROW_WITH_ZLIB=ON
set ARROW_WITH_ZSTD=ON
-set CMAKE_BUILD_TYPE=Release
+set CMAKE_BUILD_TYPE=RelWithDebInfo
set CMAKE_GENERATOR=Ninja
+@rem Set CMAKE_CXX_FLAGS_RELWITHDEBINFO and CMAKE_C_FLAGS_RELWITHDEBINFO to
+@rem override default /DNDEBUG to be dropped so assertions are maintained.
+@rem A debug build would require linking against python3xx_d.lib (debug).
+@rem See details of discussion on PR GH-50406
Review Comment:
No, I want users to go to the discussion on the PR not the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4922966187
Can you set `CMAKE_MSVC_DEBUG_INFORMATION_FORMAT` to `Embedded` to make ccache effective here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50406:
URL: https://github.com/apache/arrow/pull/50406#discussion_r3545107643
##########
ci/scripts/python_build.bat:
##########
@@ -59,8 +59,14 @@ set ARROW_WITH_LZ4=ON
set ARROW_WITH_SNAPPY=ON
set ARROW_WITH_ZLIB=ON
set ARROW_WITH_ZSTD=ON
-set CMAKE_BUILD_TYPE=Release
+set CMAKE_BUILD_TYPE=RelWithDebInfo
set CMAKE_GENERATOR=Ninja
+@rem Set CMAKE_CXX_FLAGS_RELWITHDEBINFO and CMAKE_C_FLAGS_RELWITHDEBINFO to
+@rem override default /DNDEBUG to be dropped so assertions are maintained.
+@rem A debug build would require linking against python3xx_d.lib (debug).
+@rem See details of discussion on PR GH-50406
+set CMAKE_CXX_FLAGS_RELWITHDEBINFO=/O2 /Ob1
+set CMAKE_C_FLAGS_RELWITHDEBINFO=/O2 /Ob1
Review Comment:
Overriding `CMAKE_*_FLAGS_RELWITHDEBINFO` to `/O2 /Ob1` drops MSVC’s default debug-info flag for RelWithDebInfo (typically `/Zi`). That undermines the stated goal of keeping debug information while enabling assertions (by removing `/DNDEBUG`). Consider keeping `/Zi` (or `/Z7`) while omitting `/DNDEBUG`.
##########
ci/scripts/python_build.bat:
##########
@@ -59,8 +59,14 @@ set ARROW_WITH_LZ4=ON
set ARROW_WITH_SNAPPY=ON
set ARROW_WITH_ZLIB=ON
set ARROW_WITH_ZSTD=ON
-set CMAKE_BUILD_TYPE=Release
+set CMAKE_BUILD_TYPE=RelWithDebInfo
set CMAKE_GENERATOR=Ninja
+@rem Set CMAKE_CXX_FLAGS_RELWITHDEBINFO and CMAKE_C_FLAGS_RELWITHDEBINFO to
+@rem override default /DNDEBUG to be dropped so assertions are maintained.
+@rem A debug build would require linking against python3xx_d.lib (debug).
+@rem See details of discussion on PR GH-50406
Review Comment:
The comment references "PR GH-50406", but this PR is GH-47583 and the linked discussion in the PR metadata points to GH-47567. This looks like a stale/incorrect reference and may confuse future readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4916238038
This seems to be good, on this PR:
> [26/94] "C:\Program Files\Git\usr\bin\ccache.exe" C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1444~1.352\bin\Hostx64\x64\cl.exe /nologo /TP -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_PYTHON_EXPORTING -D_CRT_SECURE_NO_WARNINGS -D__SSE2__ -D__SSE4_1__ -D__SSE4_2__ -Darrow_python_EXPORTS -ID:\a\arrow\arrow\python\pyarrow\src -IC:\Users\runneradmin\AppData\Local\Temp\tmpxly05kia\build\pyarrow\src -IC:\hostedtoolcache\windows\Python\3.13.14\x64\Include -I"C:\Program Files\Git\usr\include" -IC:\hostedtoolcache\windows\Python\3.13.14\x64\Lib\site-packages\numpy\_core\include /EHsc /wd5105 /bigobj /utf-8 /W3 /wd4800 /wd4996 /wd4065 /arch:SSE4.2 /DWIN32 /D_WINDOWS /EHsc /D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING /wd4190 /wd4293 /wd4800 /wd4551 /O2 /Ob1 -std:c++20 -MD -Zi /showIncludes /FoCMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\arrow_to_pandas.cc.obj /FdCMakeFiles\arrow_pyt
hon.dir\ /FS -c D:\a\arrow\arrow\python\pyarrow\src\arrow\python\arrow_to_pandas.cc
No `/DNDEBUG` and debug symbols present `-Zi`.
Previously for Release `/DNDEBUG` was present and `-Zi` was not:
> [25/94] "C:\Program Files\Git\usr\bin\ccache.exe" C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1444~1.352\bin\Hostx64\x64\cl.exe /nologo /TP -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_PYTHON_EXPORTING -D_CRT_SECURE_NO_WARNINGS -D__SSE2__ -D__SSE4_1__ -D__SSE4_2__ -Darrow_python_EXPORTS -ID:\a\arrow\arrow\python\pyarrow\src -IC:\Users\runneradmin\AppData\Local\Temp\tmpgjv_4rcf\build\pyarrow\src -IC:\hostedtoolcache\windows\Python\3.13.14\x64\Include -I"C:\Program Files\Git\usr\include" -IC:\hostedtoolcache\windows\Python\3.13.14\x64\Lib\site-packages\numpy\_core\include /EHsc /wd5105 /bigobj /utf-8 /W3 /wd4800 /wd4996 /wd4065 /arch:SSE4.2 /DWIN32 /D_WINDOWS /EHsc /D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING /wd4190 /wd4293 /wd4800 /wd4551 /O2 /Ob2 /DNDEBUG -std:c++20 -MD /showIncludes /FoCMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\arrow_to_pandas.cc.obj /FdCMakeFiles\arro
w_python.dir\ /FS -c D:\a\arrow\arrow\python\pyarrow\src\arrow\python\arrow_to_pandas.cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4924129591
> Can you set `CMAKE_MSVC_DEBUG_INFORMATION_FORMAT` to `Embedded` to make ccache effective here?
Done, this should be ready for review @pitrou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow Debug build on Windows CI job [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4904232659
Ouch, I had missed that `_DEBUG` implied `Py_DEBUG`. This is annoying: we would like our own assertions (and the C++ stdlib's) without needing a special build of CPython :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50406:
URL: https://github.com/apache/arrow/pull/50406#discussion_r3550135225
##########
ci/scripts/python_build.bat:
##########
@@ -134,6 +144,10 @@ pushd %SOURCE_DIR%\python
%PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1
@REM Build PyArrow
-%PYTHON_CMD% -m pip install --no-deps --no-build-isolation -vv -C build.verbose=true . || exit /B 1
+%PYTHON_CMD% -m pip install --no-deps --no-build-isolation -vv ^
+ -C build.verbose=true ^
+ -C cmake.build-type=%CMAKE_BUILD_TYPE% ^
+ -C cmake.define.CMAKE_C_FLAGS_RELWITHDEBINFO="%CMAKE_C_FLAGS_RELWITHDEBINFO%" ^
+ -C cmake.define.CMAKE_CXX_FLAGS_RELWITHDEBINFO="%CMAKE_CXX_FLAGS_RELWITHDEBINFO%" . || exit /B 1
Review Comment:
For the PyArrow build, `-C cmake.define.*` is not the pattern documented in `docs/source/developers/python/building.rst` for passing extra CMake cache entries (it recommends `-C cmake.args=...`). Using `cmake.args` here matches the other CI scripts (e.g. `python_wheel_windows_build.bat`) and avoids any ambiguity about whether `cmake.define` is intended for arbitrary `CMAKE_*` variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow Debug build on Windows CI job [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4901036130
:warning: GitHub issue #47583 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow Debug build on Windows CI job [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4904403822
> I think we require installing CPython on Debug mode (manually?) which would not allow us to use the setup-python action
We can of course do this, assuming we find a readily available debug build of CPython on Windows, but then some dependencies might need to be recompiled from source and we'd rather avoid that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow Debug build on Windows CI job [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4904262570
See https://discuss.python.org/t/building-debug-version-of-extensions-on-windows-without-debug-binaries/105012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow Debug build on Windows CI job [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4904347633
According to the discussion above, perhaps we should use `RelWithDebInfo` (or even plain `Release`) and undefine the C preprocessor macro `NDEBUG`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4951336199
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 98a92c2f5dd6ae2c96557eaadeb0591b03bf1d04.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/86654435762) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #50406:
URL: https://github.com/apache/arrow/pull/50406#discussion_r3550145651
##########
ci/scripts/python_build.bat:
##########
@@ -134,6 +144,10 @@ pushd %SOURCE_DIR%\python
%PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1
@REM Build PyArrow
-%PYTHON_CMD% -m pip install --no-deps --no-build-isolation -vv -C build.verbose=true . || exit /B 1
+%PYTHON_CMD% -m pip install --no-deps --no-build-isolation -vv ^
+ -C build.verbose=true ^
+ -C cmake.build-type=%CMAKE_BUILD_TYPE% ^
+ -C cmake.define.CMAKE_C_FLAGS_RELWITHDEBINFO="%CMAKE_C_FLAGS_RELWITHDEBINFO%" ^
+ -C cmake.define.CMAKE_CXX_FLAGS_RELWITHDEBINFO="%CMAKE_CXX_FLAGS_RELWITHDEBINFO%" . || exit /B 1
Review Comment:
we shouldn't document all possible scikit build core flags. We only have some examples on our docs!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4924571862
Thanks! Total runtime for the Windows CI job went down from 43m to 16m thanks to caching!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd merged PR #50406:
URL: https://github.com/apache/arrow/pull/50406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #50406:
URL: https://github.com/apache/arrow/pull/50406#discussion_r3545127506
##########
ci/scripts/python_build.bat:
##########
@@ -59,8 +59,14 @@ set ARROW_WITH_LZ4=ON
set ARROW_WITH_SNAPPY=ON
set ARROW_WITH_ZLIB=ON
set ARROW_WITH_ZSTD=ON
-set CMAKE_BUILD_TYPE=Release
+set CMAKE_BUILD_TYPE=RelWithDebInfo
set CMAKE_GENERATOR=Ninja
+@rem Set CMAKE_CXX_FLAGS_RELWITHDEBINFO and CMAKE_C_FLAGS_RELWITHDEBINFO to
+@rem override default /DNDEBUG to be dropped so assertions are maintained.
+@rem A debug build would require linking against python3xx_d.lib (debug).
+@rem See details of discussion on PR GH-50406
+set CMAKE_CXX_FLAGS_RELWITHDEBINFO=/O2 /Ob1
+set CMAKE_C_FLAGS_RELWITHDEBINFO=/O2 /Ob1
Review Comment:
`Zi` is already part of the build flags as shown on the comment, that's LLM hallucinating :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47583: [CI][Python] Enable PyArrow Debug build on Windows CI job [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50406:
URL: https://github.com/apache/arrow/pull/50406#issuecomment-4903444345
@pitrou the problem seems to be that Debug builds try to link `python313_d.lib` instead of `python313.lib. This seems to be defined upstream on Python itself here (link for 3.16 but 3.13 is the same):
https://github.com/python/cpython/blob/2cd5b79284dd2331cbd9e11afabbfbf7e906103d/PC/pyconfig.h#L340-L342
And it fails with:
```
LINK : fatal error LNK1104: cannot open file 'python313_d.lib'
```
On the Windows CI job we use actions/setup-python which do not allow to install CPython debug. I could try `RelWithDebInfo` instead of `DEBUG`otherwise I think we require installing CPython on Debug mode (manually?) which would not allow us to use the setup-python action.
Do you have other ideas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org