You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "NoahFournier (via GitHub)" <gi...@apache.org> on 2023/02/13 16:04:04 UTC
[GitHub] [arrow] NoahFournier opened a new pull request, #34164: GH-34163: [C++] [CI] Fix typo in build_orc macro
NoahFournier opened a new pull request, #34164:
URL: https://github.com/apache/arrow/pull/34164
This PR fixes the build_orc macro, allowing it to use the same zstd root as the one used for the main arrow build command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #34164: GH-34163: [C++][CI] Ensure using the same Zstandard with bundled ORC
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #34164:
URL: https://github.com/apache/arrow/pull/34164#issuecomment-1431665452
Benchmark runs are scheduled for baseline = 306026d8611e4bd9676788f6e5a195cae8dc8610 and contender = 13a9ad988e908846aca960515d9ca855c3328e7c. 13a9ad988e908846aca960515d9ca855c3328e7c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/4517d216d3604cf2b36b89e7d7625c1e...501fe0c85c2e48b181e1df8585d4435c/)
[Failed :arrow_down:0.21% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/c216c5e89fc64cb78d9abc8bfdefeb79...a4f80f3b213b4265928d7d36f5bc610f/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/1acc896a38244a8dab43c1a53a64a37a...613529863c544274962b62809ad4ea7f/)
[Finished :arrow_down:0.13% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9ac83056e48c41d38b31896b0f389ccb...ad8ebbd756504889902a6b66e48e6a84/)
Buildkite builds:
[Finished] [`13a9ad98` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2376)
[Finished] [`13a9ad98` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2406)
[Finished] [`13a9ad98` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2374)
[Finished] [`13a9ad98` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2398)
[Finished] [`306026d8` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2375)
[Failed] [`306026d8` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2405)
[Finished] [`306026d8` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2373)
[Finished] [`306026d8` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2397)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-34163: [C++][CI] Ensure using the same Zstandard with bundled ORC [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #34164:
URL: https://github.com/apache/arrow/pull/34164#discussion_r1573485269
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -4446,7 +4446,9 @@ macro(build_orc)
"-DPROTOBUF_LIBRARY=${ORC_PROTOBUF_LIBRARY}"
"-DPROTOC_LIBRARY=${ORC_PROTOBUF_LIBRARY}"
"-DLZ4_HOME=${ORC_LZ4_ROOT}"
- "-DZSTD_HOME=${ORZ_ZSTD_ROOT}")
+ "-DZSTD_HOME=${ORC_ZSTD_ROOT}"
+ "-DZSTD_INCLUDE_DIR=$<TARGET_PROPERTY:${ARROW_ZSTD_LIBZSTD},INTERFACE_INCLUDE_DIRECTORIES>"
Review Comment:
Thanks for the explanation! I will restart the work on the ORC side after I have finished reading all the CMake code on the Arrow side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou merged pull request #34164: GH-34163: [C++][CI] Ensure using the same Zstandard with bundled ORC
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou merged PR #34164:
URL: https://github.com/apache/arrow/pull/34164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #34164: GH-34163: [C++][CI] Ensure using the same Zstandard with bundled ORC
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #34164:
URL: https://github.com/apache/arrow/pull/34164#issuecomment-1431668899
['Python', 'R'] benchmarks have high level of regressions.
[test-mac-arm](https://conbench.ursa.dev/compare/runs/c216c5e89fc64cb78d9abc8bfdefeb79...a4f80f3b213b4265928d7d36f5bc610f/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #34164: GH-34163: [C++][CI] Fix typo in build_orc macro
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #34164:
URL: https://github.com/apache/arrow/pull/34164#issuecomment-1429209148
Good catch!
We need to fix Windows CI job failure: https://github.com/apache/arrow/actions/runs/4165359530/jobs/7208928002
```text
FAILED: orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure D:/a/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure
cmd.exe /C "cd /D D:\a\arrow\arrow\build\cpp\orc_ep-prefix\src\orc_ep-build && "C:\Program Files\CMake\bin\cmake.exe" -P D:/a/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-DEBUG.cmake && "C:\Program Files\CMake\bin\cmake.exe" -E touch D:/a/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure"
CMake Error at D:/a/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-DEBUG.cmake:37 (message):
Command failed: 1
'C:/Program Files/CMake/bin/cmake.exe' '-DCMAKE_C_COMPILER=C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe' '-DCMAKE_CXX_COMPILER=C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe' '-DCMAKE_AR=C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/lib.exe' '-DCMAKE_RANLIB=:' '-DBUILD_SHARED_LIBS=OFF' '-DBUILD_STATIC_LIBS=ON' '-DBUILD_TESTING=OFF' '-DCMAKE_BUILD_TYPE=DEBUG' '-DCMAKE_CXX_FLAGS= /D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING' '-DCMAKE_CXX_FLAGS_DEBUG=/MDd /Z7 /Ob0 /Od /RTC1 ' '-DCMAKE_CXX_FLAGS_MISIZEREL=/MD /O1 /Ob1 /DNDEBUG' '-DCMAKE_CXX_FLAGS_RELEASE=/MD /O2 /Ob2 /DNDEBUG' '-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=/MD /Z7 /O2 /Ob1 /DNDEBUG' '-DCMAKE_CXX_STANDARD=17' '-DCMAKE_C_FLAGS=' '-DCMAKE_C_FLAGS_DEBUG=/MDd /Z7 /Ob0 /Od /RTC1 ' '-DCMAKE_C_FLAGS_MISIZEREL=/MD /O1 /Ob1 /DNDEBUG' '-DCMAKE_C_FLAGS
_RELEASE=/MD /O2 /Ob2 /DNDEBUG' '-DCMAKE_C_FLAGS_RELWITHDEBINFO=/MD /Z7 /O2 /Ob1 /DNDEBUG' '-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=' '-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=' '-DCMAKE_INSTALL_LIBDIR=lib' '-DCMAKE_VERBOSE_MAKEFILE=OFF' '-DCMAKE_C_COMPILER_LAUNCHER=C:/Program Files/Git/usr/bin/ccache.exe' '-DCMAKE_CXX_COMPILER_LAUNCHER=C:/Program Files/Git/usr/bin/ccache.exe' '-DCMAKE_INSTALL_PREFIX=D:/a/arrow/arrow/build/cpp/orc_ep-install' '-DSTOP_BUILD_ON_WARNING=OFF' '-DBUILD_LIBHDFSPP=OFF' '-DBUILD_JAVA=OFF' '-DBUILD_TOOLS=OFF' '-DBUILD_CPP_TESTS=OFF' '-DINSTALL_VENDORED_LIBS=OFF' '-DSNAPPY_HOME=D:/a/arrow/arrow/build/cpp/snappy_ep/src/snappy_ep-install' '-DSNAPPY_INCLUDE_DIR=D:/a/arrow/arrow/build/cpp/snappy_ep/src/snappy_ep-install/include' '-DPROTOBUF_EXECUTABLE=D:/a/arrow/arrow/build/cpp/protobuf_ep-install/bin/protoc' '-DPROTOBUF_HOME=D:/a/arrow/arrow/build/cpp/protobuf_ep-install' '-DPROTOBUF_INCLUDE_DIR=D:/a/arrow/arrow/build/cpp/protobuf_ep-install/include' '-DPROTOBUF_LIBR
ARY=D:/a/arrow/arrow/build/cpp/protobuf_ep-install/lib/libprotobuf.lib' '-DPROTOC_LIBRARY=D:/a/arrow/arrow/build/cpp/protobuf_ep-install/lib/libprotobuf.lib' '-DLZ4_HOME=D:/a/arrow/arrow/build/cpp/lz4_ep-install' '-DZSTD_HOME=D:/a/arrow/arrow/build/cpp/zstd_ep-install' '-DPROTOBUF_EXECUTABLE:FILEPATH=D:/a/arrow/arrow/build/cpp/protobuf_ep-install/bin/protoc' '-GNinja' 'D:/a/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep'
See also
D:/a/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-*.log
-- stdout output is:
-- The C compiler identification is MSVC 19.29.30147.0
-- The CXX compiler identification is MSVC 19.29.30147.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type: DEBUG
-- compiler MSVC version 19.29.30147.0
-- Performing Test ORC_CXX_HAS_INITIALIZER_LIST
-- Performing Test ORC_CXX_HAS_INITIALIZER_LIST - Success
-- Performing Test ORC_CXX_HAS_NOEXCEPT
-- Performing Test ORC_CXX_HAS_NOEXCEPT - Success
-- Performing Test ORC_CXX_HAS_NULLPTR
-- Performing Test ORC_CXX_HAS_NULLPTR - Success
-- Performing Test ORC_CXX_HAS_OVERRIDE
-- Performing Test ORC_CXX_HAS_OVERRIDE - Success
-- Performing Test ORC_CXX_HAS_UNIQUE_PTR
-- Performing Test ORC_CXX_HAS_UNIQUE_PTR - Success
-- Performing Test ORC_CXX_HAS_CSTDINT
-- Performing Test ORC_CXX_HAS_CSTDINT - Success
-- Performing Test ORC_CXX_HAS_THREAD_LOCAL
-- Performing Test ORC_CXX_HAS_THREAD_LOCAL - Success
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Found Threads: TRUE
-- SNAPPY_HOME: D:/a/arrow/arrow/build/cpp/snappy_ep/src/snappy_ep-install
-- Found the Snappy header: D:/a/arrow/arrow/build/cpp/snappy_ep/src/snappy_ep-install/include/snappy.h
-- Found the Snappy library: D:/a/arrow/arrow/build/cpp/snappy_ep/src/snappy_ep-install/lib/snappy.lib
-- ZSTD_HOME: D:/a/arrow/arrow/build/cpp/zstd_ep-install
-- Configuring incomplete, errors occurred!
See also "D:/a/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build/CMakeFiles/CMakeOutput.log".
See also "D:/a/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build/CMakeFiles/CMakeError.log".
-- stderr output is:
CMake Error at cmake_modules/FindZSTD.cmake:61 (message):
Could not find zstd in system search paths.
Call Stack (most recent call first):
cmake_modules/ThirdpartyToolchain.cmake:166 (find_package)
CMakeLists.txt:141 (INCLUDE)
CMake Error at D:/a/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-DEBUG.cmake:47 (message):
Stopping after outputting logs.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #34164: GH-34163: [C++] [CI] Fix typo in build_orc macro
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34164:
URL: https://github.com/apache/arrow/pull/34164#issuecomment-1428202958
* Closes: #34163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-34163: [C++][CI] Ensure using the same Zstandard with bundled ORC [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #34164:
URL: https://github.com/apache/arrow/pull/34164#discussion_r1573402648
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -4446,7 +4446,9 @@ macro(build_orc)
"-DPROTOBUF_LIBRARY=${ORC_PROTOBUF_LIBRARY}"
"-DPROTOC_LIBRARY=${ORC_PROTOBUF_LIBRARY}"
"-DLZ4_HOME=${ORC_LZ4_ROOT}"
- "-DZSTD_HOME=${ORZ_ZSTD_ROOT}")
+ "-DZSTD_HOME=${ORC_ZSTD_ROOT}"
+ "-DZSTD_INCLUDE_DIR=$<TARGET_PROPERTY:${ARROW_ZSTD_LIBZSTD},INTERFACE_INCLUDE_DIRECTORIES>"
Review Comment:
No problem!
It's for using with bundled Zstandard. If we use bundled Zstandard and bundled Apache ORC, the bundled Zstandard isn't built yet when we run `cmake` for the bundled Apache ORC. So the bundled Apache ORC can't detect the bundled Zstandard's include directory and library path. Because they don't exist yet. So we specify them here explicitly.
> I guess these definitions are not necessary to build Apache ORC.
Right.
> However, I have found `list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake_modules")` in the cpp/CMakeLists.txt of Apache Arrow. Does it mean that we have overridden the module path so Apache ORC no longer finds package using the default FindXXX.cmake and these flags are missing so we have to provide on our own?
The `list(...)` doesn't affect Apache ORC. Because `externaproject_add()` uses separated `cmake` for each external project. `CMAKE_MODULE_PATH` in Apache Arrow isn't used for external projects including bundled Apache ORC implicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-34163: [C++][CI] Ensure using the same Zstandard with bundled ORC [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #34164:
URL: https://github.com/apache/arrow/pull/34164#discussion_r1573190465
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -4446,7 +4446,9 @@ macro(build_orc)
"-DPROTOBUF_LIBRARY=${ORC_PROTOBUF_LIBRARY}"
"-DPROTOC_LIBRARY=${ORC_PROTOBUF_LIBRARY}"
"-DLZ4_HOME=${ORC_LZ4_ROOT}"
- "-DZSTD_HOME=${ORZ_ZSTD_ROOT}")
+ "-DZSTD_HOME=${ORC_ZSTD_ROOT}"
+ "-DZSTD_INCLUDE_DIR=$<TARGET_PROPERTY:${ARROW_ZSTD_LIBZSTD},INTERFACE_INCLUDE_DIRECTORIES>"
Review Comment:
I don't quite understand why we have to define `ZSTD_INCLUDE_DIR` and `ZSTD_LIBRARY`? Isn't it enough to define `ZSTD_HOME` for building ORC as the FindZSTD.cmake provided by ORC will set `ZSTD_INCLUDE_DIR` and `ZSTD_LIBRARY` properly.
I guess these definitions are not necessary to build Apache ORC. However, I have found `list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake_modules")` in the cpp/CMakeLists.txt of Apache Arrow. Does it mean that we have overridden the module path so Apache ORC no longer finds package using the default FindXXX.cmake and these flags are missing so we have to provide on our own?
Sorry for the newbie question. I'm still not a CMake expert. @kou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org