You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "h-vetinari (via GitHub)" <gi...@apache.org> on 2023/06/07 22:32:08 UTC
[GitHub] [arrow] h-vetinari opened a new issue, #35986: [C++] Compatibility with protobuf 4.23.x
h-vetinari opened a new issue, #35986:
URL: https://github.com/apache/arrow/issues/35986
### Describe the bug, including details regarding any error messages, version, and platform.
Protobuf has changed their versioning [scheme](https://protobuf.dev/support/version-support) with 3.20.x (though it really took effect in 3.21.x), they now use the minor number as the feature-generation, and the major number (which moved independently per language) as indicating breaking changes. For example protobuf-C++ 3.21 came together with protobuf-python 4.21, whereas for the short-lived 22-series, also the C++ major number got increased (4.22 for both).
This is partly due to protobuf now relying on abseil, and partly because some include paths changed. However, grpc jumped from protobuf 3.21 directly to 4.23 and never supported the 22-series. Therefore the first real occasion to build arrow against a protobuf 4.x world was as soon as grpc 1.55 and protobuf 4.23 were available.
Trying to [build](https://github.com/conda-forge/arrow-cpp-feedstock/pull/1082) arrow in conda-forge against protobuf 4 currently runs into:
```
[294/483] Building CXX object src/arrow/engine/CMakeFiles/arrow_substrait_objlib.dir/substrait/serde.cc.o
FAILED: src/arrow/engine/CMakeFiles/arrow_substrait_objlib.dir/substrait/serde.cc.o
/home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/_build_env/bin/x86_64-conda-linux-gnu-c++ -DARROW_ENGINE_EXPORTING -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HDFS -DARROW_MIMALLOC -DARROW_S3_HAS_CRT -DARROW_USE_GLOG -DARROW_WITH_BROTLI -DARROW_WITH_BZ2 -DARROW_WITH_LZ4 -DARROW_WITH_RE2 -DARROW_WITH_SNAPPY -DARROW_WITH_TIMING_TESTS -DARROW_WITH_UTF8PROC -DARROW_WITH_ZLIB -DARROW_WITH_ZSTD -DAWS_AUTH_USE_IMPORT_EXPORT -DAWS_CAL_USE_IMPORT_EXPORT -DAWS_CHECKSUMS_USE_IMPORT_EXPORT -DAWS_COMMON_USE_IMPORT_EXPORT -DAWS_COMPRESSION_USE_IMPORT_EXPORT -DAWS_CRT_CPP_USE_IMPORT_EXPORT -DAWS_EVENT_STREAM_USE_IMPORT_EXPORT -DAWS_HTTP_USE_IMPORT_EXPORT -DAWS_IO_USE_IMPORT_EXPORT -DAWS_MQTT_USE_IMPORT_EXPORT -DAWS_MQTT_WITH_WEBSOCKETS -DAWS_S3_USE_IMPORT_EXPORT -DAWS_SDKUTILS_USE_IMPORT_EXPORT -DAWS_SDK_VERSION_MAJOR=1 -DAWS_SDK_VERSION_MINOR=10 -DAWS_SDK_VERSION_PATCH=57 -DAWS_USE_EPOLL -DURI_STATIC_B
UILD -I/home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work/cpp/build/src -I/home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work/cpp/src -I/home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work/cpp/src/generated -I/home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work/cpp/build/substrait_ep-generated -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work/cpp/thirdparty/flatbuffers/include -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work/cpp/thirdparty/hadoop/include -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work/cpp/build/jemalloc_ep-prefix/src -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work/cpp/build/mimalloc_ep/src/mimalloc_ep/include/mimalloc-2.0 -Wno-noexcept-type -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vecto
rize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work=/usr/local/src/conda/libarrow-12.0.0 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol=/usr/local/src/conda-prefix -isystem /usr/local/cuda/include -fdiagnostics-color=always -fuse-ld=gold -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -f
PIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work=/usr/local/src/conda/libarrow-12.0.0 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol=/usr/local/src/conda-prefix -isystem /usr/local/cuda/include -O3 -DNDEBUG -O2 -ftree-vectorize -std=c++1z -fPIC -pthread -DS2N_KYBER512R3_AVX2_BMI2 -DS2N_STACKTRACE -DS2N_CPUID_AVAILABLE -DS2N_FEATURES_AVAILABLE -fPIC -DS2N_FALL_THROUGH_SUPPORTED -DS
2N___RESTRICT__SUPPORTED -DS2N_MADVISE_SUPPORTED -DS2N_CLONE_SUPPORTED -DS2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH -DS2N_LIBCRYPTO_SUPPORTS_EVP_RC4 -DS2N_LIBCRYPTO_SUPPORTS_EVP_MD_CTX_SET_PKEY_CTX -MD -MT src/arrow/engine/CMakeFiles/arrow_substrait_objlib.dir/substrait/serde.cc.o -MF src/arrow/engine/CMakeFiles/arrow_substrait_objlib.dir/substrait/serde.cc.o.d -o src/arrow/engine/CMakeFiles/arrow_substrait_objlib.dir/substrait/serde.cc.o -c /home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work/cpp/src/arrow/engine/substrait/serde.cc
/home/conda/feedstock_root/build_artifacts/apache-arrow_1686142324384/work/cpp/src/arrow/engine/substrait/serde.cc:27:10: fatal error: google/protobuf/stubs/status.h: No such file or directory
#include <google/protobuf/stubs/status.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
```
There will probably be other issues as well...
### Component(s)
C++, Packaging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #35986: [C++] Compatibility with protobuf 4.23.x
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #35986:
URL: https://github.com/apache/arrow/issues/35986#issuecomment-1616186653
So I patched our shared builds of grpc to also export that symbol required by arrow_flight, and our [builds](https://github.com/conda-forge/arrow-cpp-feedstock/pull/1082) have been live for two weeks with no reported problems. Closing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari closed issue #35986: [C++] Compatibility with protobuf 4.23.x
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari closed issue #35986: [C++] Compatibility with protobuf 4.23.x
URL: https://github.com/apache/arrow/issues/35986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #35986: [C++] Compatibility with protobuf 4.23.x
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #35986:
URL: https://github.com/apache/arrow/issues/35986#issuecomment-1581858889
Looking at #35987, it seems that `main` has already progress a bit further on this, and looking at the affected file, I see https://github.com/apache/arrow/commit/b80c6551570b8324298a12dc36be1b5eba76e277. Let me try backporting that...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #35986: [C++] Compatibility with protobuf 4.23.x
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #35986:
URL: https://github.com/apache/arrow/issues/35986#issuecomment-1581882255
Next issue on windows:
```
[102/129] Linking CXX shared library release\arrow_flight.dll
FAILED: release/arrow_flight.dll release/arrow_flight.lib
cmd.exe /C "cd . && D:\bld\apache-arrow_1686197619141\_build_env\Library\bin\cmake.exe -E vs_link_dll --intdir=src\arrow\flight\CMakeFiles\arrow_flight_shared.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests -- C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\link.exe /nologo @CMakeFiles\arrow_flight_shared.rsp /out:release\arrow_flight.dll /implib:release\arrow_flight.lib /pdb:release\arrow_flight.pdb /dll /version:1200.0 /machine:x64 /NODEFAULTLIB:LIBCMT /INCREMENTAL:NO -Wl,--version-script=D:/bld/apache-arrow_1686197619141/work/cpp/src/arrow/symbols.map && cd ."
LINK: command "C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\link.exe /nologo @CMakeFiles\arrow_flight_shared.rsp /out:release\arrow_flight.dll /implib:release\arrow_flight.lib /pdb:release\arrow_flight.pdb /dll /version:1200.0 /machine:x64 /NODEFAULTLIB:LIBCMT /INCREMENTAL:NO -Wl,--version-script=D:/bld/apache-arrow_1686197619141/work/cpp/src/arrow/symbols.map /MANIFEST /MANIFESTFILE:release\arrow_flight.dll.manifest" failed (exit code 1120) with the following output:
LINK : warning LNK4044: unrecognized option '/Wl,--version-script=D:/bld/apache-arrow_1686197619141/work/cpp/src/arrow/symbols.map'; ignored
Creating library release\arrow_flight.lib and object release\arrow_flight.exp
unity_1_cxx.cxx.obj : error LNK2019: unresolved external symbol "const grpc::experimental::StaticDataCertificateProvider::`vftable'" (??_7StaticDataCertificateProvider@experimental@grpc@@6B@) referenced in function "void __cdecl std::_Construct_in_place<class grpc::experimental::StaticDataCertificateProvider,char const (&)[1018]>(class grpc::experimental::StaticDataCertificateProvider &,char const (&)[1018])" (??$_Construct_in_place@VStaticDataCertificateProvider@experimental@grpc@@AEAY0DPK@$$CBD@std@@YAXAEAVStaticDataCertificateProvider@experimental@grpc@@AEAY0DPK@$$CBD@Z)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #35986: [C++] Compatibility with protobuf 4.23.x
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #35986:
URL: https://github.com/apache/arrow/issues/35986#issuecomment-1581863394
I had searched for issues before opening one, but hadn't seen #35538 for some reason, sorry. I'm backporting https://github.com/apache/arrow/commit/b80c6551570b8324298a12dc36be1b5eba76e277 now, will report back if something else doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #35986: [C++] Compatibility with protobuf 4.23.x
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #35986:
URL: https://github.com/apache/arrow/issues/35986#issuecomment-1581882767
Ah, nevermind, that's most likely because we switched our grpc builds to be shared in conda-forge...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org