You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/17 11:20:59 UTC
[GitHub] [spark] grundprinzip commented on pull request #38609: [SPARK-40593][BUILD][CONNECT] Make user can build and test `connect` module by specifying the user-defined `protoc` and `protoc-gen-grpc-java`
grundprinzip commented on PR #38609:
URL: https://github.com/apache/spark/pull/38609#issuecomment-1318486206
Since I cannot directly edit the PR description or title, I would kindly ask you to do the following changes:
Title: [SPARK-40593][BUILD][CONNECT] Support user configurable `protoc` and `protoc-gen-grpc-java` executables when building Spark Connect.
> ### What changes were proposed in this pull request?
> This PR adds a new profile named `-Puser-defined-protoc` to support that users can build and test `connect` module by specifying custom `protoc ` and `protoc-gen-grpc-java` executables.
>
> ### Why are the changes needed?
> As described in [SPARK-40593](https://issues.apache.org/jira/browse/SPARK-40593), the latest versions of `protoc` and `protoc-gen-grpc-java` have minimum version requirements for basic libraries such as `glibc` and `glibcxx`. Because of that it is not possible to compile the `connect` module out of the box on CentOS 6 or CentOS 7. Instead the following error messages is shown:
>
> ```
> [ERROR] PROTOC FAILED: /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe)
> /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe)
> /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe)
> /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.5' not found (required by /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe)
> ```
>
> ### Does this PR introduce _any_ user-facing change?
> No, the way to using official pre-release `protoc` and `protoc-gen-grpc-java` binary files is activated by default.
>
> ### How was this patch tested?
> * Pass GitHub Actions
> * Manual test on CentOS6u3 and CentOS7u4
>
> ```shell
> export CONNECT_PROTOC_EXEC_PATH=/path-to-protoc-exe
> export CONNECT_PLUGIN_EXEC_PATH=/path-to-protoc-gen-grpc-java-exe
> ./build/mvn clean install -pl connector/connect -Puser-defined-protoc -am -DskipTests
> ./build/mvn clean test -pl connector/connect -Puser-defined-protoc
> ```
>
> and
>
> ```shell
> export CONNECT_PROTOC_EXEC_PATH=/path-to-protoc-exe
> export CONNECT_PLUGIN_EXEC_PATH=/path-to-protoc-gen-grpc-java-exe
> ./build/sbt clean "connect/compile" -Puser-defined-protoc
> ./build/sbt "connect/test" -Puser-defined-protoc
> ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org