You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/21 13:07:44 UTC
[GitHub] [spark] 19855134604 opened a new pull request, #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
19855134604 opened a new pull request, #38743:
URL: https://github.com/apache/spark/pull/38743
### What changes were proposed in this pull request?
This PR use profile named `-Puser-defined-protoc` to support that users can build and test `protobuf` module by specifying custom `protoc` executables.
### Why are the changes needed?
As described in [SPARK-41215](https://issues.apache.org/jira/browse/SPARK-41215), the latest versions of `protoc` have the minimum version requirements for basic libraries such as `glibc` and `glibcxx`. Because of that it is not possible to test-compile the `protobuf` module out of the box on CentOS 6 or CentOS 7. Instead the following error messages is shown:
```
[ERROR] /home/disk1/spark-ut/spark/connector/protobuf/src/test/resources/protobuf/timestamp.proto [0:0]: /tmp/protoc6599263403262688374.exe: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/protoc6599263403262688374.exe)
[ERROR] /home/disk1/spark-ut/spark/connector/protobuf/src/test/resources/protobuf/timestamp.proto [0:0]: /tmp/protoc6599263403262688374.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /tmp/protoc6599263403262688374.exe)
[ERROR] /home/disk1/spark-ut/spark/connector/protobuf/src/test/resources/protobuf/timestamp.proto [0:0]: /tmp/protoc6599263403262688374.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /tmp/protoc6599263403262688374.exe)
[ERROR] /home/disk1/spark-ut/spark/connector/protobuf/src/test/resources/protobuf/timestamp.proto [0:0]: /tmp/protoc6599263403262688374.exe: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.5' not found (required by /tmp/protoc6599263403262688374.exe)
```
### Does this PR introduce _any_ user-facing change?
No, the way to using official pre-release `protoc` binary files is activated by default.
### How was this patch tested?
- Pass GitHub Actions
- Manual test on CentOS6u3 and CentOS7u4
```bash
export PROTOBUF_PROTOC_EXEC_PATH=/path-to-protoc-exe
./build/mvn clean install -pl connector/protobuf -Puser-defined-protoc -am -DskipTests
./build/mvn clean test -pl connector/protobuf -Puser-defined-protoc
```
and
```bash
export PROTOBUF_PROTOC_EXEC_PATH=/path-to-protoc-exe
./build/sbt clean "protobuf/compile" -Puser-defined-protoc
./build/sbt "protobuf/test" -Puser-defined-protoc
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #38743:
URL: https://github.com/apache/spark/pull/38743#issuecomment-1322936602
cc @HyukjinKwon FYI, a similar fix as SPARK-40593
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #38743:
URL: https://github.com/apache/spark/pull/38743#discussion_r1028737655
##########
connector/protobuf/README.md:
##########
@@ -0,0 +1,37 @@
+# Spark Protobuf - Developer Documentation
+
+## Getting Started
+
+### Build
+
+```bash
+./build/mvn -Phive clean package
Review Comment:
For `protobuf` module, do we have to compile with `-Phive`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WolverineJiang commented on pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
Posted by GitBox <gi...@apache.org>.
WolverineJiang commented on PR #38743:
URL: https://github.com/apache/spark/pull/38743#issuecomment-1325891951
Thanks @HyukjinKwon @LuciferYang @AmplabJenkins~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #38743:
URL: https://github.com/apache/spark/pull/38743#issuecomment-1324806600
@HyukjinKwon Does this one need to be reviewed by others?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
URL: https://github.com/apache/spark/pull/38743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] 19855134604 commented on a diff in pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
Posted by GitBox <gi...@apache.org>.
19855134604 commented on code in PR #38743:
URL: https://github.com/apache/spark/pull/38743#discussion_r1028790815
##########
connector/protobuf/README.md:
##########
@@ -0,0 +1,37 @@
+# Spark Protobuf - Developer Documentation
+
+## Getting Started
+
+### Build
+
+```bash
+./build/mvn -Phive clean package
Review Comment:
It doesn't seem to be necessary. Let me adjust it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #38743:
URL: https://github.com/apache/spark/pull/38743#issuecomment-1323783433
checked this pr in CentOS 6u3 passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38743:
URL: https://github.com/apache/spark/pull/38743#issuecomment-1324793243
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #38743:
URL: https://github.com/apache/spark/pull/38743#issuecomment-1325892309
Congratulations @WolverineJiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38743: [SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf.
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38743:
URL: https://github.com/apache/spark/pull/38743#issuecomment-1325885697
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org