You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2019/04/24 14:30:45 UTC
[spark] branch branch-2.4 updated: [SPARK-27550][TEST][BRANCH-2.4]
Fix `test-dependencies.sh` not to use `kafka-0-8`profile for Scala-2.12
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 34fd79d [SPARK-27550][TEST][BRANCH-2.4] Fix `test-dependencies.sh` not to use `kafka-0-8`profile for Scala-2.12
34fd79d is described below
commit 34fd79d3096939981055404b0034ab7582851c50
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Wed Apr 24 07:30:19 2019 -0700
[SPARK-27550][TEST][BRANCH-2.4] Fix `test-dependencies.sh` not to use `kafka-0-8`profile for Scala-2.12
## What changes were proposed in this pull request?
Since SPARK-27274 deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more.
Since Kakfa 0.8 doesn't have Scala-2.12 artifacts, e.g., `org.apache.kafka:kafka_2.12:jar:0.8.2.1`, this PR aims to fix `test-dependencies.sh` script to understand Scala binary version.
```
$ dev/change-scala-version.sh 2.12
$ dev/test-dependencies.sh
Using `mvn` from path: /usr/local/bin/mvn
Using `mvn` from path: /usr/local/bin/mvn
Performing Maven install for hadoop-2.6
Using `mvn` from path: /usr/local/bin/mvn
[ERROR] Failed to execute goal on project spark-streaming-kafka-0-8_2.12: Could not resolve dependencies for project org.apache.spark:spark-streaming-kafka-0-8_2.12:jar:spark-335572: Could not find artifact org.apache.kafka:kafka_2.12:jar:0.8.2.1 in central (https://repo.maven.apache.org/maven2) -> [Help 1]
```
## How was this patch tested?
Manually do `dev/change-scala-version.sh 2.12` and `dev/test-dependencies.sh`.
The script should show `DO NOT MATCH` message instead of Maven `[ERROR]`.
```
$ dev/test-dependencies.sh
Using `mvn` from path: /usr/local/bin/mvn
...
Generating dependency manifest for hadoop-3.1
Using `mvn` from path: /usr/local/bin/mvn
Spark's published dependencies DO NOT MATCH the manifest file (dev/spark-deps).
To update the manifest file, run './dev/test-dependencies.sh --replace-manifest'.
diff --git a/dev/deps/spark-deps-hadoop-2.6 b/dev/pr-deps/spark-deps-hadoop-2.6
...
```
Closes #24445 from dongjoon-hyun/SPARK-27550.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
dev/test-dependencies.sh | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/dev/test-dependencies.sh b/dev/test-dependencies.sh
index 2fbd6b5..5a4a894 100755
--- a/dev/test-dependencies.sh
+++ b/dev/test-dependencies.sh
@@ -29,7 +29,6 @@ export LC_ALL=C
# TODO: This would be much nicer to do in SBT, once SBT supports Maven-style resolution.
# NOTE: These should match those in the release publishing script
-HADOOP2_MODULE_PROFILES="-Phive-thriftserver -Pmesos -Pkafka-0-8 -Pkubernetes -Pyarn -Pflume -Phive"
MVN="build/mvn"
HADOOP_PROFILES=(
hadoop-2.6
@@ -37,6 +36,16 @@ HADOOP_PROFILES=(
hadoop-3.1
)
+SCALA_VERSION=$("$MVN" help:evaluate -Dexpression=scala.binary.version $@ 2>/dev/null\
+ | grep -v "INFO"\
+ | grep -v "WARNING"\
+ | tail -n 1)
+if [ "$SCALA_VERSION" = "2.11" ]; then
+ HADOOP2_MODULE_PROFILES="-Phive-thriftserver -Pmesos -Pkafka-0-8 -Pkubernetes -Pyarn -Pflume -Phive"
+else
+ HADOOP2_MODULE_PROFILES="-Phive-thriftserver -Pmesos -Pkubernetes -Pyarn -Pflume -Phive"
+fi
+
# We'll switch the version to a temp. one, publish POMs using that new version, then switch back to
# the old version. We need to do this because the `dependency:build-classpath` task needs to
# resolve Spark's internal submodule dependencies.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org