You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/04/23 22:13:03 UTC
[GitHub] [spark] dongjoon-hyun opened a new pull request #24445:
[SPARK-27550][TEST][BRANCH-2.4] Fix `test-dependencies.sh` not to use
`kafka-0-8`profile for Scala-2.12
dongjoon-hyun opened a new pull request #24445: [SPARK-27550][TEST][BRANCH-2.4] Fix `test-dependencies.sh` not to use `kafka-0-8`profile for Scala-2.12
URL: https://github.com/apache/spark/pull/24445
## What changes were proposed in this pull request?
Since SPARK-27274 deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more.
Kakfa 0.8 doesn't have Scala-2.12 artifacts, e.g., `org.apache.kafka:kafka_2.12:jar:0.8.2.1`. This PR aims to fix `test-dependencies.sh` script to understand Scala binary version.
```
$ dev/change-scala-version.sh 2.12
$ dev/test-dependencies.sh
Using `mvn` from path: /usr/local/bin/mvn
Using `mvn` from path: /usr/local/bin/mvn
Performing Maven install for hadoop-2.6
Using `mvn` from path: /usr/local/bin/mvn
[ERROR] Failed to execute goal on project spark-streaming-kafka-0-8_2.12: Could not resolve dependencies for project org.apache.spark:spark-streaming-kafka-0-8_2.12:jar:spark-335572: Could not find artifact org.apache.kafka:kafka_2.12:jar:0.8.2.1 in central (https://repo.maven.apache.org/maven2) -> [Help 1]
```
## How was this patch tested?
Manually do `dev/change-scala-version.sh 2.12` and `dev/test-dependencies.sh`.
The script should show `DO NOT MATCH` message instead of Maven `[ERROR]`.
```
$ dev/test-dependencies.sh
Using `mvn` from path: /usr/local/bin/mvn
...
Generating dependency manifest for hadoop-3.1
Using `mvn` from path: /usr/local/bin/mvn
Spark's published dependencies DO NOT MATCH the manifest file (dev/spark-deps).
To update the manifest file, run './dev/test-dependencies.sh --replace-manifest'.
diff --git a/dev/deps/spark-deps-hadoop-2.6 b/dev/pr-deps/spark-deps-hadoop-2.6
...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org