You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2018/09/17 10:59:37 UTC
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/22441
[SPARK-25445][BUILD] the release script should be able to publish a scala-2.12 build
## What changes were proposed in this pull request?
update the package and publish steps, to support scala 2.12
## How was this patch tested?
manual test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark scala
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22441.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22441
----
commit 6fdd041a1eaf29d3e9bfca0703dbeb6a95fb76d7
Author: Wenchen Fan <we...@...>
Date: 2018-09-17T07:07:00Z
publish scala-2.12 build
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22441
**[Test build #96168 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96168/testReport)** for PR 22441 at commit [`110913a`](https://github.com/apache/spark/commit/110913a6d4a10087ee31247d5357a0432ffbd8b7).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3181/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218304060
--- Diff: dev/create-release/release-build.sh ---
@@ -183,8 +188,17 @@ if [[ "$1" == "package" ]]; then
# Updated for each binary build
make_binary_release() {
NAME=$1
- FLAGS="$MVN_EXTRA_OPTS -B $SCALA_2_11_PROFILES $BASE_RELEASE_PROFILES $2"
- BUILD_PACKAGE=$3
+ SCALA_VERSION=$2
+ SCALA_PROFILES=
+ if [[ SCALA_VERSION == "2.10" ]]; then
--- End diff --
People may run the release script in master branch to release Spark 2.1. I'd like to keep the handling of scala 2.10 until we announce that we no longer support Spark version prior to 2.3.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22441
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218486554
--- Diff: dev/create-release/release-build.sh ---
@@ -111,13 +111,17 @@ fi
# different versions of Scala are supported.
BASE_PROFILES="-Pmesos -Pyarn"
PUBLISH_SCALA_2_10=0
+PUBLISH_SCALA_2_12=0
SCALA_2_10_PROFILES="-Pscala-2.10"
SCALA_2_11_PROFILES=
-SCALA_2_12_PROFILES="-Pscala-2.12"
+SCALA_2_12_PROFILES="-Pscala-2.12 -Pkafka-0-8"
--- End diff --
Hm, it shouldn't, in that there's no 2.12 artifacts for Kafka 0.8
https://search.maven.org/search?q=g:org.apache.kafka%20AND%20a:kafka_2.12&core=gav
vs
```
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_${scala.binary.version}</artifactId>
<version>0.8.2.1</version>
```
Maybe it actually fails at the end of the 2.12 build, after a lot of it has succeeded
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22441
**[Test build #96167 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96167/testReport)** for PR 22441 at commit [`b5237ec`](https://github.com/apache/spark/commit/b5237ecfed1831305cec482ea982c0050e0e3970).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218303693
--- Diff: dev/create-release/release-build.sh ---
@@ -111,13 +111,17 @@ fi
# different versions of Scala are supported.
BASE_PROFILES="-Pmesos -Pyarn"
PUBLISH_SCALA_2_10=0
+PUBLISH_SCALA_2_12=0
SCALA_2_10_PROFILES="-Pscala-2.10"
SCALA_2_11_PROFILES=
SCALA_2_12_PROFILES="-Pscala-2.12"
--- End diff --
Here we may miss the arg "-Pkafka-0-8"
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218079177
--- Diff: dev/create-release/release-build.sh ---
@@ -112,11 +112,16 @@ fi
# different versions of Scala are supported.
BASE_PROFILES="-Pmesos -Pyarn"
PUBLISH_SCALA_2_10=0
+PUBLISH_SCALA_2_12=0
SCALA_2_10_PROFILES="-Pscala-2.10"
SCALA_2_11_PROFILES=
SCALA_2_12_PROFILES="-Pscala-2.12"
-if [[ $SPARK_VERSION > "2.3" ]]; then
+if [[ $SPARK_VERSION > "2.4" ]]; then
+ BASE_PROFILES="$BASE_PROFILES -Pkubernetes -Pflume"
+ SCALA_2_11_PROFILES="-Pkafka-0-8"
--- End diff --
I suppose you could separate these two if conditions, to avoid repeating the two lines above. Optional.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3184/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3154/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22441
**[Test build #96131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96131/testReport)** for PR 22441 at commit [`6fdd041`](https://github.com/apache/spark/commit/6fdd041a1eaf29d3e9bfca0703dbeb6a95fb76d7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3180/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22441
**[Test build #96167 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96167/testReport)** for PR 22441 at commit [`b5237ec`](https://github.com/apache/spark/commit/b5237ecfed1831305cec482ea982c0050e0e3970).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96168/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22441
**[Test build #96171 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96171/testReport)** for PR 22441 at commit [`110913a`](https://github.com/apache/spark/commit/110913a6d4a10087ee31247d5357a0432ffbd8b7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22441
**[Test build #96168 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96168/testReport)** for PR 22441 at commit [`110913a`](https://github.com/apache/spark/commit/110913a6d4a10087ee31247d5357a0432ffbd8b7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22441
Is it possible to test this on Jenkins?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96131/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218465898
--- Diff: dev/create-release/release-build.sh ---
@@ -111,13 +111,17 @@ fi
# different versions of Scala are supported.
BASE_PROFILES="-Pmesos -Pyarn"
PUBLISH_SCALA_2_10=0
+PUBLISH_SCALA_2_12=0
SCALA_2_10_PROFILES="-Pscala-2.10"
SCALA_2_11_PROFILES=
-SCALA_2_12_PROFILES="-Pscala-2.12"
+SCALA_2_12_PROFILES="-Pscala-2.12 -Pkafka-0-8"
--- End diff --
So we only set the kafka 0.8 profile if and only the spark version > 2.3 and it's a scala 2.11 build?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218078222
--- Diff: dev/create-release/release-build.sh ---
@@ -411,15 +435,15 @@ if [[ "$1" == "publish-release" ]]; then
-DskipTests $PUBLISH_PROFILES $SCALA_2_10_PROFILES clean install
fi
- #./dev/change-scala-version.sh 2.12
- #$MVN -DzincPort=$ZINC_PORT -Dmaven.repo.local=$tmp_repo \
- # -DskipTests $SCALA_2_12_PROFILES §$PUBLISH_PROFILES clean install
+ if ! is_dry_run && [[ $PUBLISH_SCALA_2_12 = 1 ]]; then
+ ./dev/change-scala-version.sh 2.12
+ $MVN -DzincPort=$((ZINC_PORT + 1)) -Dmaven.repo.local=$tmp_repo -Dscala-2.12 \
+ -DskipTests $PUBLISH_PROFILES $SCALA_2_12_PROFILES clean install
--- End diff --
Not sure, but it is probably tiday to run `./dev/change-scala-version.sh 2.11` or some kind of git revert of the modifications locally here as a cleanup step, after the build.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218481751
--- Diff: dev/create-release/release-build.sh ---
@@ -111,13 +111,17 @@ fi
# different versions of Scala are supported.
BASE_PROFILES="-Pmesos -Pyarn"
PUBLISH_SCALA_2_10=0
+PUBLISH_SCALA_2_12=0
SCALA_2_10_PROFILES="-Pscala-2.10"
SCALA_2_11_PROFILES=
-SCALA_2_12_PROFILES="-Pscala-2.12"
+SCALA_2_12_PROFILES="-Pscala-2.12 -Pkafka-0-8"
--- End diff --
somehow it works when I run it... Anyway I'll send a followup PR to change it back.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96171/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96167/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218166616
--- Diff: dev/create-release/release-build.sh ---
@@ -183,8 +188,17 @@ if [[ "$1" == "package" ]]; then
# Updated for each binary build
make_binary_release() {
NAME=$1
- FLAGS="$MVN_EXTRA_OPTS -B $SCALA_2_11_PROFILES $BASE_RELEASE_PROFILES $2"
- BUILD_PACKAGE=$3
+ SCALA_VERSION=$2
+ SCALA_PROFILES=
+ if [[ SCALA_VERSION == "2.10" ]]; then
+ SCALA_PROFILES=$SCALA_2_10_PROFILES
--- End diff --
Use quotes after the `=`; also in other places.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218464733
--- Diff: dev/create-release/release-build.sh ---
@@ -414,15 +437,15 @@ if [[ "$1" == "publish-release" ]]; then
-DskipTests $PUBLISH_PROFILES $SCALA_2_10_PROFILES clean install
fi
- #./dev/change-scala-version.sh 2.12
- #$MVN -DzincPort=$ZINC_PORT -Dmaven.repo.local=$tmp_repo \
- # -DskipTests $SCALA_2_12_PROFILES §$PUBLISH_PROFILES clean install
+ if ! is_dry_run && [[ $PUBLISH_SCALA_2_12 = 1 ]]; then
+ ./dev/change-scala-version.sh 2.12
+ $MVN -DzincPort=$((ZINC_PORT + 2)) -Dmaven.repo.local=$tmp_repo -Dscala-2.12 \
+ -DskipTests $PUBLISH_PROFILES $SCALA_2_12_PROFILES clean install
+ fi
--- End diff --
We tried to do it before but it's commented out: https://github.com/apache/spark/pull/22441/files#diff-01ca42240614718522afde4d4885b40dL424
I didn't look into it and seems it's OK as the release script is able to do scala 2.10 build.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218301766
--- Diff: dev/create-release/release-build.sh ---
@@ -111,13 +111,17 @@ fi
# different versions of Scala are supported.
BASE_PROFILES="-Pmesos -Pyarn"
PUBLISH_SCALA_2_10=0
+PUBLISH_SCALA_2_12=0
SCALA_2_10_PROFILES="-Pscala-2.10"
SCALA_2_11_PROFILES=
SCALA_2_12_PROFILES="-Pscala-2.12"
if [[ $SPARK_VERSION > "2.3" ]]; then
BASE_PROFILES="$BASE_PROFILES -Pkubernetes -Pflume"
SCALA_2_11_PROFILES="-Pkafka-0-8"
+ if [[ $SPARK_VERSION > "2.4" ]]; then
--- End diff --
nit: I think we can move this branch out of the scope of `if [[ $SPARK_VERSION > "2.3" ]]; then`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218078496
--- Diff: dev/create-release/release-build.sh ---
@@ -183,8 +188,17 @@ if [[ "$1" == "package" ]]; then
# Updated for each binary build
make_binary_release() {
NAME=$1
- FLAGS="$MVN_EXTRA_OPTS -B $SCALA_2_11_PROFILES $BASE_RELEASE_PROFILES $2"
- BUILD_PACKAGE=$3
+ SCALA_VERSION=$2
+ SCALA_PROFILES=
+ if [[ SCALA_VERSION == "2.10" ]]; then
--- End diff --
I think the 2.10-related lines can be deleted as of branch 2.4 and beyond, as there are no 2.10 builds since 2.3. Optional.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22441
Since I've verified it manually, I'm merging it to master/2.4, thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218304237
--- Diff: dev/create-release/release-build.sh ---
@@ -111,13 +111,17 @@ fi
# different versions of Scala are supported.
BASE_PROFILES="-Pmesos -Pyarn"
PUBLISH_SCALA_2_10=0
+PUBLISH_SCALA_2_12=0
SCALA_2_10_PROFILES="-Pscala-2.10"
SCALA_2_11_PROFILES=
SCALA_2_12_PROFILES="-Pscala-2.12"
if [[ $SPARK_VERSION > "2.3" ]]; then
BASE_PROFILES="$BASE_PROFILES -Pkubernetes -Pflume"
SCALA_2_11_PROFILES="-Pkafka-0-8"
+ if [[ $SPARK_VERSION > "2.4" ]]; then
--- End diff --
then we need to set the BASE_PROFILES and SCALA_2_11_PROFILES again.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22441
**[Test build #96171 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96171/testReport)** for PR 22441 at commit [`110913a`](https://github.com/apache/spark/commit/110913a6d4a10087ee31247d5357a0432ffbd8b7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22441
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218458224
--- Diff: dev/create-release/release-build.sh ---
@@ -414,15 +437,15 @@ if [[ "$1" == "publish-release" ]]; then
-DskipTests $PUBLISH_PROFILES $SCALA_2_10_PROFILES clean install
fi
- #./dev/change-scala-version.sh 2.12
- #$MVN -DzincPort=$ZINC_PORT -Dmaven.repo.local=$tmp_repo \
- # -DskipTests $SCALA_2_12_PROFILES §$PUBLISH_PROFILES clean install
+ if ! is_dry_run && [[ $PUBLISH_SCALA_2_12 = 1 ]]; then
+ ./dev/change-scala-version.sh 2.12
+ $MVN -DzincPort=$((ZINC_PORT + 2)) -Dmaven.repo.local=$tmp_repo -Dscala-2.12 \
+ -DskipTests $PUBLISH_PROFILES $SCALA_2_12_PROFILES clean install
+ fi
--- End diff --
I still think we want a cleanup step that always changes the scala version back to the 'default' of 2.11 right now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218470455
--- Diff: dev/create-release/release-build.sh ---
@@ -111,13 +111,17 @@ fi
# different versions of Scala are supported.
BASE_PROFILES="-Pmesos -Pyarn"
PUBLISH_SCALA_2_10=0
+PUBLISH_SCALA_2_12=0
SCALA_2_10_PROFILES="-Pscala-2.10"
SCALA_2_11_PROFILES=
-SCALA_2_12_PROFILES="-Pscala-2.12"
+SCALA_2_12_PROFILES="-Pscala-2.12 -Pkafka-0-8"
--- End diff --
Yes that's correct. It shouldn't be built for Scala 2.10 or 2.12, and did not exist before 2.3.0.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22441: [SPARK-25445][BUILD] the release script should be...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22441#discussion_r218471225
--- Diff: dev/create-release/release-build.sh ---
@@ -414,15 +437,15 @@ if [[ "$1" == "publish-release" ]]; then
-DskipTests $PUBLISH_PROFILES $SCALA_2_10_PROFILES clean install
fi
- #./dev/change-scala-version.sh 2.12
- #$MVN -DzincPort=$ZINC_PORT -Dmaven.repo.local=$tmp_repo \
- # -DskipTests $SCALA_2_12_PROFILES §$PUBLISH_PROFILES clean install
+ if ! is_dry_run && [[ $PUBLISH_SCALA_2_12 = 1 ]]; then
+ ./dev/change-scala-version.sh 2.12
+ $MVN -DzincPort=$((ZINC_PORT + 2)) -Dmaven.repo.local=$tmp_repo -Dscala-2.12 \
+ -DskipTests $PUBLISH_PROFILES $SCALA_2_12_PROFILES clean install
+ fi
--- End diff --
I believe I added it, commented out, for the time when we enabled a 2.12 build. My concern is just that it leaves the checkout in a state that builds for 2.12. If you tried a 2.11 build afterwards it would fail. But I suppose it would clearly fail. I suppose it will work here as it's the last thing this tries to do, so the end state doesn't matter. Tidy, but optional to set it back.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22441
The test failure is a known issue: https://issues.apache.org/jira/browse/SPARK-25456
Actually PR builld can't verify release scripts, so we don't need to wait for it...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22441
I'm not sure if we can trigger a [packaging job](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.4-package/) with a PR , cc @shaneknapp @JoshRosen
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22441
cc @srowen @vanzin @shaneknapp @felixcheung @jerryshao
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22441
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22441: [SPARK-25445][BUILD] the release script should be able t...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22441
**[Test build #96131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96131/testReport)** for PR 22441 at commit [`6fdd041`](https://github.com/apache/spark/commit/6fdd041a1eaf29d3e9bfca0703dbeb6a95fb76d7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org