You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2018/01/14 08:09:14 UTC
[GitHub] spark pull request #20264: [SPARK-23070] Bump previousSparkVersion in MimaBu...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/20264
[SPARK-23070] Bump previousSparkVersion in MimaBuild.scala to be 2.2.0 [WIP]
## What changes were proposed in this pull request?
Bump previousSparkVersion in MimaBuild.scala to be 2.2.0
Add the new exclusion to `v23excludes` in `MimaExcludes`
[TODO]: un-exclude any new subprojects / artifacts that were added in 2.2.0
## How was this patch tested?
The existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark bump22
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20264.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20264
----
commit 70446962f1a462143397d67994b4cf15d68acfda
Author: gatorsmile <ga...@...>
Date: 2018-01-14T08:04:57Z
fix.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20264
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20264
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86115/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20264
**[Test build #86119 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86119/testReport)** for PR 20264 at commit [`7044696`](https://github.com/apache/spark/commit/70446962f1a462143397d67994b4cf15d68acfda).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20264
Thanks! Merged to master/2.3
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20264
**[Test build #86115 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86115/testReport)** for PR 20264 at commit [`7044696`](https://github.com/apache/spark/commit/70446962f1a462143397d67994b4cf15d68acfda).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20264
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86119/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20264: [SPARK-23070] Bump previousSparkVersion in MimaBu...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20264#discussion_r161391276
--- Diff: project/MimaExcludes.scala ---
@@ -102,7 +102,40 @@ object MimaExcludes {
// [SPARK-21087] CrossValidator, TrainValidationSplit expose sub models after fitting: Scala
ProblemFilters.exclude[FinalClassProblem]("org.apache.spark.ml.tuning.CrossValidatorModel$CrossValidatorModelWriter"),
- ProblemFilters.exclude[FinalClassProblem]("org.apache.spark.ml.tuning.TrainValidationSplitModel$TrainValidationSplitModelWriter")
+ ProblemFilters.exclude[FinalClassProblem]("org.apache.spark.ml.tuning.TrainValidationSplitModel$TrainValidationSplitModelWriter"),
+
+ // [SPARK-21728][CORE] Allow SparkSubmit to use Logging
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkSubmit.downloadFileList"),
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkSubmit.downloadFile"),
+
+ // [SPARK-21714][CORE][YARN] Avoiding re-uploading remote resources in yarn client mode
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment"),
+
+ // [SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0
+ ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.network.util.AbstractFileRegion.transfered"),
--- End diff --
lgtm. This is inside the network module.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20264: [SPARK-23070] Bump previousSparkVersion in MimaBu...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20264#discussion_r161418395
--- Diff: project/MimaBuild.scala ---
@@ -88,7 +88,7 @@ object MimaBuild {
def mimaSettings(sparkHome: File, projectRef: ProjectRef) = {
val organization = "org.apache.spark"
- val previousSparkVersion = "2.0.0"
+ val previousSparkVersion = "2.2.0"
--- End diff --
Yeah, 2.2.0 is right.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20264: [SPARK-23070] Bump previousSparkVersion in MimaBu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20264#discussion_r161412925
--- Diff: project/MimaBuild.scala ---
@@ -88,7 +88,7 @@ object MimaBuild {
def mimaSettings(sparkHome: File, projectRef: ProjectRef) = {
val organization = "org.apache.spark"
- val previousSparkVersion = "2.0.0"
+ val previousSparkVersion = "2.2.0"
--- End diff --
I checked the change history, we always use x.y.0 release. cc @JoshRosen
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20264: [SPARK-23070] Bump previousSparkVersion in MimaBu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20264#discussion_r161390389
--- Diff: project/MimaExcludes.scala ---
@@ -102,7 +102,40 @@ object MimaExcludes {
// [SPARK-21087] CrossValidator, TrainValidationSplit expose sub models after fitting: Scala
ProblemFilters.exclude[FinalClassProblem]("org.apache.spark.ml.tuning.CrossValidatorModel$CrossValidatorModelWriter"),
- ProblemFilters.exclude[FinalClassProblem]("org.apache.spark.ml.tuning.TrainValidationSplitModel$TrainValidationSplitModelWriter")
+ ProblemFilters.exclude[FinalClassProblem]("org.apache.spark.ml.tuning.TrainValidationSplitModel$TrainValidationSplitModelWriter"),
+
+ // [SPARK-21728][CORE] Allow SparkSubmit to use Logging
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkSubmit.downloadFileList"),
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkSubmit.downloadFile"),
+
+ // [SPARK-21714][CORE][YARN] Avoiding re-uploading remote resources in yarn client mode
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment"),
+
+ // [SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0
+ ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.network.util.AbstractFileRegion.transfered"),
+
+ // [SPARK-20643][CORE] Add listener implementation to collect app state
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskData.<init>$default$5"),
+
+ // [SPARK-20648][CORE] Port JobsTab and StageTab to the new UI backend
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskData.<init>$default$12"),
+
+ // [SPARK-21462][SS] Added batchId to StreamingQueryProgress.json
+ // [SPARK-21409][SS] Expose state store memory usage in SQL metrics and progress updates
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StateOperatorProgress.this"),
--- End diff --
This break is OK, right? @zsxwing @tdas
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20264
**[Test build #86115 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86115/testReport)** for PR 20264 at commit [`7044696`](https://github.com/apache/spark/commit/70446962f1a462143397d67994b4cf15d68acfda).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20264
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20264
cc @JoshRosen @sameeragarwal
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20264: [SPARK-23070] Bump previousSparkVersion in MimaBu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/20264#discussion_r161406899
--- Diff: project/MimaBuild.scala ---
@@ -88,7 +88,7 @@ object MimaBuild {
def mimaSettings(sparkHome: File, projectRef: ProjectRef) = {
val organization = "org.apache.spark"
- val previousSparkVersion = "2.0.0"
+ val previousSparkVersion = "2.2.0"
--- End diff --
does this matter if this is 2.2.1 (latest release)?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20264: [SPARK-23070] Bump previousSparkVersion in MimaBu...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20264
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20264
**[Test build #86119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86119/testReport)** for PR 20264 at commit [`7044696`](https://github.com/apache/spark/commit/70446962f1a462143397d67994b4cf15d68acfda).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20264: [SPARK-23070] Bump previousSparkVersion in MimaBu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20264#discussion_r161390352
--- Diff: project/MimaExcludes.scala ---
@@ -102,7 +102,40 @@ object MimaExcludes {
// [SPARK-21087] CrossValidator, TrainValidationSplit expose sub models after fitting: Scala
ProblemFilters.exclude[FinalClassProblem]("org.apache.spark.ml.tuning.CrossValidatorModel$CrossValidatorModelWriter"),
- ProblemFilters.exclude[FinalClassProblem]("org.apache.spark.ml.tuning.TrainValidationSplitModel$TrainValidationSplitModelWriter")
+ ProblemFilters.exclude[FinalClassProblem]("org.apache.spark.ml.tuning.TrainValidationSplitModel$TrainValidationSplitModelWriter"),
+
+ // [SPARK-21728][CORE] Allow SparkSubmit to use Logging
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkSubmit.downloadFileList"),
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkSubmit.downloadFile"),
+
+ // [SPARK-21714][CORE][YARN] Avoiding re-uploading remote resources in yarn client mode
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment"),
+
+ // [SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0
+ ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.network.util.AbstractFileRegion.transfered"),
--- End diff --
Please check whether the related changes are intentional to break compatibility.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20264: [SPARK-23070] Bump previousSparkVersion in MimaBuild.sca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20264
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org