You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2021/11/19 04:54:17 UTC
[spark] branch master updated: [SPARK-35672][FOLLOWUP][TESTS] Add an exclusion rule to MimaExcludes.scala for Scala 2.13
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 145436cb [SPARK-35672][FOLLOWUP][TESTS] Add an exclusion rule to MimaExcludes.scala for Scala 2.13
145436cb is described below
commit 145436cb37ee77ba28aac39be2e7d4c379836360
Author: Kousuke Saruta <sa...@oss.nttdata.com>
AuthorDate: Thu Nov 18 20:53:23 2021 -0800
[SPARK-35672][FOLLOWUP][TESTS] Add an exclusion rule to MimaExcludes.scala for Scala 2.13
### What changes were proposed in this pull request?
This PR mitigate an issue that MiMa fails for Scala 2.13 after SPARK-35672 (#34120).
```
$ dev/change-scala-version.sh 2.13
$ dev/mima
...
[error] spark-core: Failed binary compatibility check against org.apache.spark:spark-core_2.13:3.2.0! Found 8 potential problems (filtered 905)
[error] * method userClassPath()scala.collection.mutable.ListBuffer in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments does not have a correspondent in current version
[error] filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.userClassPath")
[error] * method copy(java.lang.String,java.lang.String,java.lang.String,java.lang.String,Int,java.lang.String,scala.Option,scala.collection.mutable.ListBuffer,scala.Option,Int)org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments does not have a correspondent in current version
[error] filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.copy")
[error] * synthetic method copy$default$10()Int in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments does not have a correspondent in current version
[error] filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.copy$default$10")
[error] * synthetic method copy$default$8()scala.collection.mutable.ListBuffer in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments has a different result type in current version, where it is scala.Option rather than scala.collection.mutable.ListBuffer
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.copy$default$8")
[error] * synthetic method copy$default$9()scala.Option in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments has a different result type in current version, where it is Int rather than scala.Option
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.copy$default$9")
[error] * method this(java.lang.String,java.lang.String,java.lang.String,java.lang.String,Int,java.lang.String,scala.Option,scala.collection.mutable.ListBuffer,scala.Option,Int)Unit in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments does not have a correspondent in current version
[error] filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.this")
[error] * the type hierarchy of object org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments is different in current version. Missing types {scala.runtime.AbstractFunction10}
[error] filter with: ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend$Arguments$")
[error] * method apply(java.lang.String,java.lang.String,java.lang.String,java.lang.String,Int,java.lang.String,scala.Option,scala.collection.mutable.ListBuffer,scala.Option,Int)org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments in object org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments does not have a correspondent in current version
[error] filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.apply")
...
```
It's funny that the class `Arguments` is `public` but it's a member class of `CoarseGrainedExecutorBackend` which is `package private` and MiMa doesn't raise error for Scala 2.12, but adding an exclusion rule is one workaround.
### Why are the changes needed?
To keep the build stable.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Confirmed MiMa passed.
```
$ dev/change-scala-version.sh 2.13
$ dev/mima
```
Closes #34649 from sarutak/followup-SPARK-35672-mima.
Authored-by: Kousuke Saruta <sa...@oss.nttdata.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
project/MimaExcludes.scala | 3 +++
1 file changed, 3 insertions(+)
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index 487b319..15df3d4 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -36,6 +36,9 @@ object MimaExcludes {
// Exclude rules for 3.3.x from 3.2.0
lazy val v33excludes = v32excludes ++ Seq(
+ // [SPARK-35672][CORE][YARN] Pass user classpath entries to executors using config instead of command line
+ // This is necessary for Scala 2.13.
+ ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.*"),
)
// Exclude rules for 3.2.x from 3.1.1
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org