[spark] branch master updated: [SPARK-38040][BUILD] Enable binary compatibility check for APIs in Catalyst, KVStore and Avro modules

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 1930116  [SPARK-38040][BUILD] Enable binary compatibility check for APIs in Catalyst, KVStore and Avro modules
1930116 is described below

commit 193011632ba41dc4035460c429374981a8ebe0b7
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Thu Jan 27 13:36:27 2022 +0900

    [SPARK-38040][BUILD] Enable binary compatibility check for APIs in Catalyst, KVStore and Avro modules
    
    ### What changes were proposed in this pull request?
    
    We don't currently run binary compatibility check in below modules:
    
    ```
    [info] spark-parent: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-network-common: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-tags: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-unsafe: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-network-shuffle: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-kvstore: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-tools: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-token-provider-kafka-0-10: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-streaming-kafka-0-10-assembly: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-catalyst: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-repl: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-avro: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-sql-kafka-0-10: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-hive: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-assembly: mimaPreviousArtifacts not set, not analyzing binary compatibility
    [info] spark-examples: mimaPreviousArtifacts not set, not analyzing binary compatibility
    ```
    
    However, there are some APIs under these modules. For example, https://github.com/apache/spark/blob/master/external/avro/src/main/scala/org/apache/spark/sql/avro/functions.scala for Avro,  https://github.com/apache/spark/tree/master/common/kvstore/src/main/java/org/apache/spark/util/kvstore for KVStore (to be API), and https://github.com/apache/spark/tree/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector for Catalyst
    
    ### Why are the changes needed?
    
    To detect binary compatibility.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, dev-only.
    
    ### How was this patch tested?
    
    Manually tested via running `dev/mima`.
    
    Closes #35339 from HyukjinKwon/SPARK-38040.
    
    Authored-by: Hyukjin Kwon <gu...@apache.org>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 project/MimaExcludes.scala | 6 ++++++
 project/SparkBuild.scala   | 4 ++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index b985f95..f77bc5c 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -66,6 +66,12 @@ object MimaExcludes {
     ProblemFilters.exclude[Problem]("org.apache.spark.sql.catalyst.*"),
     ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
     ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
+    ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+    // DSv2 catalog and expression APIs are unstable yet. We should enable this back.
+    ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.catalog.*"),
+    ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.expressions.*"),
+    // Avro source implementation is internal.
+    ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
 
     // [SPARK-34848][CORE] Add duration to TaskMetricDistributions
     ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 02ffa23..ad9aef5 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -376,8 +376,8 @@ object SparkBuild extends PomBuild {
 
   val mimaProjects = allProjects.filterNot { x =>
     Seq(
-      spark, hive, hiveThriftServer, catalyst, repl, networkCommon, networkShuffle, networkYarn,
-      unsafe, tags, tokenProviderKafka010, sqlKafka010, kvstore, avro
+      spark, hive, hiveThriftServer, repl, networkCommon, networkShuffle, networkYarn,
+      unsafe, tags, tokenProviderKafka010, sqlKafka010
     ).contains(x)
   }
 

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org