You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2021/07/02 18:02:00 UTC
[spark] branch branch-3.2 updated: [SPARK-35990][BUILD] Remove
avro-sbt plugin dependency
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 9b0971c [SPARK-35990][BUILD] Remove avro-sbt plugin dependency
9b0971c is described below
commit 9b0971c40ebbc5309f6684edfef3f7e219c9a4b2
Author: Kousuke Saruta <sa...@oss.nttdata.com>
AuthorDate: Fri Jul 2 11:00:59 2021 -0700
[SPARK-35990][BUILD] Remove avro-sbt plugin dependency
### What changes were proposed in this pull request?
This PR removes sbt-avro plugin dependency.
In the current master, Build with SBT depends on the plugin but it seems never used.
Originally, the plugin was introduced for `flume-sink` in SPARK-1729 (#807) but `flume-sink` is no longer in Spark repository.
After SBT was upgraded to 1.x in SPARK-21708 (#29286), `avroGenerate` part was introduced in `object SQL` in `SparkBuild.scala`.
It's confusable but I understand `Test / avroGenerate := (Compile / avroGenerate).value` is for suppressing sbt-avro for `sql` sub-module.
In fact, Test/compile will fail if `Test / avroGenerate :=(Compile / avroGenerate).value` is commented out.
`sql` sub-module contains `parquet-compat.avpr` and `parquet-compat.avdl` but according to `sql/core/src/test/README.md`, they are intended to be handled by `gen-avro.sh`.
Also, in terms of Maven build, there seems to be no definition to handle `*.avpr` or `*.avdl`.
Based on the above, I think we can remove `sbt-avro`.
### Why are the changes needed?
If `sbt-avro` is really no longer used, it's confusable that `sbt-avro` related configurations are in `SparkBuild.scala`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
GA.
Closes #33190 from sarutak/remove-avro-from-sbt.
Authored-by: Kousuke Saruta <sa...@oss.nttdata.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 6c4616b2ac75f30db63c49ab91a784285b147b11)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
project/SparkBuild.scala | 7 +------
project/plugins.sbt | 3 ---
2 files changed, 1 insertion(+), 9 deletions(-)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index b1d6668..3480f4f 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -737,9 +737,6 @@ object Catalyst {
}
object SQL {
-
- import sbtavro.SbtAvro.autoImport._
-
lazy val settings = Seq(
(console / initialCommands) :=
"""
@@ -761,10 +758,8 @@ object SQL {
|import sqlContext.implicits._
|import sqlContext._
""".stripMargin,
- (console / cleanupCommands) := "sc.stop()",
- Test / avroGenerate := (Compile / avroGenerate).value
+ (console / cleanupCommands) := "sc.stop()"
)
-
}
object Hive {
diff --git a/project/plugins.sbt b/project/plugins.sbt
index 8130ed0..e733f86 100644
--- a/project/plugins.sbt
+++ b/project/plugins.sbt
@@ -33,9 +33,6 @@ addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.9.2")
addSbtPlugin("com.eed3si9n" % "sbt-unidoc" % "0.4.3")
-addSbtPlugin("com.cavorite" % "sbt-avro" % "2.1.1")
-libraryDependencies += "org.apache.avro" % "avro-compiler" % "1.10.2"
-
addSbtPlugin("io.spray" % "sbt-revolver" % "0.9.1")
libraryDependencies += "org.ow2.asm" % "asm" % "9.1"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org