You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by li...@apache.org on 2018/07/25 15:42:49 UTC
spark git commit: [SPARK-24768][FOLLOWUP][SQL] Avro migration
followup: change artifactId to spark-avro
Repository: spark
Updated Branches:
refs/heads/master 7a5fd4a91 -> c44eb561e
[SPARK-24768][FOLLOWUP][SQL] Avro migration followup: change artifactId to spark-avro
## What changes were proposed in this pull request?
After rethinking on the artifactId, I think it should be `spark-avro` instead of `spark-sql-avro`, which is simpler, and consistent with the previous artifactId. I think we need to change it before Spark 2.4 release.
Also a tiny change: use `spark.sessionState.newHadoopConf()` to get the hadoop configuration, thus the related hadoop configurations in SQLConf will come into effect.
## How was this patch tested?
Unit test
Author: Gengliang Wang <ge...@databricks.com>
Closes #21866 from gengliangwang/avro_followup.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c44eb561
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c44eb561
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c44eb561
Branch: refs/heads/master
Commit: c44eb561ec371af0405710d2e9358f9797655145
Parents: 7a5fd4a
Author: Gengliang Wang <ge...@databricks.com>
Authored: Wed Jul 25 08:42:45 2018 -0700
Committer: Xiao Li <ga...@gmail.com>
Committed: Wed Jul 25 08:42:45 2018 -0700
----------------------------------------------------------------------
external/avro/pom.xml | 2 +-
.../src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/c44eb561/external/avro/pom.xml
----------------------------------------------------------------------
diff --git a/external/avro/pom.xml b/external/avro/pom.xml
index ad7df1f..8f118ba 100644
--- a/external/avro/pom.xml
+++ b/external/avro/pom.xml
@@ -25,7 +25,7 @@
<relativePath>../../pom.xml</relativePath>
</parent>
- <artifactId>spark-sql-avro_2.11</artifactId>
+ <artifactId>spark-avro_2.11</artifactId>
<properties>
<sbt.project.name>avro</sbt.project.name>
</properties>
http://git-wip-us.apache.org/repos/asf/spark/blob/c44eb561/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
----------------------------------------------------------------------
diff --git a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
index b043252..c6b3c13 100755
--- a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
+++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
@@ -56,7 +56,7 @@ private[avro] class AvroFileFormat extends FileFormat with DataSourceRegister {
spark: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
- val conf = spark.sparkContext.hadoopConfiguration
+ val conf = spark.sessionState.newHadoopConf()
val parsedOptions = new AvroOptions(options, conf)
// Schema evolution is not supported yet. Here we only pick a single random sample file to
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org