You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by li...@apache.org on 2017/12/15 07:11:17 UTC
spark git commit: [SPARK-22753][SQL] Get rid of
dataSource.writeAndRead
Repository: spark
Updated Branches:
refs/heads/master 3fea5c4f1 -> 3775dd31e
[SPARK-22753][SQL] Get rid of dataSource.writeAndRead
## What changes were proposed in this pull request?
As the discussion in https://github.com/apache/spark/pull/16481 and https://github.com/apache/spark/pull/18975#discussion_r155454606
Currently the BaseRelation returned by `dataSource.writeAndRead` only used in `CreateDataSourceTableAsSelect`, planForWriting and writeAndRead has some common code paths.
In this patch I removed the writeAndRead function and added the getRelation function which only use in `CreateDataSourceTableAsSelectCommand` while saving data to non-existing table.
## How was this patch tested?
Existing UT
Author: Yuanjian Li <xy...@gmail.com>
Closes #19941 from xuanyuanking/SPARK-22753.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3775dd31
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3775dd31
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3775dd31
Branch: refs/heads/master
Commit: 3775dd31ee86c32b6161ca99d8fd5cfd7c1a758e
Parents: 3fea5c4
Author: Yuanjian Li <xy...@gmail.com>
Authored: Thu Dec 14 23:11:13 2017 -0800
Committer: gatorsmile <ga...@gmail.com>
Committed: Thu Dec 14 23:11:13 2017 -0800
----------------------------------------------------------------------
.../sql/execution/command/InsertIntoDataSourceDirCommand.scala | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/3775dd31/sql/core/src/main/scala/org/apache/spark/sql/execution/command/InsertIntoDataSourceDirCommand.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/InsertIntoDataSourceDirCommand.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/InsertIntoDataSourceDirCommand.scala
index 9e35190..1dc24b3 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/InsertIntoDataSourceDirCommand.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/InsertIntoDataSourceDirCommand.scala
@@ -67,8 +67,7 @@ case class InsertIntoDataSourceDirCommand(
val saveMode = if (overwrite) SaveMode.Overwrite else SaveMode.ErrorIfExists
try {
- sparkSession.sessionState.executePlan(dataSource.planForWriting(saveMode, query))
- dataSource.writeAndRead(saveMode, query)
+ sparkSession.sessionState.executePlan(dataSource.planForWriting(saveMode, query)).toRdd
} catch {
case ex: AnalysisException =>
logError(s"Failed to write to directory " + storage.locationUri.toString, ex)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org