You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2019/03/13 02:38:00 UTC
[jira] [Commented] (SPARK-27140) The feature is 'insert overwrite
local directory' has an inconsistent behavior in different environment.
[ https://issues.apache.org/jira/browse/SPARK-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16791236#comment-16791236 ]
Apache Spark commented on SPARK-27140:
--------------------------------------
User 'beliefer' has created a pull request for this issue:
https://github.com/apache/spark/pull/23950
> The feature is 'insert overwrite local directory' has an inconsistent behavior in different environment.
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-27140
> URL: https://issues.apache.org/jira/browse/SPARK-27140
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0, 2.4.0, 3.0.0
> Reporter: jiaan.geng
> Priority: Major
>
> In local[*] mode, maropu give a test case as follows:
> {code:java}
> $ls /tmp/noexistdir
> ls: /tmp/noexistdir: No such file or directory
> scala> sql("""create table t(c0 int, c1 int)""")
> scala> spark.table("t").explain
> == Physical Plan ==
> Scan hive default.t [c0#5, c1#6], HiveTableRelation `default`.`t`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c0#5, c1#6]
> scala> sql("""insert into t values(1, 1)""")
> scala> sql("""select * from t""").show
> +---+---+
> | c0| c1|
> +---+---+
> | 1| 1|
> +---+---+
> scala> sql("""insert overwrite local directory '/tmp/noexistdir/t' select * from t""")
> $ls /tmp/noexistdir/t/
> _SUCCESS part-00000-bbea4213-071a-49b4-aac8-8510e7263d45-c000
> {code}
> This test case prove spark will create the not exists path and move middle result from local temporary path to created path.This test based on newest master.
> I follow the test case provided by maropu,but find another behavior.
> I run these SQL maropu provided on local[*] deploy mode based on 2.3.0.
> Inconsistent behavior appears as follows:
> {code:java}
> ls /tmp/noexistdir
> ls: cannot access /tmp/noexistdir: No such file or directory
> scala> sql("""create table t(c0 int, c1 int)""")
> res0: org.apache.spark.sql.DataFrame = []
> scala> spark.table("t").explain
> == Physical Plan ==
> HiveTableScan [c0#5, c1#6], HiveTableRelation `default`.`t`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c0#5, c1#6]
> scala> sql("""insert into t values(1, 1)""")
> scala> sql("""select * from t""").show
> +---+---+
> | c0| c1|
> +---+---+
> | 1| 1|
> +---+---+
> scala> sql("""insert overwrite local directory '/tmp/noexistdir/t' select * from t""")
> res1: org.apache.spark.sql.DataFrame = []
> ls /tmp/noexistdir/t/
> /tmp/noexistdir/t
> vi /tmp/noexistdir/t
> 1
> {code}
> Then I pull the master branch and compile it and deploy it on my hadoop cluster.I get the inconsistent behavior again. The spark version to test is 3.0.0.
> {code:java}
> ls /tmp/noexistdir
> ls: cannot access /tmp/noexistdir: No such file or directory
> Java HotSpot(TM) 64-Bit Server VM warning: Using the ParNew young collector with the Serial old collector is deprecated and will likely be removed in a future release
> Spark context Web UI available at http://10.198.66.204:55326
> Spark context available as 'sc' (master = local[*], app id = local-1551259036573).
> Spark session available as 'spark'.
> Welcome to spark version 3.0.0-SNAPSHOT
> Using Scala version 2.12.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> sql("""select * from t""").show
> +---+---+
> | c0| c1|
> +---+---+
> | 1| 1|
> +---+---+
> scala> sql("""insert overwrite local directory '/tmp/noexistdir/t' select * from t""")
> res1: org.apache.spark.sql.DataFrame = []
> scala>
> ll /tmp/noexistdir/t
> -rw-r--r-- 1 xitong xitong 0 Feb 27 17:19 /tmp/noexistdir/t
> vi /tmp/noexistdir/t
> 1
> {code}
> The /tmp/noexistdir/t is a file too.
> I create a PR `https://github.com/apache/spark/pull/23950` used for test the behavior by UT.
> UT results are the same as those of maropu's test, but different from mine.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org