You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/09/21 17:00:00 UTC
[jira] [Commented] (SPARK-25477) “INSERT OVERWRITE LOCAL DIRECTORY”， the data files allocated on the non-driver node will not be written to the specified output directory
[ https://issues.apache.org/jira/browse/SPARK-25477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623877#comment-16623877 ]
Apache Spark commented on SPARK-25477:
--------------------------------------
User 'httfighter' has created a pull request for this issue:
https://github.com/apache/spark/pull/22487
> “INSERT OVERWRITE LOCAL DIRECTORY”， the data files allocated on the non-driver node will not be written to the specified output directory
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-25477
> URL: https://issues.apache.org/jira/browse/SPARK-25477
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.0, 2.3.1
> Reporter: hantiantian
> Priority: Major
>
> The "INSERT OVERWRITE LOCAL DIRECTORY" features use the local staging directory to load data into the specified output directory. If the executor is executed on a non-driver node, the data files allocated on that node will not be written to the specified output directory.
> The "spark-sql" application has two executors, one on the driver node and one on the non-driver node.
> spark-sql> select * from aa;
> 1 10000
> 2 30000
> 2 10000
> 3 30000
> As follows, view the data file of the aa table on HDFS,
> /spark/aa/part-00000-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
> /spark/aa/part-00001-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
> /spark/aa/part-00002-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
> /spark/aa/part-00003-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
>
> spark-sql> insert overwrite local directory "/test" (select * from aa);
> As follows, view the contents of the output directory "/test", there are only two data files,
> part-00000-16312194-288a-4085-9cb8-de0a685f7af0-c000
> part-00002-16312194-288a-4085-9cb8-de0a685f7af0-c000
> _SUCCESS
> As follows, view the executor log on non-driver node, the other two data files is Is assigned to this node. And the execution results of the tasks is saved to the local staging directory.
> INFO org.apache.spark.rdd.HadoopRDD: Input split: hdfs://nameservice/spark/aa/part-00001-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000:0+8
> INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_20180920151705_0002_m_000001_0' to file:/spark-tmp/stagedir/.hive-staging_hive_2018-09-20_15-17-01_313_4124694630008666159-1/-ext-10000/_temporary/0/task_20180920151705_0002_m_000001
>
> INFO org.apache.spark.rdd.HadoopRDD: Input split: hdfs://nameservice/spark/aa/part-00003-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000:0+8
> INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_20180920151705_0002_m_000003_0' to file:/spark-tmp/stagedir/.hive-staging_hive_2018-09-20_15-17-01_313_4124694630008666159-1/-ext-10000/_temporary/0/task_20180920151705_0002_m_000003
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org