You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "hantiantian (JIRA)" <ji...@apache.org> on 2018/09/20 07:47:00 UTC
[jira] [Created] (SPARK-25477) “INSERT OVERWRITE LOCAL DIRECTORY”， the data files allocated on the non-driver node will not be written to the specified output directory
hantiantian created SPARK-25477:
-----------------------------------
Summary: “INSERT OVERWRITE LOCAL DIRECTORY”， the data files allocated on the non-driver node will not be written to the specified output directory
Key: SPARK-25477
URL: https://issues.apache.org/jira/browse/SPARK-25477
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.3.1, 2.3.0
Reporter: hantiantian
The "INSERT OVERWRITE LOCAL DIRECTORY" features use the local staging directory to load data into the specified output directory. If the executor is executed on a non-driver node, the data files allocated on that node will not be written to the specified output directory.
The "spark-sql" application has two executors, one on the driver node and one on the non-driver node.
spark-sql> select * from aa;
1 10000
2 30000
2 10000
3 30000
As follows, view the data file of the aa table on HDFS,
/spark/aa/part-00000-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
/spark/aa/part-00001-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
/spark/aa/part-00002-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
/spark/aa/part-00003-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
spark-sql> insert overwrite local directory "/test" (select * from aa);
As follows, view the contents of the output directory "/test", there are only two data files,
part-00000-16312194-288a-4085-9cb8-de0a685f7af0-c000
part-00002-16312194-288a-4085-9cb8-de0a685f7af0-c000
_SUCCESS
As follows, view the executor log on non-driver node, the other two data files is Is assigned to this node. And the execution results of the tasks is saved to the local staging directory.
INFO org.apache.spark.rdd.HadoopRDD: Input split: hdfs://nameservice/spark/aa/part-00001-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000:0+8
INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_20180920151705_0002_m_000001_0' to file:/spark-tmp/stagedir/.hive-staging_hive_2018-09-20_15-17-01_313_4124694630008666159-1/-ext-10000/_temporary/0/task_20180920151705_0002_m_000001
INFO org.apache.spark.rdd.HadoopRDD: Input split: hdfs://nameservice/spark/aa/part-00003-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000:0+8
INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_20180920151705_0002_m_000003_0' to file:/spark-tmp/stagedir/.hive-staging_hive_2018-09-20_15-17-01_313_4124694630008666159-1/-ext-10000/_temporary/0/task_20180920151705_0002_m_000003
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org