You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "bailin (Jira)" <ji...@apache.org> on 2019/09/17 02:09:00 UTC
[jira] [Commented] (SPARK-29047) use spark-submit Not a file:
hdfs://
[ https://issues.apache.org/jira/browse/SPARK-29047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931013#comment-16931013 ]
bailin commented on SPARK-29047:
--------------------------------
i solved, it's code problem. Generate and read parquet,use "config("spark.sql.sources.partitionColumnTypeInference.enabled","false" both
> use spark-submit Not a file: hdfs://
> --------------------------------------
>
> Key: SPARK-29047
> URL: https://issues.apache.org/jira/browse/SPARK-29047
> Project: Spark
> Issue Type: Question
> Components: Deploy, Spark Submit
> Affects Versions: 2.1.3
> Environment: spark1.3 ；hadoop2.6
> Reporter: bailin
> Priority: Critical
> Attachments: hadfs file is already existed.png
>
>
> when i submit a spark application :
> {code:java}
> /spark-submit --class com.yto.log.SparkStatCleanJobYARN --name TopNStatJobYARN --master yarn --executor-memory 1G --num-executors 1 /home/hadoop/lib/sql-1.0-jar-with-dependencies.jar hdfs://hadoop001:8020/clean 20161110{code}
> the hadfs file is already existed，but not recognition
>
> {code:java}
> 19/09/11 14:18:50 INFO mapred.FileInputFormat: Total input paths to process : 219/09/11 14:18:50 INFO mapred.FileInputFormat: Total input paths to process : 219/09/11 14:18:50 ERROR datasources.FileFormatWriter: Aborting job null.java.io.IOException: Not a file: hdfs://hadoop001:8020/clean/day= at org.apache.hadoop.mapred.FileInputFormat.getSplits
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org