You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yong Tang (JIRA)" <ji...@apache.org> on 2015/04/26 20:03:38 UTC
[jira] [Created] (SPARK-7155) SparkContext's newAPIHadoopFile does
not support comma-separated list of files, but the other API hadoopFile
does.
Yong Tang created SPARK-7155:
--------------------------------
Summary: SparkContext's newAPIHadoopFile does not support comma-separated list of files, but the other API hadoopFile does.
Key: SPARK-7155
URL: https://issues.apache.org/jira/browse/SPARK-7155
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.3.1
Environment: Ubuntu 14.04
Reporter: Yong Tang
SparkContext's newAPIHadoopFile() does not support comma-separated list of files. For example, the following:
sc.newAPIHadoopFile("/root/file1.txt,/root/file2.txt", classOf[TextInputFormat], classOf[LongWritable], classOf[Text])
will throw
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/root/file1.txt,/root/file2.txt
However, the other API hadoopFile() is able to process comma-separated list of files correctly. In addition, since sc.textFile() uses hadoopFile(), it is also able to process comma-separated list of files correctly.
The problem is that newAPIHadoopFile() use addInputPath() to add the file path into NewHadoopRDD. See Ln 928-931, master branch:
val job = new NewHadoopJob(conf)
NewFileInputFormat.addInputPath(job, new Path(path))
val updatedConf = job.getConfiguration
new NewHadoopRDD(this, fClass, kClass, vClass, updatedConf).setName(path)
Change addInputPath(job, new Path(path)) to addInputPaths(job, path) will resolve this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org