[jira] [Commented] (SPARK-10596) SparkContext.textFile() cannot handle escape character for spaces

["Klaus Thul (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065638#comment-15065638 ] 

Klaus Thul commented on SPARK-10596:
------------------------------------

sc.textFile wants to have real spaces and not %20 as part of the path in the argument. 

The argument to {{sc.textFile}} gets passed to {{org.apache.hadoop.mapred.FileInputFormat.setInputPaths}}. Documentation at https://hadoop.apache.org/docs/r2.6.2/api/org/apache/hadoop/fs/Path.html#Path(java.lang.String)
explicitly states ??Path strings are URIs, but with unescaped elements and some additional normalisation??.

This is different from what you get from {{java.net.URI.toString method}}.  I could resolve a similar issue by using something like:

{quote}
val fileURI = new java.io.File(filename).toURI
sc.textFile(java.net.URLDecoder(fileURI.toString))
{quote}

This is likely not a bug, although documentation could be improved.


> SparkContext.textFile() cannot handle escape character for spaces
> -----------------------------------------------------------------
>
>                 Key: SPARK-10596
>                 URL: https://issues.apache.org/jira/browse/SPARK-10596
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: Jason Pohl
>
> Trying to use SparkContext.textFile() with an S3 URI, but there does not seem to be an accepted way to escape spaces in the path (which is a valid S3 naming convention for folders)
> get an exception: "org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:"
> Tried:
> +
> %20
> \%20
> \%20\
> None of these work



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org