[jira] [Commented] (SPARK-17132) binaryFiles method can't handle paths with embedded commas

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-17132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426962#comment-15426962 ] 

Sean Owen commented on SPARK-17132:
-----------------------------------

Yeah, that would be a solution. It actually affects all related API methods of SparkContext, not just one. I'm not clear if it's worth adding a bunch to the RDD API now in Spark 2, but it's not out of the question.

It should work to escape the commas with \, or at least that's what the Hadoop classes appear to want done. I suppose that's the intended usage, though I also would prefer a more explicit seq argument.

> binaryFiles method can't handle paths with embedded commas
> ----------------------------------------------------------
>
>                 Key: SPARK-17132
>                 URL: https://issues.apache.org/jira/browse/SPARK-17132
>             Project: Spark
>          Issue Type: Bug
>          Components: Java API
>    Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0, 1.6.1, 1.6.2, 2.0.0
>            Reporter: Maximilian Najork
>
> A path with an embedded comma is treated as two separate paths by binaryFiles. Since commas are legal characters in paths, this behavior is incorrect. I recommend overloading binaryFiles to accept an array of path strings in addition to a string of comma-separated paths. Since setInputPaths is already overloaded to accept either form, this should be relatively low-effort.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org