[jira] [Commented] (SPARK-1415) Add a minSplits parameter to wholeTextFiles

["Xusen Yin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13960908#comment-13960908 ] 

Xusen Yin commented on SPARK-1415:
----------------------------------

Hi Matei, I just looked around in those Hadoop APIs. I find that the new Hadoop API deprecates the minSplit, instead of minSplit, they prefer minSplitSize and maxSplitSize to control the split. minSplit is negative correlated with maxSplitSize, so I think we have 2 ways to fix the issue:

1. We just provide a new API with maxSplitSize, say, wholeTextFiles(path: String, maxSplitSize: Long);

2. We write a delegation to compute the maxSplitSize using minSplit (easy to write, taking old Hadoop API as an example), and provide the API wholeTextFile(path: String, minSplit: Int);

I also think we can provide the two APIs simultaneously. What do you think?

> Add a minSplits parameter to wholeTextFiles
> -------------------------------------------
>
>                 Key: SPARK-1415
>                 URL: https://issues.apache.org/jira/browse/SPARK-1415
>             Project: Spark
>          Issue Type: Bug
>            Reporter: Matei Zaharia
>            Assignee: Xusen Yin
>              Labels: Starter
>
> This probably requires adding one to newAPIHadoopFile too.



--
This message was sent by Atlassian JIRA
(v6.2#6252)