[jira] Commented: (HADOOP-4565) MultiFileInputSplit can use data locality information to create splits

["Joydeep Sen Sarma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662999#action_12662999 ] 

Joydeep Sen Sarma commented on HADOOP-4565:
-------------------------------------------

Comments:
- min/maxSplitSize - i found the initialization confusing. is it the intent that values supplied by setXXXSplitSize take precedence over config values? (I would find that logical - but the implementation seems different)
- OneFileInfo - it seems to me that this class has no use except that it's constructor has a side-effect of populating some external structures. As a style issue - this seems unnecessary - we could just have a simple helper function instead.
- for(boolean doIterate ..): i found this variable pretty confusing. could we just replace doIterator with blockToNodes.size()>0 ?

- one of the comments from the last review was that CombineFileSplit would be nice to treat as an extension of MultiFileSplit. I would still make that argument - there should be a single base class/interface for these multisplit classes - because then the CombineFileRecordReader can be made to work with all of them. 

  Looking at the apis - it actually seems that MultiFileSplit can perhaps be treated as a special implementation (extension) of CombineFileSplit (with get*offset functions always returning 0). 

  Any thoughts on this?

Not that familiar with miniMR stuff - would be good if someone else can review the test code as well ..

> MultiFileInputSplit can use data locality information to create splits
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4565
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: CombineMultiFile.patch, CombineMultiFile2.patch, CombineMultiFile3.patch, CombineMultiFile4.patch, CombineMultiFile5.patch
>
>
> The MultiFileInputFormat takes a set of paths and creates splits based on file sizes. Each splits contains a few files an each split are roughly equal in size. It would be efficient if we can extend this InputFormat to create splits such each all the blocks in one split and either node-local or rack-local.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.