[jira] [Resolved] (MAPREDUCE-5646) Option to shuffle splits of equal size

["Mike Liddell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Liddell resolved MAPREDUCE-5646.
-------------------------------------

    Resolution: Won't Fix

> Option to shuffle splits of equal size
> --------------------------------------
>
>                 Key: MAPREDUCE-5646
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5646
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: trunk, 1-win
>            Reporter: Mike Liddell
>            Assignee: Mike Liddell
>         Attachments: MAPREDUCE-5646.branch-1-win.1.patch
>
>
> Mapreduce split calculation has the following base logic (via JobClient and the major InputFormat implementations ):
> ◾enumerate input files in natural (aka linear) order.
> ◾create one split for each 'block-size' of each input. Apart from rack-awareness, combining and so on, the input file order remains in its natural order.
> ◾sort the splits by size using a stable sort based on splitsize.
> When data from multiple storage services are used in a single hadoop job, we get better I/O utilization if the list of splits does round-robin or random-access across the services. 
> The particular scenario arises in Azure HDInsight where jobs can easily read from many storage accounts and each storage account has hard limits on throughtput.  Concurrent access to the accounts is substantially better than 
>  
> Two common scenarios can cause non-ideal access pattern:
>  1. many/all input files are the same size
>  2. files have different sizes, but many/all input files have size>blocksize.
>  In the second scenario, for each file will have one or more splits with size exactly equal to block size so it basically degenerates to the first scenario.
> There are various ways to solve the problem but the simplest is to alter the mapreduce JobClient to sort splits by size _and_ randomize the order of splits with equal size. This keeps the old behavior effectively unchanged while also fixing both common problematic scenarios.
> Some rare scenarios will still suffer bad access patterns due. For example if two storage accounts are used and the files from one storage account are all smaller than from the other then problems can arise. Addressing these scenarios would be further work, perhaps by completely randomizing the split order. These problematic scenarios are considered rare and not requiring immediate attention.
> If further algorithms for split ordering are necessary, the implementation in JobClient will change to being interface-based (eg interface splitOrderer) with various standard implementations.  At this time there is only the need for two implementations and so simple Boolean flag and if/then logic is used.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)