[jira] [Commented] (MAPREDUCE-7194) New Method For CombineFile

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18047329#comment-18047329 ] 

ASF GitHub Bot commented on MAPREDUCE-7194:
-------------------------------------------

github-actions[bot] commented on PR #618:
URL: https://github.com/apache/hadoop/pull/618#issuecomment-3688271592

   We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
   Thanks all for your contribution.




> New Method For CombineFile
> --------------------------
>
>                 Key: MAPREDUCE-7194
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7194
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 3.2.0
>            Reporter: David Mollitor
>            Assignee: David Mollitor
>            Priority: Major
>
> Rhe {{CombineFileInputFormat}} class is responsible for grouping blocks together to form larger splits.  The current implementation is very naive.  It iterates over the list of available blocks and as long as the current group of blocks is less than the maximum split size, it will keep added blocks.  The check for if a split has reached its maximum size happens *after* each block is added.  For example given a certain maximum "M", and two blocks which are both 7/8M, they will be grouped together to create a split which is 14/8M.  If M is a large number, this split will be very large and not what the operator would expect.
> I'll propose a general clean up and also, enforcing that, unless a files cannot be split, that its splits will not be larger than the configured maximum size.  This will provide operators a much more straight-forward way of calculating the expected number of splits.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org