[jira] [Commented] (PIG-2647) Split Combining drops splits with empty getLocations()

["Travis Woodruff (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231705#comment-14231705 ] 

Travis Woodruff commented on PIG-2647:
--------------------------------------

Ran into this in a production environment, We lost several DataNodes, and this caused incorrect results to be produced (!!!), as files with no available blocks were silently ignored.

> Split Combining drops splits with empty getLocations()
> ------------------------------------------------------
>
>                 Key: PIG-2647
>                 URL: https://issues.apache.org/jira/browse/PIG-2647
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Alex Levenson
>            Assignee: Travis Woodruff
>
> in:
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil#getCombinePigSplits
> which is used by PigInputFormat
> There is an assumption that every split's getLocations() will return a non-empty array.
> If the following criteria are met:
> 1) Split combining is turned on
> 2) There is more than one split
> 3) There is at least one split that is smaller than the maxCombineSplitSize
> splits with empty getLocations() will simply be dropped (ignored) without warning.
> The hadoop API does not specify that all splits must return a location and there are cases where a split may want to return no locations (if the data is not in HDFS for example, or if the data is a directory full of HDFS files in which case there's not much gained by having locality)
> This is due to the implementation of org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil#getCombinePigSplits
> scans all splits eligible for combining and creates a map of Nodes -> splits, then laster iterates through the MAP (not the splits) to do the combining.
> One solution would be to inject a dummy "empty node" into the map.
> Overall the logic in getCombinePigSplits is very complicated and has a lot of edge cases, it might be worth cleaning up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)