You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Kazuki Ohta (JIRA)" <ji...@apache.org> on 2011/08/19 18:56:27 UTC
[jira] [Created] (MAPREDUCE-2862) Infinite loop in
CombineFileInputFormat#getMoreSplits(), with missing blocks
Infinite loop in CombineFileInputFormat#getMoreSplits(), with missing blocks
----------------------------------------------------------------------------
Key: MAPREDUCE-2862
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2862
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Kazuki Ohta
Hi, we met the infinite loop on CombineFileInputFormat#getMoreSplits().
At first, we lost some blocks by mis-operation :-(. Then, one job tried to use these missing blocks. At that time getMoreSplits() goes into the infinite loop.
>From our investigation, this List could be an empty array.
> https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L363
Then 'for' loop just after that line does nothing, and entry is not removed from 'blockToNodes'.
Finally this line goes into the infinite loop.
> https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L348
We're now creating the patch against this problem...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2862) Infinite loop in
CombineFileInputFormat#getMoreSplits(), with missing blocks
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088022#comment-13088022 ]
Todd Lipcon commented on MAPREDUCE-2862:
----------------------------------------
Hey Sadayuki. Good to see you here on JIRA :) I think the patch you've attached is against the 0.20 branch. Can you please provide a patch against trunk as well? Thanks.
> Infinite loop in CombineFileInputFormat#getMoreSplits(), with missing blocks
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-2862
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2862
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Kazuki Ohta
> Attachments: MAPREDUCE-2862-warn-and-ignore-corrupted-blocks.patch
>
>
> Hi, we met the infinite loop on CombineFileInputFormat#getMoreSplits().
> At first, we lost some blocks by mis-operation :-(. Then, one job tried to use these missing blocks. At that time getMoreSplits() goes into the infinite loop.
> From our investigation, this List could be an empty array.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L363
> Then 'for' loop just after that line does nothing, and entry is not removed from 'blockToNodes'.
> Finally this line goes into the infinite loop.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L348
> We're now creating the patch against this problem...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2862) Infinite loop in
CombineFileInputFormat#getMoreSplits(), with missing blocks
Posted by "Sadayuki Furuhashi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sadayuki Furuhashi updated MAPREDUCE-2862:
------------------------------------------
Attachment: MAPREDUCE-2862-warn-and-ignore-corrupted-blocks.patch
Breaks infinite loop in CombineFileInputFormat.getMoreSplits by ignoring corrupted blocks.
> Infinite loop in CombineFileInputFormat#getMoreSplits(), with missing blocks
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-2862
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2862
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Kazuki Ohta
> Attachments: MAPREDUCE-2862-warn-and-ignore-corrupted-blocks.patch
>
>
> Hi, we met the infinite loop on CombineFileInputFormat#getMoreSplits().
> At first, we lost some blocks by mis-operation :-(. Then, one job tried to use these missing blocks. At that time getMoreSplits() goes into the infinite loop.
> From our investigation, this List could be an empty array.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L363
> Then 'for' loop just after that line does nothing, and entry is not removed from 'blockToNodes'.
> Finally this line goes into the infinite loop.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L348
> We're now creating the patch against this problem...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2862) Infinite loop in
CombineFileInputFormat#getMoreSplits(), with missing blocks
Posted by "Sadayuki Furuhashi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087844#comment-13087844 ]
Sadayuki Furuhashi commented on MAPREDUCE-2862:
-----------------------------------------------
I attached a patch for the git HEAD. It shows error messages and ignores corrupted blocks.
> Infinite loop in CombineFileInputFormat#getMoreSplits(), with missing blocks
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-2862
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2862
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Kazuki Ohta
> Attachments: MAPREDUCE-2862-warn-and-ignore-corrupted-blocks.patch
>
>
> Hi, we met the infinite loop on CombineFileInputFormat#getMoreSplits().
> At first, we lost some blocks by mis-operation :-(. Then, one job tried to use these missing blocks. At that time getMoreSplits() goes into the infinite loop.
> From our investigation, this List could be an empty array.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L363
> Then 'for' loop just after that line does nothing, and entry is not removed from 'blockToNodes'.
> Finally this line goes into the infinite loop.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L348
> We're now creating the patch against this problem...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2862) Infinite loop in
CombineFileInputFormat#getMoreSplits(), with missing blocks
Posted by "Kazuki Ohta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087814#comment-13087814 ]
Kazuki Ohta commented on MAPREDUCE-2862:
----------------------------------------
Would be some options...
* cause an error
* add option to ignore these errors (log missing blocks in Warning level)
> Infinite loop in CombineFileInputFormat#getMoreSplits(), with missing blocks
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-2862
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2862
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Kazuki Ohta
>
> Hi, we met the infinite loop on CombineFileInputFormat#getMoreSplits().
> At first, we lost some blocks by mis-operation :-(. Then, one job tried to use these missing blocks. At that time getMoreSplits() goes into the infinite loop.
> From our investigation, this List could be an empty array.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L363
> Then 'for' loop just after that line does nothing, and entry is not removed from 'blockToNodes'.
> Finally this line goes into the infinite loop.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L348
> We're now creating the patch against this problem...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2862) Infinite loop in
CombineFileInputFormat#getMoreSplits(), with missing blocks
Posted by "Chris Nauroth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088783#comment-13088783 ]
Chris Nauroth commented on MAPREDUCE-2862:
------------------------------------------
Sadayuki, thank you for submitting a patch on this. I've been bitten by this one too.
This patch would log warnings about "corrupted files". Is it really true that this indicates corruption? My experience has been that I've seen this happen when CombineFileInputFormat tries to read newly written files that have not yet had their first block flushed. This isn't really corruption, so I'm wondering if logging warnings about corrupt files would give a user the wrong impression that the cluster is suffering from corruption.
To workaround this, I've been running my jobs with a private patch of CombineFileInputFormat that adds this to the constructor for OneFileInfo:
// Bail out if the block has no locations. This guards against an
// infinite loop in getMoreSplits. This change is not present in open
// source Hadoop.
if (oneblock.length <= 0)
continue;
That prevents these blocks from ever entering the getMoreSplits logic in the first place. If you're interested in that approach instead, let me know, and I'll put the patch together. I'd still need to add a unit test for it too.
Thanks again,
--Chris
> Infinite loop in CombineFileInputFormat#getMoreSplits(), with missing blocks
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-2862
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2862
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Kazuki Ohta
> Attachments: MAPREDUCE-2862-warn-and-ignore-corrupted-blocks.patch
>
>
> Hi, we met the infinite loop on CombineFileInputFormat#getMoreSplits().
> At first, we lost some blocks by mis-operation :-(. Then, one job tried to use these missing blocks. At that time getMoreSplits() goes into the infinite loop.
> From our investigation, this List could be an empty array.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L363
> Then 'for' loop just after that line does nothing, and entry is not removed from 'blockToNodes'.
> Finally this line goes into the infinite loop.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L348
> We're now creating the patch against this problem...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2862) Infinite loop in
CombineFileInputFormat#getMoreSplits(), with missing blocks
Posted by "Subroto Sanyal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175513#comment-13175513 ]
Subroto Sanyal commented on MAPREDUCE-2862:
-------------------------------------------
@Kazuki
Are you hitting to same problem as MAPREDUCE-2185?
> Infinite loop in CombineFileInputFormat#getMoreSplits(), with missing blocks
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-2862
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2862
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Kazuki Ohta
> Attachments: MAPREDUCE-2862-warn-and-ignore-corrupted-blocks.patch
>
>
> Hi, we met the infinite loop on CombineFileInputFormat#getMoreSplits().
> At first, we lost some blocks by mis-operation :-(. Then, one job tried to use these missing blocks. At that time getMoreSplits() goes into the infinite loop.
> From our investigation, this List could be an empty array.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L363
> Then 'for' loop just after that line does nothing, and entry is not removed from 'blockToNodes'.
> Finally this line goes into the infinite loop.
> > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L348
> We're now creating the patch against this problem...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira