You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon (JIRA)" <ji...@apache.org> on 2013/05/10 07:55:15 UTC
[jira] [Commented] (HAMA-756) Timing issue and file merging
algorithm in PartitioningRunner make job fail
[ https://issues.apache.org/jira/browse/HAMA-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13653568#comment-13653568 ]
Edward J. Yoon commented on HAMA-756:
-------------------------------------
You seems looking at old source code.
See http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/PartitioningRunner.java
By calling sync() method, there's no race condition. The 2nd reported issue also is already fixed.
{code}
peer.sync();
FileStatus[] status = fs.listStatus(partitionDir);
...
if (assignedID == peer.getNumPeers())
assignedID = assignedID - 1;
{code}
> Timing issue and file merging algorithm in PartitioningRunner make job fail
> ---------------------------------------------------------------------------
>
> Key: HAMA-756
> URL: https://issues.apache.org/jira/browse/HAMA-756
> Project: Hama
> Issue Type: Bug
> Reporter: MaoYuan Xian
> Assignee: MaoYuan Xian
>
> There are two major problems in bsp methor of PartitioningRunner may make the partitioning fail:
> 1. The call to peer.getNumPeers() may trigger the timing issue. In the special situation when some tasks complete the bsp call but some others just enter the "for (FileStatus statu : status)" loop, these remaining task calling to peer.getNumPeers() will trigger the problem.
> 2. The algorithm of merging the sequence files has the problem: e.g. when desiredNum is 8 and partitioning task number (peer.getNumPeers()) is 6, the part-7 directory can not find the handler to merging it as a file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira