You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Hudson (JIRA)" <ji...@apache.org> on 2008/12/15 08:43:44 UTC
[jira] Commented: (HAMA-133) To reduce disk I/O operations, Remove
'reduce phase' from blocking_mapred
[ https://issues.apache.org/jira/browse/HAMA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656567#action_12656567 ]
Hudson commented on HAMA-133:
-----------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12396060/HAMA-133.patch
against trunk revision 726076.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hama-Patch/132/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hama-Patch/132/artifact/trunk/build/reports/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hama-Patch/132/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hama-Patch/132/console
This message is automatically generated.
> To reduce disk I/O operations, Remove 'reduce phase' from blocking_mapred
> -------------------------------------------------------------------------
>
> Key: HAMA-133
> URL: https://issues.apache.org/jira/browse/HAMA-133
> Project: Hama
> Issue Type: Sub-task
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-133.patch
>
>
> > If we remove 'reduce phase', I guess we can reduce the disk I/O operations.
> Yes.
> >
> >
> > In the map, read { Constants.BLOCK_STARTROW, Constants.BLOCK_ENDROW,
> > Constants.BLOCK_STARTCOLUMN, Constants.BLOCK_ENDCOLUMN } instead of {
> > Constants.COLUMN }, and write directly blocks.
> Two methods to be considered:
> 1) We need a InputFormat that partitions the matrix table according to the
> row boundaries of the blocks.
> This should be carefully to make sure a single block will not divied
> into two or more mappers.
> 2) Like what RandomMatrixMap does, we just tell the mappers the row/column
> boundaries of the blocks of a matrix-table.
> Scanner the portion of the table will be done in a mapper.
> I think 1) may be better than 2).
> An InputFormat can get the locality of a range of table to let MR know how
> to move the mr computations close to it.
> In 2), if we do it like RandomMatrixMap, we may lose some locality
> informations of the table. so that the network transfer overhead may be
> increase.
> It is just my guess and thoughts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.