You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon (JIRA)" <ji...@apache.org> on 2008/12/09 08:41:44 UTC
[jira] Created: (HAMA-129) Improving speed of matrix multiplication
Improving speed of matrix multiplication
----------------------------------------
Key: HAMA-129
URL: https://issues.apache.org/jira/browse/HAMA-129
Project: Hama
Issue Type: Improvement
Components: implementation
Affects Versions: 0.1.0
Reporter: Edward J. Yoon
Assignee: Edward J. Yoon
Fix For: 0.1.0
We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655065#action_12655065 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
Hmm.. This patch causes 'UnknownScannerException'
----
08/12/10 10:14:43 INFO mapred.JobClient: map 50% reduce 16%
08/12/10 10:16:29 INFO mapred.JobClient: Task Id : attempt_200812100956_0005_m_000000_0, Status : FAILED
org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: -598694650616426837
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1077)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:856)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1155)
at org.apache.hama.mapred.BlockInputFormatBase$TableRecordReader.next(BlockInputFormatBase.java:177)
at org.apache.hama.mapred.BlockInputFormatBase$TableRecordReader.next(BlockInputFormatBase.java:58)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:165)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: patch01.txt
test patch.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, patch01.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: HAMA-129.patch
More simplifed patch to avoid 'OOM'
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659798#action_12659798 ]
Hudson commented on HAMA-129:
-----------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12396907/HAMA-129_v06.patch
against trunk revision 729823.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 6 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hama-Patch/137/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hama-Patch/137/artifact/trunk/build/reports/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hama-Patch/137/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hama-Patch/137/console
This message is automatically generated.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657267#action_12657267 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
for (int j = 0; j < blockSize; j++) {
SubMatrix b = matrix_b.getBlock(key.getColumn(), j);
SubMatrix c = a.mult(b);
output.collect(new BlockID(key.getRow(), j), new BlockWritable(c));
}
we can just use scanner.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656889#action_12656889 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
sometimes i recieved a NullPointerException.
----
08/12/16 14:18:34 INFO mapred.JobClient: Running job: job_200812152014_0058
08/12/16 14:18:35 INFO mapred.JobClient: map 0% reduce 0%
08/12/16 14:18:47 INFO mapred.JobClient: map 50% reduce 0%
08/12/16 14:18:55 INFO mapred.JobClient: map 50% reduce 8%
08/12/16 14:18:58 INFO mapred.JobClient: map 50% reduce 16%
08/12/16 14:19:16 INFO mapred.JobClient: Task Id : attempt_200812152014_0058_m_000001_0, Status : FAILED
java.lang.NullPointerException
at org.apache.hama.DenseMatrix.getBlock(DenseMatrix.java:481)
at org.apache.hama.algebra.BlockCyclicMultiplyMap.map(BlockCyclicMultiplyMap.java:75)
at org.apache.hama.algebra.BlockCyclicMultiplyMap.map(BlockCyclicMultiplyMap.java:39)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: HAMA-129_v03.patch
replaced scanner
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Samuel Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663284#action_12663284 ]
Samuel Guo commented on HAMA-129:
---------------------------------
great!
BTW, can you share the tablets' key-range distribution of matrix a, matrix b, and result matrix?
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Samuel Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663285#action_12663285 ]
Samuel Guo commented on HAMA-129:
---------------------------------
BTW, you can get the tablets' key range distribution from HBase's web UI.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: HAMA-129_v04.patch
- Repaced to scanner from each get(row, column).
- BlockID read/write string.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: HAMA-129_v02.patch
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: HAMA-129_v01.patch
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663987#action_12663987 ]
Hudson commented on HAMA-129:
-----------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12397936/HAMA-129_newApproach_v01.patch
against trunk revision 732267.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 6 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs -1. The patch appears to introduce 1 new Findbugs warnings.
core tests +1. The patch passed core unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hama-Patch/143/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hama-Patch/143/artifact/trunk/build/reports/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hama-Patch/143/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hama-Patch/143/console
This message is automatically generated.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_newApproach_v01.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: HAMA-129_newApproach.patch
Here's the patch.
After collect blocks from each matrix to collectionTable, start multiplication.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657671#action_12657671 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
Each submatrix should be as large as possible. However, large cell causes RetriesExhaustedException during HTable.commit.
Looks like HADOOP-4802 (in hbase, its part of HBASE-900). Big cells can trigger OOME.
Let's update hadoop/hbase.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657671#action_12657671 ]
udanax edited comment on HAMA-129 at 12/17/08 8:48 PM:
---------------------------------------------------------------
Each submatrix should be as large as possible. However, large cell causes RetriesExhaustedException during HTable.commit.
>> Looks like HADOOP-4802 (in hbase, its part of HBASE-900). Big cells can trigger OOME.
I would like to update hadoop/hbase. I think we can't release with hadoop/hbase 0.18.x
was (Author: udanax):
Each submatrix should be as large as possible. However, large cell causes RetriesExhaustedException during HTable.commit.
Looks like HADOOP-4802 (in hbase, its part of HBASE-900). Big cells can trigger OOME.
Let's update hadoop/hbase.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: patch02.txt
test patch 2
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656956#action_12656956 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
5000 * 5000, 4*4 = 16 blocks
I'll trace it. But, I guess it's a really null.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657974#action_12657974 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
HBASE-867 also related with this.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: HAMA-129_newApproach_v01.patch
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_newApproach_v01.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661497#action_12661497 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
> If we collect blocks to one table during blocking_mapred(), locality
> will be provided and more faster.
>
> row Key column:A column:B
> c(0, 0) += a(0, 0) * b(0, 0)
> c(0, 0) += a(0, 1) * b(1, 0)
> c(0, 0) += a(0, 2) * b(2, 0)
> c(0, 0) += a(0, 3) * b(3, 0)
> c(0, 1) += a(0, 0) * b(0, 1)
> c(0, 1) += a(0, 1) * b(1, 1)
> ...
I'll test this approach.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Status: Open (was: Patch Available)
>> Retries exhausted, it took too long to wait for the table DenseMatrix_randeyuvg to be disabled.
It's a hbase bug. I just committed this.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663340#action_12663340 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
OK, I'll
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Status: Patch Available (was: Open)
Current trunk is buggy. I'll preferentially commit this patch.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: HAMA-129_v05.patch
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Attachment: HAMA-129_v06.patch
OOME fixed.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659802#action_12659802 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
3000 * 3000 matrices multiplication result:
[d8g053:/root/hama-trunk]# bin/hama examples mult AAA BBB 25
08/12/30 20:54:11 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/30 20:54:15 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randwzruq
08/12/30 20:54:15 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
e arguments. Applications should implement Tool for the same.
08/12/30 20:54:15 WARN mapred.JobClient: Use genericOptions for the option -libj
ars
08/12/30 20:54:15 WARN mapred.JobClient: No job jar file set. User classes may
not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/30 20:54:15 INFO mapred.TableInputFormatBase: split: 0->d8g054.nhncorp.com
:,000000000001678
08/12/30 20:54:15 INFO mapred.TableInputFormatBase: split: 1->d8g054.nhncorp.com
:000000000001678,
08/12/30 20:54:15 INFO mapred.JobClient: Running job: job_200812301027_0061
08/12/30 20:54:16 INFO mapred.JobClient: map 0% reduce 0%
08/12/30 20:56:03 INFO mapred.JobClient: map 50% reduce 0%
08/12/30 20:56:16 INFO mapred.JobClient: map 50% reduce 8%
08/12/30 20:56:17 INFO mapred.JobClient: map 50% reduce 16%
08/12/30 20:56:18 INFO mapred.JobClient: map 100% reduce 16%
08/12/30 20:56:28 INFO mapred.JobClient: map 100% reduce 58%
08/12/30 20:56:32 INFO mapred.JobClient: map 100% reduce 100%
08/12/30 20:56:41 INFO mapred.JobClient: Job complete: job_200812301027_0061
08/12/30 20:56:41 INFO mapred.JobClient: Counters: 15
08/12/30 20:56:41 INFO mapred.JobClient: File Systems
08/12/30 20:56:41 INFO mapred.JobClient: Local bytes read=606272934
08/12/30 20:56:41 INFO mapred.JobClient: Local bytes written=909388562
08/12/30 20:56:41 INFO mapred.JobClient: Job Counters
08/12/30 20:56:41 INFO mapred.JobClient: Launched reduce tasks=3
08/12/30 20:56:41 INFO mapred.JobClient: Rack-local map tasks=2
08/12/30 20:56:41 INFO mapred.JobClient: Launched map tasks=3
08/12/30 20:56:41 INFO mapred.JobClient: Data-local map tasks=1
08/12/30 20:56:41 INFO mapred.JobClient: Map-Reduce Framework
08/12/30 20:56:41 INFO mapred.JobClient: Reduce input groups=25
08/12/30 20:56:41 INFO mapred.JobClient: Combine output records=0
08/12/30 20:56:41 INFO mapred.JobClient: Map input records=3000
08/12/30 20:56:41 INFO mapred.JobClient: Reduce output records=0
08/12/30 20:56:41 INFO mapred.JobClient: Map output bytes=303069450
08/12/30 20:56:41 INFO mapred.JobClient: Map input bytes=0
08/12/30 20:56:41 INFO mapred.JobClient: Combine input records=0
08/12/30 20:56:41 INFO mapred.JobClient: Map output records=15000
08/12/30 20:56:41 INFO mapred.JobClient: Reduce input records=15000
08/12/30 20:56:41 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/30 20:56:45 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randqkczr
08/12/30 20:56:45 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
e arguments. Applications should implement Tool for the same.
08/12/30 20:56:45 WARN mapred.JobClient: Use genericOptions for the option -libj
ars
08/12/30 20:56:45 WARN mapred.JobClient: No job jar file set. User classes may
not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/30 20:56:45 INFO mapred.TableInputFormatBase: split: 0->d8g053.nhncorp.com
:,000000000001547
08/12/30 20:56:45 INFO mapred.TableInputFormatBase: split: 1->d8g053.nhncorp.com
:000000000001547,
08/12/30 20:56:45 INFO mapred.JobClient: Running job: job_200812301027_0062
08/12/30 20:56:46 INFO mapred.JobClient: map 0% reduce 0%
08/12/30 20:58:40 INFO mapred.JobClient: map 50% reduce 0%
08/12/30 20:58:41 INFO mapred.JobClient: map 100% reduce 0%
08/12/30 20:58:50 INFO mapred.JobClient: map 100% reduce 41%
08/12/30 20:58:51 INFO mapred.JobClient: map 100% reduce 92%
08/12/30 20:58:53 INFO mapred.JobClient: map 100% reduce 100%
08/12/30 20:59:03 INFO mapred.JobClient: Job complete: job_200812301027_0062
08/12/30 20:59:03 INFO mapred.JobClient: Counters: 15
08/12/30 20:59:03 INFO mapred.JobClient: File Systems
08/12/30 20:59:03 INFO mapred.JobClient: Local bytes read=606270874
08/12/30 20:59:03 INFO mapred.JobClient: Local bytes written=909388550
08/12/30 20:59:03 INFO mapred.JobClient: Job Counters
08/12/30 20:59:03 INFO mapred.JobClient: Launched reduce tasks=3
08/12/30 20:59:03 INFO mapred.JobClient: Rack-local map tasks=1
08/12/30 20:59:03 INFO mapred.JobClient: Launched map tasks=3
08/12/30 20:59:03 INFO mapred.JobClient: Data-local map tasks=2
08/12/30 20:59:03 INFO mapred.JobClient: Map-Reduce Framework
08/12/30 20:59:03 INFO mapred.JobClient: Reduce input groups=25
08/12/30 20:59:03 INFO mapred.JobClient: Combine output records=0
08/12/30 20:59:03 INFO mapred.JobClient: Map input records=3000
08/12/30 20:59:03 INFO mapred.JobClient: Reduce output records=0
08/12/30 20:59:03 INFO mapred.JobClient: Map output bytes=303069450
08/12/30 20:59:03 INFO mapred.JobClient: Map input bytes=0
08/12/30 20:59:03 INFO mapred.JobClient: Combine input records=0
08/12/30 20:59:03 INFO mapred.JobClient: Map output records=15000
08/12/30 20:59:03 INFO mapred.JobClient: Reduce input records=15000
08/12/30 20:59:03 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/30 20:59:07 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randluapj
08/12/30 20:59:07 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
e arguments. Applications should implement Tool for the same.
08/12/30 20:59:07 WARN mapred.JobClient: Use genericOptions for the option -libj
ars
08/12/30 20:59:08 WARN mapred.JobClient: No job jar file set. User classes may
not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/30 20:59:08 INFO mapred.TableInputFormatBase: split: 0->d8g053.nhncorp.com
:,
08/12/30 20:59:08 INFO mapred.JobClient: Running job: job_200812301027_0063
08/12/30 20:59:09 INFO mapred.JobClient: map 0% reduce 0%
08/12/30 21:03:37 INFO mapred.JobClient: map 100% reduce 0%
08/12/30 21:03:44 INFO mapred.JobClient: map 100% reduce 33%
08/12/30 21:03:47 INFO mapred.JobClient: map 100% reduce 67%
08/12/30 21:03:49 INFO mapred.JobClient: map 100% reduce 69%
08/12/30 21:03:54 INFO mapred.JobClient: map 100% reduce 70%
08/12/30 21:03:59 INFO mapred.JobClient: map 100% reduce 71%
08/12/30 21:04:03 INFO mapred.JobClient: map 100% reduce 72%
08/12/30 21:04:13 INFO mapred.JobClient: map 100% reduce 75%
08/12/30 21:04:17 INFO mapred.JobClient: map 100% reduce 76%
08/12/30 21:04:18 INFO mapred.JobClient: map 100% reduce 78%
08/12/30 21:04:28 INFO mapred.JobClient: map 100% reduce 79%
08/12/30 21:04:33 INFO mapred.JobClient: map 100% reduce 80%
08/12/30 21:04:38 INFO mapred.JobClient: map 100% reduce 81%
08/12/30 21:04:43 INFO mapred.JobClient: map 100% reduce 83%
08/12/30 21:04:46 INFO mapred.JobClient: map 100% reduce 84%
08/12/30 21:04:48 INFO mapred.JobClient: map 100% reduce 85%
08/12/30 21:04:49 INFO mapred.JobClient: map 100% reduce 86%
08/12/30 21:05:26 INFO mapred.JobClient: map 100% reduce 88%
08/12/30 21:05:56 INFO mapred.JobClient: map 100% reduce 89%
08/12/30 21:06:27 INFO mapred.JobClient: map 100% reduce 90%
08/12/30 21:06:57 INFO mapred.JobClient: map 100% reduce 92%
08/12/30 21:07:27 INFO mapred.JobClient: map 100% reduce 93%
08/12/30 21:07:52 INFO mapred.JobClient: map 100% reduce 95%
08/12/30 21:08:22 INFO mapred.JobClient: map 100% reduce 96%
08/12/30 21:08:47 INFO mapred.JobClient: map 100% reduce 97%
08/12/30 21:09:17 INFO mapred.JobClient: map 100% reduce 99%
08/12/30 21:09:44 INFO mapred.JobClient: map 100% reduce 100%
08/12/30 21:09:53 INFO mapred.JobClient: Job complete: job_200812301027_0063
08/12/30 21:09:53 INFO mapred.JobClient: Counters: 14
08/12/30 21:09:53 INFO mapred.JobClient: File Systems
08/12/30 21:09:53 INFO mapred.JobClient: Local bytes read=720028247
08/12/30 21:09:53 INFO mapred.JobClient: Local bytes written=1080011761
08/12/30 21:09:53 INFO mapred.JobClient: Job Counters
08/12/30 21:09:53 INFO mapred.JobClient: Launched reduce tasks=2
08/12/30 21:09:53 INFO mapred.JobClient: Launched map tasks=1
08/12/30 21:09:53 INFO mapred.JobClient: Data-local map tasks=1
08/12/30 21:09:53 INFO mapred.JobClient: Map-Reduce Framework
08/12/30 21:09:53 INFO mapred.JobClient: Reduce input groups=25
08/12/30 21:09:53 INFO mapred.JobClient: Combine output records=0
08/12/30 21:09:53 INFO mapred.JobClient: Map input records=25
08/12/30 21:09:53 INFO mapred.JobClient: Reduce output records=15000
08/12/30 21:09:53 INFO mapred.JobClient: Map output bytes=360003250
08/12/30 21:09:53 INFO mapred.JobClient: Map input bytes=0
08/12/30 21:09:53 INFO mapred.JobClient: Combine input records=0
08/12/30 21:09:53 INFO mapred.JobClient: Map output records=125
08/12/30 21:09:53 INFO mapred.JobClient: Reduce input records=125
08/12/30 21:10:43 INFO client.HBaseAdmin: Disabled DenseMatrix_randluapj
08/12/30 21:10:43 INFO client.HBaseAdmin: Deleted DenseMatrix_randluapj
[d8g053:/root/hama-trunk]#
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655483#action_12655483 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
5000 * 5000 matrix multiplication, blocks 16, 4 node
----
[d8g053:/root/hama-trunk]# bin/hama examples mult fiveA fiveB 16
08/12/11 11:45:22 INFO hama.AbstractMatrix: Create 4 * 4 blocked matrix
08/12/11 11:45:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
08/12/11 11:45:22 WARN mapred.JobClient: Use genericOptions for the option -libjars
08/12/11 11:45:22 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/11 11:45:22 INFO mapred.JobClient: Running job: job_200812100956_0045
08/12/11 11:45:23 INFO mapred.JobClient: map 0% reduce 0%
08/12/11 11:46:05 INFO mapred.JobClient: map 25% reduce 0%
08/12/11 11:46:14 INFO mapred.JobClient: map 25% reduce 2%
08/12/11 11:46:15 INFO mapred.JobClient: map 25% reduce 4%
08/12/11 11:46:16 INFO mapred.JobClient: map 25% reduce 6%
08/12/11 11:46:18 INFO mapred.JobClient: map 25% reduce 8%
08/12/11 11:47:58 INFO mapred.JobClient: map 50% reduce 8%
08/12/11 11:48:02 INFO mapred.JobClient: map 75% reduce 8%
08/12/11 11:48:03 INFO mapred.JobClient: map 75% reduce 10%
08/12/11 11:48:07 INFO mapred.JobClient: map 75% reduce 14%
08/12/11 11:48:08 INFO mapred.JobClient: map 75% reduce 18%
08/12/11 11:48:11 INFO mapred.JobClient: map 75% reduce 22%
08/12/11 11:48:15 INFO mapred.JobClient: map 75% reduce 25%
08/12/11 11:49:11 INFO mapred.JobClient: map 100% reduce 25%
08/12/11 11:49:17 INFO mapred.JobClient: map 100% reduce 35%
08/12/11 11:49:18 INFO mapred.JobClient: map 100% reduce 45%
08/12/11 11:49:20 INFO mapred.JobClient: map 100% reduce 47%
08/12/11 11:49:21 INFO mapred.JobClient: map 100% reduce 60%
08/12/11 11:49:22 INFO mapred.JobClient: map 100% reduce 70%
08/12/11 11:49:25 INFO mapred.JobClient: map 100% reduce 72%
08/12/11 11:49:26 INFO mapred.JobClient: map 100% reduce 81%
08/12/11 11:49:27 INFO mapred.JobClient: map 100% reduce 85%
08/12/11 11:49:30 INFO mapred.JobClient: map 100% reduce 87%
08/12/11 11:49:31 INFO mapred.JobClient: map 100% reduce 89%
08/12/11 11:49:33 INFO mapred.JobClient: map 100% reduce 95%
08/12/11 11:49:36 INFO mapred.JobClient: map 100% reduce 97%
08/12/11 11:49:39 INFO mapred.JobClient: Job complete: job_200812100956_0045
08/12/11 11:49:39 INFO mapred.JobClient: Counters: 13
08/12/11 11:49:39 INFO mapred.JobClient: File Systems
08/12/11 11:49:39 INFO mapred.JobClient: Local bytes read=1656774062
08/12/11 11:49:39 INFO mapred.JobClient: Local bytes written=2506547810
08/12/11 11:49:39 INFO mapred.JobClient: Job Counters
08/12/11 11:49:39 INFO mapred.JobClient: Launched reduce tasks=4
08/12/11 11:49:39 INFO mapred.JobClient: Launched map tasks=7
08/12/11 11:49:39 INFO mapred.JobClient: Map-Reduce Framework
08/12/11 11:49:39 INFO mapred.JobClient: Reduce input groups=16
08/12/11 11:49:39 INFO mapred.JobClient: Combine output records=0
08/12/11 11:49:39 INFO mapred.JobClient: Map input records=5000
08/12/11 11:49:39 INFO mapred.JobClient: Reduce output records=0
08/12/11 11:49:39 INFO mapred.JobClient: Map output bytes=844785560
08/12/11 11:49:39 INFO mapred.JobClient: Map input bytes=0
08/12/11 11:49:39 INFO mapred.JobClient: Combine input records=0
08/12/11 11:49:39 INFO mapred.JobClient: Map output records=20000
08/12/11 11:49:39 INFO mapred.JobClient: Reduce input records=20000
08/12/11 11:49:40 INFO hama.AbstractMatrix: Create 4 * 4 blocked matrix
08/12/11 11:49:40 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
08/12/11 11:49:40 WARN mapred.JobClient: Use genericOptions for the option -libjars
08/12/11 11:49:40 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/11 11:49:40 INFO mapred.JobClient: Running job: job_200812100956_0046
08/12/11 11:49:41 INFO mapred.JobClient: map 0% reduce 0%
08/12/11 11:50:13 INFO mapred.JobClient: map 25% reduce 0%
08/12/11 11:50:21 INFO mapred.JobClient: map 25% reduce 2%
08/12/11 11:50:23 INFO mapred.JobClient: map 25% reduce 4%
08/12/11 11:50:26 INFO mapred.JobClient: map 25% reduce 6%
08/12/11 11:50:27 INFO mapred.JobClient: map 25% reduce 8%
08/12/11 11:52:20 INFO mapred.JobClient: map 50% reduce 8%
08/12/11 11:52:21 INFO mapred.JobClient: map 75% reduce 8%
08/12/11 11:52:29 INFO mapred.JobClient: map 75% reduce 12%
08/12/11 11:52:32 INFO mapred.JobClient: map 75% reduce 16%
08/12/11 11:52:33 INFO mapred.JobClient: map 75% reduce 25%
08/12/11 11:54:03 INFO mapred.JobClient: map 100% reduce 25%
08/12/11 11:54:11 INFO mapred.JobClient: map 100% reduce 35%
08/12/11 11:54:14 INFO mapred.JobClient: map 100% reduce 37%
08/12/11 11:54:17 INFO mapred.JobClient: map 100% reduce 47%
08/12/11 11:54:18 INFO mapred.JobClient: map 100% reduce 72%
08/12/11 11:54:20 INFO mapred.JobClient: map 100% reduce 75%
08/12/11 11:54:22 INFO mapred.JobClient: map 100% reduce 77%
08/12/11 11:54:23 INFO mapred.JobClient: map 100% reduce 83%
08/12/11 11:54:24 INFO mapred.JobClient: map 100% reduce 85%
08/12/11 11:54:25 INFO mapred.JobClient: map 100% reduce 87%
08/12/11 11:54:26 INFO mapred.JobClient: map 100% reduce 89%
08/12/11 11:54:27 INFO mapred.JobClient: map 100% reduce 95%
08/12/11 11:54:32 INFO mapred.JobClient: Job complete: job_200812100956_0046
08/12/11 11:54:32 INFO mapred.JobClient: Counters: 13
08/12/11 11:54:32 INFO mapred.JobClient: File Systems
08/12/11 11:54:32 INFO mapred.JobClient: Local bytes read=1656774062
08/12/11 11:54:32 INFO mapred.JobClient: Local bytes written=2506547810
08/12/11 11:54:32 INFO mapred.JobClient: Job Counters
08/12/11 11:54:32 INFO mapred.JobClient: Launched reduce tasks=5
08/12/11 11:54:32 INFO mapred.JobClient: Launched map tasks=7
08/12/11 11:54:32 INFO mapred.JobClient: Map-Reduce Framework
08/12/11 11:54:32 INFO mapred.JobClient: Reduce input groups=16
08/12/11 11:54:32 INFO mapred.JobClient: Combine output records=0
08/12/11 11:54:32 INFO mapred.JobClient: Map input records=5000
08/12/11 11:54:32 INFO mapred.JobClient: Reduce output records=0
08/12/11 11:54:32 INFO mapred.JobClient: Map output bytes=844785560
08/12/11 11:54:32 INFO mapred.JobClient: Map input bytes=0
08/12/11 11:54:32 INFO mapred.JobClient: Combine input records=0
08/12/11 11:54:32 INFO mapred.JobClient: Map output records=20000
08/12/11 11:54:32 INFO mapred.JobClient: Reduce input records=20000
08/12/11 11:54:32 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/11 11:54:38 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randbcfpq
08/12/11 11:54:38 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
08/12/11 11:54:38 WARN mapred.JobClient: Use genericOptions for the option -libjars
08/12/11 11:54:38 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/11 11:54:38 INFO mapred.JobClient: Running job: job_200812100956_0047
08/12/11 11:54:39 INFO mapred.JobClient: map 0% reduce 0%
08/12/11 11:57:30 INFO mapred.JobClient: map 50% reduce 0%
08/12/11 11:57:37 INFO mapred.JobClient: map 50% reduce 4%
08/12/11 11:57:41 INFO mapred.JobClient: map 50% reduce 8%
08/12/11 11:57:42 INFO mapred.JobClient: map 50% reduce 12%
08/12/11 11:57:47 INFO mapred.JobClient: map 50% reduce 16%
08/12/11 12:00:31 INFO mapred.JobClient: map 100% reduce 16%
08/12/11 12:00:38 INFO mapred.JobClient: map 100% reduce 29%
08/12/11 12:00:43 INFO mapred.JobClient: map 100% reduce 42%
08/12/11 12:00:46 INFO mapred.JobClient: map 100% reduce 46%
08/12/11 12:00:47 INFO mapred.JobClient: map 100% reduce 59%
08/12/11 12:00:51 INFO mapred.JobClient: map 100% reduce 68%
08/12/11 12:07:08 INFO mapred.JobClient: map 100% reduce 76%
08/12/11 12:07:39 INFO mapred.JobClient: map 100% reduce 84%
08/12/11 12:07:40 INFO mapred.JobClient: map 100% reduce 92%
08/12/11 12:07:45 INFO mapred.JobClient: Job complete: job_200812100956_0047
08/12/11 12:07:45 INFO mapred.JobClient: Counters: 13
08/12/11 12:07:45 INFO mapred.JobClient: File Systems
08/12/11 12:07:45 INFO mapred.JobClient: Local bytes read=1639585414
08/12/11 12:07:45 INFO mapred.JobClient: Local bytes written=2421198052
08/12/11 12:07:45 INFO mapred.JobClient: Job Counters
08/12/11 12:07:45 INFO mapred.JobClient: Launched reduce tasks=5
08/12/11 12:07:45 INFO mapred.JobClient: Launched map tasks=3
08/12/11 12:07:45 INFO mapred.JobClient: Map-Reduce Framework
08/12/11 12:07:45 INFO mapred.JobClient: Reduce input groups=4
08/12/11 12:07:45 INFO mapred.JobClient: Combine output records=0
08/12/11 12:07:45 INFO mapred.JobClient: Map input records=4
08/12/11 12:07:45 INFO mapred.JobClient: Reduce output records=20000
08/12/11 12:07:45 INFO mapred.JobClient: Map output bytes=800808832
08/12/11 12:07:45 INFO mapred.JobClient: Map input bytes=0
08/12/11 12:07:45 INFO mapred.JobClient: Combine input records=0
08/12/11 12:07:45 INFO mapred.JobClient: Map output records=64
08/12/11 12:07:45 INFO mapred.JobClient: Reduce input records=64
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663249#action_12663249 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
A* B of 8,000 * 8,000 matrices Test result:
[d8g053:/root/hama-trunk]# bin/hama examples random -m 10 -r 10 8000 8000 eightB
09/01/13 13:20:18 INFO hama.AbstractMatrix: Initializing the matrix storage.
09/01/13 13:20:22 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randmuyao
09/01/13 13:20:22 INFO hama.AbstractMatrix: Create the 8000 * 8000 random matrix : DenseMatrix_randmuyao
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
09/01/13 13:20:24 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
09/01/13 13:20:24 WARN mapred.JobClient: Use genericOptions for the option -libjars
09/01/13 13:20:24 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
09/01/13 13:20:24 INFO mapred.FileInputFormat: Total input paths to process : 10
09/01/13 13:20:25 INFO mapred.JobClient: Running job: job_200901131229_0007
09/01/13 13:20:26 INFO mapred.JobClient: map 0% reduce 0%
09/01/13 13:20:59 INFO mapred.JobClient: map 10% reduce 0%
09/01/13 13:21:01 INFO mapred.JobClient: map 20% reduce 0%
09/01/13 13:21:02 INFO mapred.JobClient: map 30% reduce 0%
09/01/13 13:21:03 INFO mapred.JobClient: map 40% reduce 0%
09/01/13 13:21:06 INFO mapred.JobClient: map 50% reduce 0%
09/01/13 13:21:08 INFO mapred.JobClient: map 60% reduce 1%
09/01/13 13:21:12 INFO mapred.JobClient: map 60% reduce 4%
09/01/13 13:21:14 INFO mapred.JobClient: map 60% reduce 7%
09/01/13 13:21:17 INFO mapred.JobClient: map 60% reduce 8%
09/01/13 13:21:22 INFO mapred.JobClient: map 60% reduce 10%
09/01/13 13:21:26 INFO mapred.JobClient: map 70% reduce 11%
09/01/13 13:21:28 INFO mapred.JobClient: map 80% reduce 11%
09/01/13 13:21:29 INFO mapred.JobClient: map 80% reduce 12%
09/01/13 13:21:30 INFO mapred.JobClient: map 90% reduce 12%
09/01/13 13:21:31 INFO mapred.JobClient: map 100% reduce 12%
09/01/13 13:21:36 INFO mapred.JobClient: map 100% reduce 13%
09/01/13 13:21:38 INFO mapred.JobClient: map 100% reduce 14%
09/01/13 13:21:40 INFO mapred.JobClient: map 100% reduce 19%
09/01/13 13:21:41 INFO mapred.JobClient: map 100% reduce 27%
09/01/13 13:21:43 INFO mapred.JobClient: map 100% reduce 31%
09/01/13 13:21:45 INFO mapred.JobClient: map 100% reduce 36%
09/01/13 13:21:48 INFO mapred.JobClient: map 100% reduce 40%
09/01/13 13:22:11 INFO mapred.JobClient: map 100% reduce 41%
09/01/13 13:22:38 INFO mapred.JobClient: map 100% reduce 42%
09/01/13 13:22:59 INFO mapred.JobClient: map 100% reduce 43%
09/01/13 13:23:29 INFO mapred.JobClient: map 100% reduce 44%
09/01/13 13:23:47 INFO mapred.JobClient: map 100% reduce 45%
09/01/13 13:24:12 INFO mapred.JobClient: map 100% reduce 46%
09/01/13 13:24:30 INFO mapred.JobClient: map 100% reduce 47%
09/01/13 13:24:57 INFO mapred.JobClient: map 100% reduce 48%
09/01/13 13:25:16 INFO mapred.JobClient: map 100% reduce 49%
09/01/13 13:25:32 INFO mapred.JobClient: map 100% reduce 50%
09/01/13 13:25:48 INFO mapred.JobClient: map 100% reduce 51%
09/01/13 13:26:04 INFO mapred.JobClient: map 100% reduce 52%
09/01/13 13:26:20 INFO mapred.JobClient: map 100% reduce 53%
09/01/13 13:26:50 INFO mapred.JobClient: map 100% reduce 54%
09/01/13 13:27:07 INFO mapred.JobClient: map 100% reduce 55%
09/01/13 13:27:23 INFO mapred.JobClient: map 100% reduce 56%
09/01/13 13:27:38 INFO mapred.JobClient: map 100% reduce 57%
09/01/13 13:27:54 INFO mapred.JobClient: map 100% reduce 58%
09/01/13 13:28:11 INFO mapred.JobClient: map 100% reduce 59%
09/01/13 13:28:26 INFO mapred.JobClient: map 100% reduce 60%
09/01/13 13:28:27 INFO mapred.JobClient: map 100% reduce 63%
09/01/13 13:28:28 INFO mapred.JobClient: map 100% reduce 72%
09/01/13 13:28:31 INFO mapred.JobClient: map 100% reduce 79%
09/01/13 13:28:32 INFO mapred.JobClient: map 100% reduce 82%
09/01/13 13:28:33 INFO mapred.JobClient: map 100% reduce 86%
09/01/13 13:28:37 INFO mapred.JobClient: map 100% reduce 87%
09/01/13 13:28:52 INFO mapred.JobClient: map 100% reduce 88%
09/01/13 13:29:08 INFO mapred.JobClient: map 100% reduce 89%
09/01/13 13:29:22 INFO mapred.JobClient: map 100% reduce 90%
09/01/13 13:29:43 INFO mapred.JobClient: map 100% reduce 91%
09/01/13 13:30:02 INFO mapred.JobClient: map 100% reduce 92%
09/01/13 13:30:18 INFO mapred.JobClient: map 100% reduce 93%
09/01/13 13:30:34 INFO mapred.JobClient: map 100% reduce 94%
09/01/13 13:30:49 INFO mapred.JobClient: map 100% reduce 95%
09/01/13 13:31:04 INFO mapred.JobClient: map 100% reduce 96%
09/01/13 13:31:19 INFO mapred.JobClient: map 100% reduce 97%
09/01/13 13:31:36 INFO mapred.JobClient: map 100% reduce 98%
09/01/13 13:31:51 INFO mapred.JobClient: map 100% reduce 99%
09/01/13 13:32:05 INFO mapred.JobClient: map 100% reduce 100%
09/01/13 13:32:06 INFO mapred.JobClient: Job complete: job_200901131229_0007
09/01/13 13:32:06 INFO mapred.JobClient: Counters: 15
09/01/13 13:32:06 INFO mapred.JobClient: File Systems
09/01/13 13:32:06 INFO mapred.JobClient: HDFS bytes read=1080
09/01/13 13:32:06 INFO mapred.JobClient: Local bytes read=4335630656
09/01/13 13:32:06 INFO mapred.JobClient: Local bytes written=6501793690
09/01/13 13:32:06 INFO mapred.JobClient: Job Counters
09/01/13 13:32:06 INFO mapred.JobClient: Launched reduce tasks=10
09/01/13 13:32:06 INFO mapred.JobClient: Launched map tasks=10
09/01/13 13:32:06 INFO mapred.JobClient: Data-local map tasks=10
09/01/13 13:32:06 INFO mapred.JobClient: Map-Reduce Framework
09/01/13 13:32:06 INFO mapred.JobClient: Reduce input groups=8000
09/01/13 13:32:06 INFO mapred.JobClient: Combine output records=0
09/01/13 13:32:06 INFO mapred.JobClient: Map input records=10
09/01/13 13:32:06 INFO mapred.JobClient: Reduce output records=8000
09/01/13 13:32:06 INFO mapred.JobClient: Map output bytes=2167222890
09/01/13 13:32:06 INFO mapred.JobClient: Map input bytes=160
09/01/13 13:32:06 INFO mapred.JobClient: Combine input records=0
09/01/13 13:32:06 INFO mapred.JobClient: Map output records=8000
09/01/13 13:32:06 INFO mapred.JobClient: Reduce input records=8000
[d8g053:/root/hama-trunk]# bin/hama examples mult -m 10 -r 10 eightA eightB 1600
09/01/13 13:40:12 INFO hama.AbstractMatrix: Initializing the matrix storage.
09/01/13 13:40:17 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randujbhy
09/01/13 13:40:17 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
09/01/13 13:40:17 WARN mapred.JobClient: Use genericOptions for the option -libjars
09/01/13 13:40:17 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
09/01/13 13:40:17 INFO mapred.TableInputFormatBase: split: 0->d8g055.nhncorp.com:,000000000000850
09/01/13 13:40:17 INFO mapred.TableInputFormatBase: split: 1->d8g054.nhncorp.com:000000000000850,000000000001802
09/01/13 13:40:17 INFO mapred.TableInputFormatBase: split: 2->d8g054.nhncorp.com:000000000001802,000000000002505
09/01/13 13:40:17 INFO mapred.TableInputFormatBase: split: 3->d8g055.nhncorp.com:000000000002505,000000000003213
09/01/13 13:40:17 INFO mapred.TableInputFormatBase: split: 4->d8g053.nhncorp.com:000000000003213,000000000003952
09/01/13 13:40:17 INFO mapred.TableInputFormatBase: split: 5->d8g055.nhncorp.com:000000000003952,000000000004775
09/01/13 13:40:17 INFO mapred.TableInputFormatBase: split: 6->d8g053.nhncorp.com:000000000004775,000000000005602
09/01/13 13:40:17 INFO mapred.TableInputFormatBase: split: 7->d8g053.nhncorp.com:000000000005602,000000000006644
09/01/13 13:40:17 INFO mapred.TableInputFormatBase: split: 8->d8g054.nhncorp.com:000000000006644,000000000007321
09/01/13 13:40:17 INFO mapred.TableInputFormatBase: split: 9->d8g054.nhncorp.com:000000000007321,
09/01/13 13:40:17 INFO mapred.JobClient: Running job: job_200901131229_0008
09/01/13 13:40:18 INFO mapred.JobClient: map 0% reduce 0%
09/01/13 13:42:38 INFO mapred.JobClient: map 10% reduce 0%
09/01/13 13:42:45 INFO mapred.JobClient: map 20% reduce 0%
09/01/13 13:42:47 INFO mapred.JobClient: map 20% reduce 1%
09/01/13 13:42:49 INFO mapred.JobClient: map 30% reduce 2%
09/01/13 13:42:59 INFO mapred.JobClient: map 30% reduce 3%
09/01/13 13:43:00 INFO mapred.JobClient: map 30% reduce 4%
09/01/13 13:43:03 INFO mapred.JobClient: map 30% reduce 5%
09/01/13 13:43:05 INFO mapred.JobClient: map 30% reduce 6%
09/01/13 13:43:08 INFO mapred.JobClient: map 40% reduce 6%
09/01/13 13:43:13 INFO mapred.JobClient: map 50% reduce 6%
09/01/13 13:43:16 INFO mapred.JobClient: map 60% reduce 6%
09/01/13 13:43:21 INFO mapred.JobClient: map 60% reduce 8%
09/01/13 13:43:23 INFO mapred.JobClient: map 60% reduce 10%
09/01/13 13:43:28 INFO mapred.JobClient: map 60% reduce 11%
09/01/13 13:43:33 INFO mapred.JobClient: map 60% reduce 12%
09/01/13 13:44:59 INFO mapred.JobClient: map 70% reduce 12%
09/01/13 13:45:07 INFO mapred.JobClient: map 80% reduce 12%
09/01/13 13:45:12 INFO mapred.JobClient: map 80% reduce 13%
09/01/13 13:45:14 INFO mapred.JobClient: map 80% reduce 14%
09/01/13 13:45:19 INFO mapred.JobClient: map 80% reduce 15%
09/01/13 13:45:28 INFO mapred.JobClient: map 90% reduce 15%
09/01/13 13:45:37 INFO mapred.JobClient: map 90% reduce 16%
09/01/13 13:45:42 INFO mapred.JobClient: map 90% reduce 17%
09/01/13 13:45:43 INFO mapred.JobClient: map 100% reduce 17%
09/01/13 13:45:49 INFO mapred.JobClient: map 100% reduce 21%
09/01/13 13:45:53 INFO mapred.JobClient: map 100% reduce 25%
09/01/13 13:45:54 INFO mapred.JobClient: map 100% reduce 29%
09/01/13 13:45:55 INFO mapred.JobClient: map 100% reduce 32%
09/01/13 13:45:58 INFO mapred.JobClient: map 100% reduce 36%
09/01/13 13:45:59 INFO mapred.JobClient: map 100% reduce 40%
09/01/13 13:46:35 INFO mapred.JobClient: map 100% reduce 41%
09/01/13 13:47:28 INFO mapred.JobClient: map 100% reduce 42%
09/01/13 13:48:32 INFO mapred.JobClient: map 100% reduce 43%
09/01/13 13:49:44 INFO mapred.JobClient: map 100% reduce 44%
09/01/13 13:50:35 INFO mapred.JobClient: map 100% reduce 45%
09/01/13 13:51:25 INFO mapred.JobClient: map 100% reduce 46%
09/01/13 13:52:29 INFO mapred.JobClient: map 100% reduce 47%
09/01/13 13:53:39 INFO mapred.JobClient: map 100% reduce 48%
09/01/13 13:54:35 INFO mapred.JobClient: map 100% reduce 49%
09/01/13 13:55:50 INFO mapred.JobClient: map 100% reduce 50%
09/01/13 13:56:33 INFO mapred.JobClient: map 100% reduce 51%
09/01/13 13:57:26 INFO mapred.JobClient: map 100% reduce 52%
09/01/13 13:58:26 INFO mapred.JobClient: map 100% reduce 53%
09/01/13 13:59:19 INFO mapred.JobClient: map 100% reduce 54%
09/01/13 14:00:08 INFO mapred.JobClient: map 100% reduce 55%
09/01/13 14:00:13 INFO mapred.JobClient: map 100% reduce 57%
09/01/13 14:00:18 INFO mapred.JobClient: map 100% reduce 58%
09/01/13 14:00:28 INFO mapred.JobClient: map 100% reduce 61%
09/01/13 14:00:35 INFO mapred.JobClient: map 100% reduce 62%
09/01/13 14:01:19 INFO mapred.JobClient: map 100% reduce 63%
09/01/13 14:02:03 INFO mapred.JobClient: map 100% reduce 64%
09/01/13 14:03:02 INFO mapred.JobClient: map 100% reduce 65%
09/01/13 14:03:42 INFO mapred.JobClient: map 100% reduce 66%
09/01/13 14:04:53 INFO mapred.JobClient: map 100% reduce 67%
09/01/13 14:05:31 INFO mapred.JobClient: map 100% reduce 69%
09/01/13 14:05:36 INFO mapred.JobClient: map 100% reduce 70%
09/01/13 14:05:41 INFO mapred.JobClient: map 100% reduce 74%
09/01/13 14:05:59 INFO mapred.JobClient: map 100% reduce 75%
09/01/13 14:06:38 INFO mapred.JobClient: map 100% reduce 78%
09/01/13 14:06:42 INFO mapred.JobClient: map 100% reduce 79%
09/01/13 14:06:43 INFO mapred.JobClient: map 100% reduce 82%
09/01/13 14:06:47 INFO mapred.JobClient: map 100% reduce 83%
09/01/13 14:06:52 INFO mapred.JobClient: map 100% reduce 85%
09/01/13 14:06:58 INFO mapred.JobClient: map 100% reduce 86%
09/01/13 14:07:02 INFO mapred.JobClient: map 100% reduce 89%
09/01/13 14:07:21 INFO mapred.JobClient: map 100% reduce 90%
09/01/13 14:08:21 INFO mapred.JobClient: map 100% reduce 91%
09/01/13 14:09:17 INFO mapred.JobClient: map 100% reduce 92%
09/01/13 14:10:37 INFO mapred.JobClient: map 100% reduce 93%
09/01/13 14:11:59 INFO mapred.JobClient: map 100% reduce 94%
09/01/13 14:13:34 INFO mapred.JobClient: map 100% reduce 95%
09/01/13 14:14:44 INFO mapred.JobClient: map 100% reduce 96%
09/01/13 14:16:01 INFO mapred.JobClient: map 100% reduce 97%
09/01/13 14:17:32 INFO mapred.JobClient: map 100% reduce 98%
09/01/13 14:19:07 INFO mapred.JobClient: map 100% reduce 99%
09/01/13 14:20:29 INFO mapred.JobClient: map 100% reduce 100%
09/01/13 14:20:41 INFO mapred.JobClient: Job complete: job_200901131229_0008
09/01/13 14:20:41 INFO mapred.JobClient: Counters: 15
09/01/13 14:20:41 INFO mapred.JobClient: File Systems
09/01/13 14:20:41 INFO mapred.JobClient: Local bytes read=4355105034
09/01/13 14:20:41 INFO mapred.JobClient: Local bytes written=6530991934
09/01/13 14:20:41 INFO mapred.JobClient: Job Counters
09/01/13 14:20:41 INFO mapred.JobClient: Launched reduce tasks=12
09/01/13 14:20:41 INFO mapred.JobClient: Rack-local map tasks=4
09/01/13 14:20:41 INFO mapred.JobClient: Launched map tasks=14
09/01/13 14:20:41 INFO mapred.JobClient: Data-local map tasks=10
09/01/13 14:20:41 INFO mapred.JobClient: Map-Reduce Framework
09/01/13 14:20:41 INFO mapred.JobClient: Reduce input groups=1600
09/01/13 14:20:41 INFO mapred.JobClient: Combine output records=0
09/01/13 14:20:41 INFO mapred.JobClient: Map input records=8000
09/01/13 14:20:41 INFO mapred.JobClient: Reduce output records=64000
09/01/13 14:20:41 INFO mapred.JobClient: Map output bytes=2175715600
09/01/13 14:20:41 INFO mapred.JobClient: Map input bytes=0
09/01/13 14:20:41 INFO mapred.JobClient: Combine input records=0
09/01/13 14:20:41 INFO mapred.JobClient: Map output records=320000
09/01/13 14:20:41 INFO mapred.JobClient: Reduce input records=320000
09/01/13 14:20:43 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
09/01/13 14:20:43 WARN mapred.JobClient: Use genericOptions for the option -libjars
09/01/13 14:20:46 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
09/01/13 14:20:47 INFO mapred.TableInputFormatBase: split: 0->d8g055.nhncorp.com:,000000000000751
09/01/13 14:20:47 INFO mapred.TableInputFormatBase: split: 1->d8g053.nhncorp.com:000000000000751,000000000001365
09/01/13 14:20:47 INFO mapred.TableInputFormatBase: split: 2->d8g054.nhncorp.com:000000000001365,000000000002364
09/01/13 14:20:47 INFO mapred.TableInputFormatBase: split: 3->d8g054.nhncorp.com:000000000002364,000000000003271
09/01/13 14:20:47 INFO mapred.TableInputFormatBase: split: 4->d8g055.nhncorp.com:000000000003271,000000000004175
09/01/13 14:20:47 INFO mapred.TableInputFormatBase: split: 5->d8g053.nhncorp.com:000000000004175,000000000005069
09/01/13 14:20:47 INFO mapred.TableInputFormatBase: split: 6->d8g053.nhncorp.com:000000000005069,000000000005965
09/01/13 14:20:47 INFO mapred.TableInputFormatBase: split: 7->d8g054.nhncorp.com:000000000005965,000000000006982
09/01/13 14:20:47 INFO mapred.TableInputFormatBase: split: 8->d8g055.nhncorp.com:000000000006982,000000000007490
09/01/13 14:20:47 INFO mapred.TableInputFormatBase: split: 9->d8g055.nhncorp.com:000000000007490,
09/01/13 14:20:47 INFO mapred.JobClient: Running job: job_200901131229_0009
09/01/13 14:20:48 INFO mapred.JobClient: map 0% reduce 0%
09/01/13 14:24:01 INFO mapred.JobClient: map 10% reduce 0%
09/01/13 14:24:16 INFO mapred.JobClient: map 10% reduce 2%
09/01/13 14:24:22 INFO mapred.JobClient: map 20% reduce 2%
09/01/13 14:24:35 INFO mapred.JobClient: map 20% reduce 3%
09/01/13 14:24:37 INFO mapred.JobClient: map 20% reduce 4%
09/01/13 14:24:57 INFO mapred.JobClient: map 30% reduce 4%
09/01/13 14:25:07 INFO mapred.JobClient: map 40% reduce 4%
09/01/13 14:25:11 INFO mapred.JobClient: map 40% reduce 5%
09/01/13 14:25:13 INFO mapred.JobClient: map 40% reduce 6%
09/01/13 14:25:18 INFO mapred.JobClient: map 50% reduce 7%
09/01/13 14:25:22 INFO mapred.JobClient: map 60% reduce 7%
09/01/13 14:25:28 INFO mapred.JobClient: map 60% reduce 8%
09/01/13 14:25:31 INFO mapred.JobClient: map 60% reduce 9%
09/01/13 14:25:33 INFO mapred.JobClient: map 60% reduce 10%
09/01/13 14:25:36 INFO mapred.JobClient: map 60% reduce 11%
09/01/13 14:27:07 INFO mapred.JobClient: map 70% reduce 11%
09/01/13 14:27:14 INFO mapred.JobClient: map 70% reduce 12%
09/01/13 14:27:19 INFO mapred.JobClient: map 70% reduce 13%
09/01/13 14:28:05 INFO mapred.JobClient: map 80% reduce 13%
09/01/13 14:28:12 INFO mapred.JobClient: map 80% reduce 14%
09/01/13 14:28:20 INFO mapred.JobClient: map 80% reduce 15%
09/01/13 14:29:13 INFO mapred.JobClient: map 90% reduce 15%
09/01/13 14:29:23 INFO mapred.JobClient: map 90% reduce 16%
09/01/13 14:29:26 INFO mapred.JobClient: map 90% reduce 17%
09/01/13 14:29:56 INFO mapred.JobClient: map 100% reduce 17%
09/01/13 14:30:04 INFO mapred.JobClient: map 100% reduce 21%
09/01/13 14:30:05 INFO mapred.JobClient: map 100% reduce 25%
09/01/13 14:30:06 INFO mapred.JobClient: map 100% reduce 29%
09/01/13 14:30:09 INFO mapred.JobClient: map 100% reduce 32%
09/01/13 14:30:10 INFO mapred.JobClient: map 100% reduce 36%
09/01/13 14:30:11 INFO mapred.JobClient: map 100% reduce 40%
09/01/13 14:31:20 INFO mapred.JobClient: map 100% reduce 41%
09/01/13 14:32:57 INFO mapred.JobClient: map 100% reduce 42%
09/01/13 14:34:56 INFO mapred.JobClient: map 100% reduce 43%
09/01/13 14:36:02 INFO mapred.JobClient: map 100% reduce 44%
09/01/13 14:36:51 INFO mapred.JobClient: map 100% reduce 45%
09/01/13 14:38:20 INFO mapred.JobClient: map 100% reduce 46%
09/01/13 14:39:36 INFO mapred.JobClient: map 100% reduce 47%
09/01/13 14:40:39 INFO mapred.JobClient: map 100% reduce 48%
09/01/13 14:42:09 INFO mapred.JobClient: map 100% reduce 49%
09/01/13 14:43:10 INFO mapred.JobClient: map 100% reduce 50%
09/01/13 14:45:05 INFO mapred.JobClient: map 100% reduce 51%
09/01/13 14:47:17 INFO mapred.JobClient: map 100% reduce 52%
09/01/13 14:48:27 INFO mapred.JobClient: map 100% reduce 53%
09/01/13 14:49:25 INFO mapred.JobClient: map 100% reduce 54%
09/01/13 14:50:41 INFO mapred.JobClient: map 100% reduce 55%
09/01/13 14:51:37 INFO mapred.JobClient: map 100% reduce 56%
09/01/13 14:52:40 INFO mapred.JobClient: map 100% reduce 57%
09/01/13 14:53:37 INFO mapred.JobClient: map 100% reduce 58%
09/01/13 14:55:06 INFO mapred.JobClient: map 100% reduce 59%
09/01/13 14:55:42 INFO mapred.JobClient: map 100% reduce 62%
09/01/13 14:55:54 INFO mapred.JobClient: map 100% reduce 65%
09/01/13 14:55:59 INFO mapred.JobClient: map 100% reduce 72%
09/01/13 14:56:21 INFO mapred.JobClient: map 100% reduce 73%
09/01/13 14:56:57 INFO mapred.JobClient: map 100% reduce 75%
09/01/13 14:57:02 INFO mapred.JobClient: map 100% reduce 78%
09/01/13 14:57:07 INFO mapred.JobClient: map 100% reduce 79%
09/01/13 14:57:12 INFO mapred.JobClient: map 100% reduce 87%
09/01/13 14:58:20 INFO mapred.JobClient: map 100% reduce 88%
09/01/13 14:59:40 INFO mapred.JobClient: map 100% reduce 89%
09/01/13 15:01:03 INFO mapred.JobClient: map 100% reduce 90%
09/01/13 15:03:14 INFO mapred.JobClient: map 100% reduce 91%
09/01/13 15:05:03 INFO mapred.JobClient: map 100% reduce 92%
09/01/13 15:06:47 INFO mapred.JobClient: map 100% reduce 93%
09/01/13 15:09:11 INFO mapred.JobClient: map 100% reduce 94%
09/01/13 15:10:43 INFO mapred.JobClient: map 100% reduce 95%
09/01/13 15:11:58 INFO mapred.JobClient: map 100% reduce 96%
09/01/13 15:12:39 INFO mapred.JobClient: map 100% reduce 97%
09/01/13 15:13:33 INFO mapred.JobClient: map 100% reduce 98%
09/01/13 15:14:28 INFO mapred.JobClient: map 100% reduce 99%
09/01/13 15:15:32 INFO mapred.JobClient: map 100% reduce 100%
09/01/13 15:15:39 INFO mapred.JobClient: Job complete: job_200901131229_0009
09/01/13 15:15:39 INFO mapred.JobClient: Counters: 15
09/01/13 15:15:39 INFO mapred.JobClient: File Systems
09/01/13 15:15:39 INFO mapred.JobClient: Local bytes read=4355174656
09/01/13 15:15:39 INFO mapred.JobClient: Local bytes written=6530992048
09/01/13 15:15:39 INFO mapred.JobClient: Job Counters
09/01/13 15:15:39 INFO mapred.JobClient: Launched reduce tasks=12
09/01/13 15:15:39 INFO mapred.JobClient: Rack-local map tasks=4
09/01/13 15:15:39 INFO mapred.JobClient: Launched map tasks=14
09/01/13 15:15:39 INFO mapred.JobClient: Data-local map tasks=10
09/01/13 15:15:39 INFO mapred.JobClient: Map-Reduce Framework
09/01/13 15:15:39 INFO mapred.JobClient: Reduce input groups=1600
09/01/13 15:15:39 INFO mapred.JobClient: Combine output records=0
09/01/13 15:15:39 INFO mapred.JobClient: Map input records=8000
09/01/13 15:15:39 INFO mapred.JobClient: Reduce output records=64000
09/01/13 15:15:39 INFO mapred.JobClient: Map output bytes=2175715600
09/01/13 15:15:39 INFO mapred.JobClient: Map input bytes=0
09/01/13 15:15:39 INFO mapred.JobClient: Combine input records=0
09/01/13 15:15:39 INFO mapred.JobClient: Map output records=320000
09/01/13 15:15:39 INFO mapred.JobClient: Reduce input records=320000
09/01/13 15:15:40 INFO hama.AbstractMatrix: Initializing the matrix storage.
09/01/13 15:15:44 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randcrwcp
09/01/13 15:15:44 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
09/01/13 15:15:44 WARN mapred.JobClient: Use genericOptions for the option -libjars
09/01/13 15:15:44 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
09/01/13 15:15:44 INFO mapred.TableInputFormatBase: split: 0->d8g054.nhncorp.com:,00000000000,15,35-25403
09/01/13 15:15:44 INFO mapred.TableInputFormatBase: split: 1->d8g055.nhncorp.com:00000000000,15,35-25403,00000000000,20,17-32709
09/01/13 15:15:44 INFO mapred.TableInputFormatBase: split: 2->d8g054.nhncorp.com:00000000000,20,17-32709,00000000000,24,26-39446
09/01/13 15:15:44 INFO mapred.TableInputFormatBase: split: 3->d8g053.nhncorp.com:00000000000,24,26-39446,00000000000,28,39-46363
09/01/13 15:15:44 INFO mapred.TableInputFormatBase: split: 4->d8g053.nhncorp.com:00000000000,28,39-46363,00000000000,32,19-51978
09/01/13 15:15:44 INFO mapred.TableInputFormatBase: split: 5->d8g054.nhncorp.com:00000000000,32,19-51978,00000000000,37,13-59758
09/01/13 15:15:44 INFO mapred.TableInputFormatBase: split: 6->d8g054.nhncorp.com:00000000000,37,13-59758,000000000000,1,26-2645
09/01/13 15:15:44 INFO mapred.TableInputFormatBase: split: 7->d8g054.nhncorp.com:000000000000,1,26-2645,000000000000,26,2-41710
09/01/13 15:15:44 INFO mapred.TableInputFormatBase: split: 8->d8g053.nhncorp.com:000000000000,26,2-41710,000000000000,5,31-9247
09/01/13 15:15:44 INFO mapred.TableInputFormatBase: split: 9->d8g055.nhncorp.com:000000000000,5,31-9247,
09/01/13 15:15:44 INFO mapred.JobClient: Running job: job_200901131229_0010
09/01/13 15:15:45 INFO mapred.JobClient: map 0% reduce 0%
09/01/13 15:22:40 INFO mapred.JobClient: map 10% reduce 0%
09/01/13 15:22:53 INFO mapred.JobClient: map 10% reduce 1%
09/01/13 15:22:58 INFO mapred.JobClient: map 10% reduce 2%
09/01/13 15:25:17 INFO mapred.JobClient: map 20% reduce 2%
09/01/13 15:25:25 INFO mapred.JobClient: map 30% reduce 2%
09/01/13 15:25:32 INFO mapred.JobClient: map 30% reduce 3%
09/01/13 15:25:36 INFO mapred.JobClient: map 30% reduce 4%
09/01/13 15:25:40 INFO mapred.JobClient: map 30% reduce 5%
09/01/13 15:25:42 INFO mapred.JobClient: map 30% reduce 6%
09/01/13 15:25:43 INFO mapred.JobClient: map 40% reduce 6%
09/01/13 15:25:55 INFO mapred.JobClient: map 40% reduce 7%
09/01/13 15:29:14 INFO mapred.JobClient: map 50% reduce 7%
09/01/13 15:29:27 INFO mapred.JobClient: map 50% reduce 8%
09/01/13 15:29:29 INFO mapred.JobClient: map 50% reduce 9%
09/01/13 15:35:18 INFO mapred.JobClient: map 60% reduce 9%
09/01/13 15:35:28 INFO mapred.JobClient: map 60% reduce 10%
09/01/13 15:35:36 INFO mapred.JobClient: map 60% reduce 11%
09/01/13 15:36:04 INFO mapred.JobClient: map 70% reduce 11%
09/01/13 15:36:14 INFO mapred.JobClient: map 70% reduce 12%
09/01/13 15:36:19 INFO mapred.JobClient: map 70% reduce 13%
09/01/13 15:38:13 INFO mapred.JobClient: map 80% reduce 13%
09/01/13 15:38:23 INFO mapred.JobClient: map 80% reduce 14%
09/01/13 15:38:29 INFO mapred.JobClient: map 80% reduce 15%
09/01/13 15:42:34 INFO mapred.JobClient: map 90% reduce 15%
09/01/13 15:42:44 INFO mapred.JobClient: map 90% reduce 16%
09/01/13 15:42:49 INFO mapred.JobClient: map 90% reduce 17%
09/01/13 15:45:07 INFO mapred.JobClient: map 100% reduce 17%
09/01/13 15:45:12 INFO mapred.JobClient: map 100% reduce 21%
09/01/13 15:45:16 INFO mapred.JobClient: map 100% reduce 22%
09/01/13 15:45:20 INFO mapred.JobClient: map 100% reduce 25%
09/01/13 15:45:21 INFO mapred.JobClient: map 100% reduce 29%
09/01/13 15:45:26 INFO mapred.JobClient: map 100% reduce 33%
09/01/13 15:45:31 INFO mapred.JobClient: map 100% reduce 34%
09/01/13 15:45:36 INFO mapred.JobClient: map 100% reduce 40%
09/01/13 15:45:41 INFO mapred.JobClient: map 100% reduce 41%
09/01/13 15:45:56 INFO mapred.JobClient: map 100% reduce 42%
09/01/13 15:46:12 INFO mapred.JobClient: map 100% reduce 43%
09/01/13 15:46:31 INFO mapred.JobClient: map 100% reduce 44%
09/01/13 15:46:56 INFO mapred.JobClient: map 100% reduce 45%
09/01/13 15:47:12 INFO mapred.JobClient: map 100% reduce 46%
09/01/13 15:47:32 INFO mapred.JobClient: map 100% reduce 47%
09/01/13 15:47:51 INFO mapred.JobClient: map 100% reduce 48%
09/01/13 15:48:21 INFO mapred.JobClient: map 100% reduce 49%
09/01/13 15:48:45 INFO mapred.JobClient: map 100% reduce 50%
09/01/13 15:49:07 INFO mapred.JobClient: map 100% reduce 51%
09/01/13 15:49:31 INFO mapred.JobClient: map 100% reduce 52%
09/01/13 15:49:52 INFO mapred.JobClient: map 100% reduce 53%
09/01/13 15:50:17 INFO mapred.JobClient: map 100% reduce 54%
09/01/13 15:50:46 INFO mapred.JobClient: map 100% reduce 55%
09/01/13 15:51:12 INFO mapred.JobClient: map 100% reduce 56%
09/01/13 15:51:36 INFO mapred.JobClient: map 100% reduce 57%
09/01/13 15:51:56 INFO mapred.JobClient: map 100% reduce 58%
09/01/13 15:52:12 INFO mapred.JobClient: map 100% reduce 59%
09/01/13 15:52:27 INFO mapred.JobClient: map 100% reduce 60%
09/01/13 15:52:45 INFO mapred.JobClient: map 100% reduce 61%
09/01/13 15:52:56 INFO mapred.JobClient: map 100% reduce 62%
09/01/13 15:53:01 INFO mapred.JobClient: map 100% reduce 63%
09/01/13 15:53:25 INFO mapred.JobClient: map 100% reduce 64%
09/01/13 15:53:27 INFO mapred.JobClient: map 100% reduce 65%
09/01/13 15:53:32 INFO mapred.JobClient: map 100% reduce 66%
09/01/13 15:53:36 INFO mapred.JobClient: map 100% reduce 67%
09/01/13 15:54:17 INFO mapred.JobClient: map 100% reduce 68%
09/01/13 15:54:22 INFO mapred.JobClient: map 100% reduce 69%
09/01/13 15:54:47 INFO mapred.JobClient: map 100% reduce 70%
09/01/13 15:55:29 INFO mapred.JobClient: map 100% reduce 71%
09/01/13 15:55:57 INFO mapred.JobClient: map 100% reduce 75%
09/01/13 15:56:04 INFO mapred.JobClient: map 100% reduce 78%
09/01/13 15:56:06 INFO mapred.JobClient: map 100% reduce 79%
09/01/13 15:56:16 INFO mapred.JobClient: map 100% reduce 83%
09/01/13 15:56:21 INFO mapred.JobClient: map 100% reduce 86%
09/01/13 15:56:44 INFO mapred.JobClient: map 100% reduce 87%
09/01/13 15:57:12 INFO mapred.JobClient: map 100% reduce 88%
09/01/13 15:57:52 INFO mapred.JobClient: map 100% reduce 89%
09/01/13 15:58:39 INFO mapred.JobClient: map 100% reduce 90%
09/01/13 15:59:25 INFO mapred.JobClient: map 100% reduce 91%
09/01/13 16:00:02 INFO mapred.JobClient: map 100% reduce 92%
09/01/13 16:00:52 INFO mapred.JobClient: map 100% reduce 93%
09/01/13 16:01:47 INFO mapred.JobClient: map 100% reduce 94%
09/01/13 16:02:43 INFO mapred.JobClient: map 100% reduce 95%
09/01/13 16:03:23 INFO mapred.JobClient: map 100% reduce 96%
09/01/13 16:04:08 INFO mapred.JobClient: map 100% reduce 97%
09/01/13 16:04:38 INFO mapred.JobClient: map 100% reduce 98%
09/01/13 16:05:12 INFO mapred.JobClient: map 100% reduce 99%
09/01/13 16:06:03 INFO mapred.JobClient: map 100% reduce 100%
09/01/13 16:06:17 INFO mapred.JobClient: Job complete: job_200901131229_0010
09/01/13 16:06:17 INFO mapred.JobClient: Counters: 15
09/01/13 16:06:17 INFO mapred.JobClient: File Systems
09/01/13 16:06:17 INFO mapred.JobClient: Local bytes read=56408501475
09/01/13 16:06:17 INFO mapred.JobClient: Local bytes written=76882028610
09/01/13 16:06:17 INFO mapred.JobClient: Job Counters
09/01/13 16:06:17 INFO mapred.JobClient: Launched reduce tasks=12
09/01/13 16:06:17 INFO mapred.JobClient: Rack-local map tasks=3
09/01/13 16:06:17 INFO mapred.JobClient: Launched map tasks=13
09/01/13 16:06:17 INFO mapred.JobClient: Data-local map tasks=10
09/01/13 16:06:17 INFO mapred.JobClient: Map-Reduce Framework
09/01/13 16:06:17 INFO mapred.JobClient: Reduce input groups=1600
09/01/13 16:06:17 INFO mapred.JobClient: Combine output records=0
09/01/13 16:06:17 INFO mapred.JobClient: Map input records=64000
09/01/13 16:06:17 INFO mapred.JobClient: Reduce output records=320000
09/01/13 16:06:17 INFO mapred.JobClient: Map output bytes=20481920000
09/01/13 16:06:17 INFO mapred.JobClient: Map input bytes=0
09/01/13 16:06:17 INFO mapred.JobClient: Combine input records=0
09/01/13 16:06:17 INFO mapred.JobClient: Map output records=64000
09/01/13 16:06:17 INFO mapred.JobClient: Reduce input records=64000
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663565#action_12663565 ]
Edward J. Yoon commented on HAMA-129:
-------------------------------------
More large:
----
09/01/13 20:49:23 INFO mapred.JobClient: map 100% reduce 40%
09/01/13 20:49:23 INFO mapred.JobClient: Task Id : attempt_200901131739_0005_r_000017_2, Status : FAILED
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 61.247.201.165:60020 for region DenseMatrix_randwkzjh,,1231846579535, row '000000000000000', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: Could not read from stream
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
java.io.IOException: java.io.IOException: Could not read from stream
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
java.io.IOException: java.io.IOException: Could not read from stream
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
java.io.IOException: java.io.IOException: Could not read from stream
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
java.io.IOException: java.io.IOException: Could not read from stream
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
java.io.IOException: java.io.IOException: Could not read from stream
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
java.io.IOException: java.io.IOException: Could not read from stream
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
java.io.IOException: java.io.IOException: Could not read from stream
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
java.io.IOException: java.io.IOException: Could not read from stream
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
java.io.IOException: java.io.IOException: Could not read from stream
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
at org.apache.hadoop.io.Text.readString(Text.java:400)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:843)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:933)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1372)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1316)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1296)
at org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:71)
at org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:51)
at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:405)
at org.apache.hama.algebra.BlockCyclicMultiplyReduce.reduce(BlockCyclicMultiplyReduce.java:83)
at org.apache.hama.algebra.BlockCyclicMultiplyReduce.reduce(BlockCyclicMultiplyReduce.java:39)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
09/01/13 20:49:29 INFO mapred.JobClient: map 100% reduce 38%
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1217)
at org.apache.hama.util.JobManager.execute(JobManager.java:33)
at org.apache.hama.DenseMatrix.mult(DenseMatrix.java:421)
at org.apache.hama.examples.MatrixMultiplication.main(MatrixMultiplication.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
at org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Status: Patch Available (was: In Progress)
This is better than old approach.
Submit patch.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_newApproach_v01.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HAMA-129) Improving speed of matrix multiplication
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-129:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_newApproach.patch, HAMA-129_newApproach_v01.patch, HAMA-129_v01.patch, HAMA-129_v02.patch, HAMA-129_v03.patch, HAMA-129_v04.patch, HAMA-129_v05.patch, HAMA-129_v06.patch, patch01.txt, patch02.txt
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HAMA-129) Improving speed of matrix
multiplication
Posted by "Samuel Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656935#action_12656935 ]
Samuel Guo commented on HAMA-129:
---------------------------------
how large the matrix is ?
what is the size of the blocks?
> Improving speed of matrix multiplication
> ----------------------------------------
>
> Key: HAMA-129
> URL: https://issues.apache.org/jira/browse/HAMA-129
> Project: Hama
> Issue Type: Improvement
> Components: implementation
> Affects Versions: 0.1.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-129.patch, HAMA-129_v01.patch, HAMA-129_v02.patch
>
>
> We use a blockinputformat (row based partitioning). but it causes "java.lang.OutOfMemoryError: Java heap space" when size is big.
> I'll write the formulas and partition them as PIEstimator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.