You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Jothi Padmanabhan (JIRA)" <ji...@apache.org> on 2008/09/08 15:18:47 UTC
[jira] Issue Comment Edited: (HADOOP-3514) Reduce seeks during
shuffle, by inline crcs
[ https://issues.apache.org/jira/browse/HADOOP-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629136#action_12629136 ]
jothipn edited comment on HADOOP-3514 at 9/8/08 6:17 AM:
-------------------------------------------------------------------
The above patch fixes the problem observed when running with the native lzo library for map output compression.
The problem was with the IFileInputStream read method which is required to return the byte read as an integer. A simple assignment of
{code}
int result = byte
{code}
does not work as the byte is interpreted as a signed byte and so the assigned integer has a wrong value. Instead the result has to be assigned as
{code}
int result = (byte & 0xFF)
{code}
correctly assigns the byte to the integer
The following are the performance improvements observed with this patch when running the loadgen program with
60 reduce copiers, 100 http threads, 450 task trackers, with the following command line (Thanks Runping for the identifying the problem configuration that best illustrates the utility of this patch)
{noformat}
bin/hadoop jar hadoop-0.19.0-dev-test.jar loadgen \
-D test.randomtextwrite.bytes_per_map=$((240*1024)) \
-D test.randomtextwrite.total_bytes=$((200*1024*100000)) \
-D mapred.compress.map.output=false \
-r 2200 \
-outKey org.apache.hadoop.io.Text \
-outValue org.apache.hadoop.io.Text \
-outFormat org.apache.hadoop.mapred.lib.NullOutputFormat \
-outdir fakeout
{noformat}
The patch showed an overall improvement of about 5% with about 10% improvement in shuffle.
Trunk Patch
Map Time 6:10 6:04
Reduce Time 17:11 16:22
Overall 23:21 22:26
was (Author: jothipn):
The above patch fixes the problem observed when running with the native lzo library for map output compression.
The problem was with the IFileInputStream read method which is required to return the byte read as an integer. A simple assignment of
{code}
int result = byte
{code}
does not work as the byte is interpreted as a signed byte and so the assigned integer has a wrong value. Instead the result has to be assigned as
{code}
int result = (byte & 0xFF)
{code}
correctly assigns the byte to the integer
The following are the performance improvements observed with this patch when running the loadgen program with
60 reduce copiers, 100 http threads, 450 task trackers, with the following command line
<pre>
bin/hadoop jar hadoop-0.19.0-dev-test.jar loadgen \
-D test.randomtextwrite.bytes_per_map=$((240*1024)) \
-D test.randomtextwrite.total_bytes=$((200*1024*100000)) \
-D mapred.compress.map.output=false \
-r 2200 \
-outKey org.apache.hadoop.io.Text \
-outValue org.apache.hadoop.io.Text \
-outFormat org.apache.hadoop.mapred.lib.NullOutputFormat \
-outdir fakeout
</pre>
The patch showed an overall improvement of about 5% with about 10% improvement in shuffle.
Trunk Patch
Map Time 6:10 6:04
Reduce Time 17:11 16:22
Overall 23:21 22:26
> Reduce seeks during shuffle, by inline crcs
> -------------------------------------------
>
> Key: HADOOP-3514
> URL: https://issues.apache.org/jira/browse/HADOOP-3514
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Devaraj Das
> Assignee: Jothi Padmanabhan
> Fix For: 0.19.0
>
> Attachments: hadoop-3514-v1.patch, hadoop-3514-v10.patch, hadoop-3514-v11.patch, hadoop-3514-v12.patch, hadoop-3514-v2.patch, hadoop-3514-v3.patch, hadoop-3514-v4.patch, hadoop-3514-v5.patch, hadoop-3514-v6.patch, hadoop-3514-v7.patch, hadoop-3514-v8.patch, hadoop-3514-v9.patch, hadoop-3514.patch
>
>
> The number of seeks can be reduced by half in the iFile if we move the crc into the iFile rather than having a separate file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.