You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Rajesh Balamohan (JIRA)" <ji...@apache.org> on 2015/05/21 09:25:59 UTC
[jira] [Comment Edited] (TEZ-2473) Consider using
RawLocalFileSystem in MapOutput.createDiskMapOutput
[ https://issues.apache.org/jira/browse/TEZ-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553779#comment-14553779 ]
Rajesh Balamohan edited comment on TEZ-2473 at 5/21/15 7:25 AM:
----------------------------------------------------------------
{noformat}
With out patch:
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000191_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=225520
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000192_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=198550
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000193_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=207220
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000194_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=216870
With patch:
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000191_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=199620
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000192_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=203500
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000193_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=203730
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000194_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=208410
{noformat}
[~sseth] - Please review when you find time.
was (Author: rajesh.balamohan):
{noformat}
With out patch:
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000191_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=225520
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000192_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=198550
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000193_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=207220
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000194_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=216870
With patch:
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000191_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=199620
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000192_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=203500
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000193_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=203730
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000194_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=208410
{noformat}
> Consider using RawLocalFileSystem in MapOutput.createDiskMapOutput
> ------------------------------------------------------------------
>
> Key: TEZ-2473
> URL: https://issues.apache.org/jira/browse/TEZ-2473
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Attachments: TEZ-2473.1.patch
>
>
> Currently it makes use of LocalFileSystem which would go through checksumming. This can save some CPU cycles in tasks involving disk merges.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)