You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Rajesh Balamohan (JIRA)" <ji...@apache.org> on 2015/05/21 09:25:00 UTC
[jira] [Updated] (TEZ-2473) Consider using RawLocalFileSystem in
MapOutput.createDiskMapOutput
[ https://issues.apache.org/jira/browse/TEZ-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajesh Balamohan updated TEZ-2473:
----------------------------------
Attachment: TEZ-2473.1.patch
{noformat}
With out patch:
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000191_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=225520
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000192_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=198550
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000193_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=207220
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000194_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=216870
With patch:
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000191_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=199620
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000192_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=203500
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000193_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=203730
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000194_0, NUM_MEM_TO_DISK_MERGES count: 2: CPU=208410
{noformat}
> Consider using RawLocalFileSystem in MapOutput.createDiskMapOutput
> ------------------------------------------------------------------
>
> Key: TEZ-2473
> URL: https://issues.apache.org/jira/browse/TEZ-2473
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Attachments: TEZ-2473.1.patch
>
>
> Currently it makes use of LocalFileSystem which would go through checksumming. This can save some CPU cycles in tasks involving disk merges.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)