You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Bikas Saha (JIRA)" <ji...@apache.org> on 2015/05/22 09:09:17 UTC
[jira] [Comment Edited] (TEZ-2478) Move OneToOne routing to store
events in Tasks
[ https://issues.apache.org/jira/browse/TEZ-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555710#comment-14555710 ]
Bikas Saha edited comment on TEZ-2478 at 5/22/15 7:08 AM:
----------------------------------------------------------
Not sure why this is so urgently needed as to target 0.7.1.
Attached a wip patch I was using on top of TEZ-2409 to compare legacy and on-demand code on a 20 node cluster. The patch allows old and new routing for 1-1 based on a config variable. With a 10Kx10K 1-1 edge empty job. There is no discernible difference in the runtimes ~30s and CPU time ~107-110s in both cases. Given that this is already a fairly large job IMO we can wait to see if there is any real issue in practice.
The pro of keeping on-demand routing is that it moves edge routing off the central dispatcher.
was (Author: bikassaha):
Not sure why this is so urgently needed as to target 0.7.1.
Attached a wip patch that I was using on top of TEZ-2409 to compare legacy and on-demand code on a 20 node cluster. With a 10Kx10K 1-1 edge empty job. There is no discernible difference in the runtimes ~30s and CPU time ~107-110s in both cases. Given that this is already a fairly large job IMO we can wait to see if there is any real issue in practice.
The pro of keeping on-demand routing is that it moves edge routing off the central dispatcher.
> Move OneToOne routing to store events in Tasks
> ----------------------------------------------
>
> Key: TEZ-2478
> URL: https://issues.apache.org/jira/browse/TEZ-2478
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: 1-1-wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)