[jira] [Comment Edited] (TEZ-2556) Document how to profile Tez DAGs

["Jonathan Eagles (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588235#comment-14588235 ] 

Jonathan Eagles edited comment on TEZ-2556 at 6/16/15 3:36 PM:
---------------------------------------------------------------

[~rajesh.balamohan], I'm quite interested in how to do this since I've always been disappointed in my results. For profiling the AM, I have placed my yourkit command below. Ideally, I would love to be able see instructions that cover a variety of profile tools including the use cases above. In addition, there may be changes or at least instructions so that I don't have to respecify the full cmd opts would be good as well. For example, the below command outputs the profile to the log directory, but only in 2.8 can you easily download a specific log file on the command line if you have log aggregation enabled.

{code}
HADOOP_CLASSPATH="$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_CONF_DIR" yarn jar $TEZ_HOME/tez-tests-*.jar mrrsleep -Dtez.am.launch.cmd-opts='-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -agentpath:libyjpagent.x86_32.so=tracing,monitors,alloceach=1,dir=<LOG_DIR>,logdir=<LOG_DIR>' -Dtez.aux.uris='/user/jeagles/yourkit/libyjpagent.x86_32.so' -m 1 -mt 0 -ir 0 -irt 0 -irs 0 -r 0 -rt 0
{code}


was (Author: jeagles):
[~rajesh.balamohan], I'm quite interested in how to do this since I've always been disappointed in my results. For profiling the AM, I have placed my yourkit command below. Ideally, I would love to be able see instructions that cover a variety of profile tools including the use cases above. In addition, there may be changes or at least instructions so that I don't have to respecify the full cmd opts would be good as well. For example, the below command outputs the profile to the log directory, but only in 2.8 can you easily download a specific log file on the command line if you have log aggregation enabled.

{quote}
HADOOP_CLASSPATH="$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_CONF_DIR" yarn jar $TEZ_HOME/tez-tests-*.jar mrrsleep -Dtez.am.launch.cmd-opts='-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -agentpath:libyjpagent.x86_32.so=tracing,monitors,alloceach=1,dir=<LOG_DIR>,logdir=<LOG_DIR>' -Dtez.aux.uris='/user/jeagles/yourkit/libyjpagent.x86_32.so' -m 1 -mt 0 -ir 0 -irt 0 -irs 0 -r 0 -rt 0
{quote}

> Document how to profile Tez DAGs 
> ---------------------------------
>
>                 Key: TEZ-2556
>                 URL: https://issues.apache.org/jira/browse/TEZ-2556
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Hitesh Shah
>            Assignee: Rajesh Balamohan
>
> Should cover basic details on profiling in addition to:
>    - how to profile a particular vertex
>    - how to profile a particular task or range of tasks
>    



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)