You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-user@hadoop.apache.org by Jakub Stransky <st...@gmail.com> on 2014/09/23 12:56:50 UTC

Cannot fine profiling log file

Hello experienced users,

I did try to use profiling of tasks during mapreduce
    <property>
        <name>mapreduce.task.profile</name>
        <value>true</value>
    </property>
    <property>
        <name>mapreduce.task.profile.maps</name>
        <value>0-5</value>
    </property>
    <property>
        <name>mapreduce.task.profile.params</name>

<value>-agentlib:hprof=cpu=samples,heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s</value>
    </property>

The file got generated I see that thrugh Resource Manager console but I
can't find it from where to download.

Where to find that file or how to download it?

Thanks for any advices!

Jakub

RE: Cannot fine profiling log file

Posted by Rohith Sharma K S <ro...@huawei.com>.

HI


Have you enable log aggregation..?


1.       If log aggregation is enabled then you can get logs from hdfs below path. Both aggregated logs and profiler will be in same file.
     ${yarn.nodemanager.remote-app-log-dir}/${user}/logs/<app-id>/


If not enabled, then check inside
    ${yarn.nodemanager.log-dirs}/<app-id>/<container-id>/profile.out(default name)


Thanks & Regards
Rohith Sharma K S

From: Jakub Stransky [mailto:stransky.ja@gmail.com]
Sent: 23 September 2014 16:27
To: user@hadoop.apache.org
Subject: Cannot fine profiling log file

Hello experienced users,

I did try to use profiling of tasks during mapreduce
    <property>
        <name>mapreduce.task.profile</name>
        <value>true</value>
    </property>
    <property>
        <name>mapreduce.task.profile.maps</name>
        <value>0-5</value>
    </property>
    <property>
        <name>mapreduce.task.profile.params</name>
        <value>-agentlib:hprof=cpu=samples,heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s</value>
    </property>

The file got generated I see that thrugh Resource Manager console but I can't find it from where to download.

Where to find that file or how to download it?

Thanks for any advices!

Jakub

RE: Cannot fine profiling log file

Posted by Rohith Sharma K S <ro...@huawei.com>.

HI


Have you enable log aggregation..?


1.       If log aggregation is enabled then you can get logs from hdfs below path. Both aggregated logs and profiler will be in same file.
     ${yarn.nodemanager.remote-app-log-dir}/${user}/logs/<app-id>/


If not enabled, then check inside
    ${yarn.nodemanager.log-dirs}/<app-id>/<container-id>/profile.out(default name)


Thanks & Regards
Rohith Sharma K S

From: Jakub Stransky [mailto:stransky.ja@gmail.com]
Sent: 23 September 2014 16:27
To: user@hadoop.apache.org
Subject: Cannot fine profiling log file

Hello experienced users,

I did try to use profiling of tasks during mapreduce
    <property>
        <name>mapreduce.task.profile</name>
        <value>true</value>
    </property>
    <property>
        <name>mapreduce.task.profile.maps</name>
        <value>0-5</value>
    </property>
    <property>
        <name>mapreduce.task.profile.params</name>
        <value>-agentlib:hprof=cpu=samples,heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s</value>
    </property>

The file got generated I see that thrugh Resource Manager console but I can't find it from where to download.

Where to find that file or how to download it?

Thanks for any advices!

Jakub

RE: Cannot fine profiling log file

Posted by Rohith Sharma K S <ro...@huawei.com>.

HI


Have you enable log aggregation..?


1.       If log aggregation is enabled then you can get logs from hdfs below path. Both aggregated logs and profiler will be in same file.
     ${yarn.nodemanager.remote-app-log-dir}/${user}/logs/<app-id>/


If not enabled, then check inside
    ${yarn.nodemanager.log-dirs}/<app-id>/<container-id>/profile.out(default name)


Thanks & Regards
Rohith Sharma K S

From: Jakub Stransky [mailto:stransky.ja@gmail.com]
Sent: 23 September 2014 16:27
To: user@hadoop.apache.org
Subject: Cannot fine profiling log file

Hello experienced users,

I did try to use profiling of tasks during mapreduce
    <property>
        <name>mapreduce.task.profile</name>
        <value>true</value>
    </property>
    <property>
        <name>mapreduce.task.profile.maps</name>
        <value>0-5</value>
    </property>
    <property>
        <name>mapreduce.task.profile.params</name>
        <value>-agentlib:hprof=cpu=samples,heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s</value>
    </property>

The file got generated I see that thrugh Resource Manager console but I can't find it from where to download.

Where to find that file or how to download it?

Thanks for any advices!

Jakub

RE: Cannot fine profiling log file

Posted by Rohith Sharma K S <ro...@huawei.com>.

HI


Have you enable log aggregation..?


1.       If log aggregation is enabled then you can get logs from hdfs below path. Both aggregated logs and profiler will be in same file.
     ${yarn.nodemanager.remote-app-log-dir}/${user}/logs/<app-id>/


If not enabled, then check inside
    ${yarn.nodemanager.log-dirs}/<app-id>/<container-id>/profile.out(default name)


Thanks & Regards
Rohith Sharma K S

From: Jakub Stransky [mailto:stransky.ja@gmail.com]
Sent: 23 September 2014 16:27
To: user@hadoop.apache.org
Subject: Cannot fine profiling log file

Hello experienced users,

I did try to use profiling of tasks during mapreduce
    <property>
        <name>mapreduce.task.profile</name>
        <value>true</value>
    </property>
    <property>
        <name>mapreduce.task.profile.maps</name>
        <value>0-5</value>
    </property>
    <property>
        <name>mapreduce.task.profile.params</name>
        <value>-agentlib:hprof=cpu=samples,heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s</value>
    </property>

The file got generated I see that thrugh Resource Manager console but I can't find it from where to download.

Where to find that file or how to download it?

Thanks for any advices!

Jakub