You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Xiaoyong Zhu <xi...@microsoft.com> on 2015/05/22 01:14:58 UTC

Tez log location?

Hi, I am wondering if I didn't configure YARN ATS integration, where would be the tez log (I mean the data available in /ws/v1/timeline/TEZ_DAG_ID if there is an integration happening) go to HDFS? Is there a configuration for that?

Thanks!

Xiaoyong

Re: Tez log location?

Posted by Hitesh Shah <hi...@apache.org>.
There is some history logging done that can be enabled via the SimpleHistoryLogger. This activates by default if ATS logger is not enabled. This is not fully compatible with the ATS data and also as it is mostly experimental, it may not have all the data. To use it, you can configure the “tez.history.logging.service.class” to “” or “org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService”.  The config property “tez.simple.history.logging.dir” controls the path on HDFS where the history is written to. If the dir path is not configured, it writes the logs as part of the Application Master container logs which can then be pulled via “bin/yarn logs -application” 

Using the HDFS logger does imply that the UI will no longer be functional. 

thanks
— Hitesh





On May 21, 2015, at 4:14 PM, Xiaoyong Zhu <xi...@microsoft.com> wrote:

> Hi, I am wondering if I didn’t configure YARN ATS integration, where would be the tez log (I mean the data available in /ws/v1/timeline/TEZ_DAG_ID if there is an integration happening) go to HDFS? Is there a configuration for that?
>  
> Thanks!
>  
> Xiaoyong

RE: Tez log location?

Posted by Bikas Saha <bi...@hortonworks.com>.
Perhaps we should consider creating a TEE in the AM and always do SimpleHistoryLogging?

From: Jianfeng (Jeff) Zhang [mailto:jzhang@hortonworks.com]
Sent: Thursday, May 21, 2015 5:33 PM
To: user@tez.apache.org
Subject: Re: Tez log location?

In that case you are using SimpleHistoryLoggingService rather than ATSHistoryLoggingService.

SimpleHistoryLoggingService will log all the messages to the container logs. So if you will find something like the following which is the data same as in /ws/v1/timeline/TEZ_DAG_ID.  But these data are only for diagnosis, if you are trying to consume these data for offline analysis, I would encourage you to use data from ATS


2015-05-21 18:52:06,245 INFO [Dispatcher thread: Central] history.HistoryEventHandler: [HISTORY][DAG:dag_1432205503669_0001_1][Event:DAG_FINISHED]: dagId=dag_1432205503669_0001_1, startTime=1432205516089, finishTime=1432205526204, timeTaken=10115, status=SUCCEEDED, diagnostics=, counters=Counters: 14, org.apache.tez.common.counters.DAGCounter, NUM_KILLED_TASKS=1, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=3, AM_CPU_MILLISECONDS=0, AM_GC_TIME_MILLIS=0, File System Counters, HDFS_BYTES_READ=0, HDFS_BYTES_WRITTEN=24, HDFS_READ_OPS=6, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=4, org.apache.tez.common.counters.TaskCounter, NUM_SPECULATIONS=1, GC_TIME_MILLIS=21, COMMITTED_HEAP_BYTES=514850816, OUTPUT_RECORDS=2


Best Regard,
Jeff Zhang


From: Xiaoyong Zhu <xi...@microsoft.com>>
Reply-To: "user@tez.apache.org<ma...@tez.apache.org>" <us...@tez.apache.org>>
Date: Friday, May 22, 2015 at 7:14 AM
To: "user@tez.apache.org<ma...@tez.apache.org>" <us...@tez.apache.org>>
Subject: Tez log location?

Hi, I am wondering if I didn't configure YARN ATS integration, where would be the tez log (I mean the data available in /ws/v1/timeline/TEZ_DAG_ID if there is an integration happening) go to HDFS? Is there a configuration for that?

Thanks!

Xiaoyong

Re: Tez log location?

Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
In that case you are using SimpleHistoryLoggingService rather than ATSHistoryLoggingService.

SimpleHistoryLoggingService will log all the messages to the container logs. So if you will find something like the following which is the data same as in /ws/v1/timeline/TEZ_DAG_ID.  But these data are only for diagnosis, if you are trying to consume these data for offline analysis, I would encourage you to use data from ATS


2015-05-21 18:52:06,245 INFO [Dispatcher thread: Central] history.HistoryEventHandler: [HISTORY][DAG:dag_1432205503669_0001_1][Event:DAG_FINISHED]: dagId=dag_1432205503669_0001_1, startTime=1432205516089, finishTime=1432205526204, timeTaken=10115, status=SUCCEEDED, diagnostics=, counters=Counters: 14, org.apache.tez.common.counters.DAGCounter, NUM_KILLED_TASKS=1, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=3, AM_CPU_MILLISECONDS=0, AM_GC_TIME_MILLIS=0, File System Counters, HDFS_BYTES_READ=0, HDFS_BYTES_WRITTEN=24, HDFS_READ_OPS=6, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=4, org.apache.tez.common.counters.TaskCounter, NUM_SPECULATIONS=1, GC_TIME_MILLIS=21, COMMITTED_HEAP_BYTES=514850816, OUTPUT_RECORDS=2


Best Regard,
Jeff Zhang


From: Xiaoyong Zhu <xi...@microsoft.com>>
Reply-To: "user@tez.apache.org<ma...@tez.apache.org>" <us...@tez.apache.org>>
Date: Friday, May 22, 2015 at 7:14 AM
To: "user@tez.apache.org<ma...@tez.apache.org>" <us...@tez.apache.org>>
Subject: Tez log location?

Hi, I am wondering if I didn't configure YARN ATS integration, where would be the tez log (I mean the data available in /ws/v1/timeline/TEZ_DAG_ID if there is an integration happening) go to HDFS? Is there a configuration for that?

Thanks!

Xiaoyong