You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by yogendra reddy <yo...@gmail.com> on 2015/04/16 06:24:21 UTC

Tez job configuration file

mapreduce stores job-conf.xml for each job run in
mapreduce.jobhistory.done-dir. Is there a similar file for tez job. If yes,
where is this stored?

Thanks,
Yogendra

Re: Tez job configuration file

Posted by Gopal Vijayaraghavan <go...@apache.org>.
Hi,

> This is for telemetry use case wherein I would like to categorize if a
>hive job is run on mr or tez.  I was actually looking for
>hive.execution.engine property in determining this


You are looking in the wrong place (and wrong list), if you want to look
at Hive query plans, the ATS data is written directly from Hive¹s
ATSHook.java.

The information in that is in enough detail to not just indicate MR or
Tez, but it has deeper information about all tables scanned and all
columns read from each table.

For instance, if I look up one of my queries via a query-id

http://rmhost:8188/ws/v1/timeline/HIVE_QUERY_ID/gopal_20150318071409_1e0200
64-62b1-4ea6-8d57-fde4cddef57f

or lookup the latest one

http://rmhost:8188/ws/v1/timeline/HIVE_QUERY_ID/?limit=1

It should contain the following in JSON.

"TEZ":true,"STATUS":true,"MAPRED":false}

If you want more info on ATSHook, mail the hive user lists.

Cheers,
Gopal



Re: Tez job configuration file

Posted by yogendra reddy <yo...@gmail.com>.
looks like this feature is not there yet. I found the ASF JIRA
https://issues.apache.org/jira/browse/TEZ-986

On Thu, Apr 16, 2015 at 11:30 AM, yogendra reddy <yo...@gmail.com>
wrote:

> Thanks for the info.
> This is for telemetry use case wherein I would like to categorize if a
> hive job is run on mr or tez.  I was actually looking for
> hive.execution.engine property in determining this and I could do this for
> mr since it emits configuration file per job but looks like this is not the
> case in tez
> can you let me know the parameter which controls the physical storage of
> dag plan?
>
> On Thu, Apr 16, 2015 at 11:03 AM, Jianfeng (Jeff) Zhang <
> jzhang@hortonworks.com> wrote:
>
>>
>>  Hi Yogendra,
>>
>>  There’s no job-conf.xml in tez. In tez the configuration is associated
>> with Input/Output/Processor, You can get these configuration through ATS
>>
>>  Please check the dag info by looking at
>> http://localhost:8188/ws/v1/timeline/TEZ_DAG_ID/
>> <http://localhost:8188/ws/v1/timeline/TEZ_DAG_ID/dag_1428458161048_0057_1>
>> [YOUR_DAG_ID]
>> It has the dagPlan info where you can see userPayloadAsText which
>> contains the configuration.
>>
>>  BTW, what’s your purpose for checking the configuration ? For
>> performance analysis ?
>>
>>
>>  Best Regard,
>> Jeff Zhang
>>
>>
>>   From: yogendra reddy <yo...@gmail.com>
>> Reply-To: "user@tez.apache.org" <us...@tez.apache.org>
>> Date: Thursday, April 16, 2015 at 12:24 PM
>> To: "user@tez.apache.org" <us...@tez.apache.org>
>> Subject: Tez job configuration file
>>
>>   mapreduce stores job-conf.xml for each job run in
>> mapreduce.jobhistory.done-dir. Is there a similar file for tez job. If yes,
>> where is this stored?
>>
>>  Thanks,
>> Yogendra
>>
>
>

Re: Tez job configuration file

Posted by yogendra reddy <yo...@gmail.com>.
Thanks for the info.
This is for telemetry use case wherein I would like to categorize if a hive
job is run on mr or tez.  I was actually looking for hive.execution.engine
property in determining this and I could do this for mr since it emits
configuration file per job but looks like this is not the case in tez
can you let me know the parameter which controls the physical storage of
dag plan?

On Thu, Apr 16, 2015 at 11:03 AM, Jianfeng (Jeff) Zhang <
jzhang@hortonworks.com> wrote:

>
>  Hi Yogendra,
>
>  There’s no job-conf.xml in tez. In tez the configuration is associated
> with Input/Output/Processor, You can get these configuration through ATS
>
>  Please check the dag info by looking at
> http://localhost:8188/ws/v1/timeline/TEZ_DAG_ID/
> <http://localhost:8188/ws/v1/timeline/TEZ_DAG_ID/dag_1428458161048_0057_1>
> [YOUR_DAG_ID]
> It has the dagPlan info where you can see userPayloadAsText which
> contains the configuration.
>
>  BTW, what’s your purpose for checking the configuration ? For
> performance analysis ?
>
>
>  Best Regard,
> Jeff Zhang
>
>
>   From: yogendra reddy <yo...@gmail.com>
> Reply-To: "user@tez.apache.org" <us...@tez.apache.org>
> Date: Thursday, April 16, 2015 at 12:24 PM
> To: "user@tez.apache.org" <us...@tez.apache.org>
> Subject: Tez job configuration file
>
>   mapreduce stores job-conf.xml for each job run in
> mapreduce.jobhistory.done-dir. Is there a similar file for tez job. If yes,
> where is this stored?
>
>  Thanks,
> Yogendra
>

Re: Tez job configuration file

Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
Hi Yogendra,

There's no job-conf.xml in tez. In tez the configuration is associated with Input/Output/Processor, You can get these configuration through ATS

Please check the dag info by looking at http://localhost:8188/ws/v1/timeline/TEZ_DAG_ID/<http://localhost:8188/ws/v1/timeline/TEZ_DAG_ID/dag_1428458161048_0057_1>[YOUR_DAG_ID]
It has the dagPlan info where you can see userPayloadAsText which contains the configuration.

BTW, what's your purpose for checking the configuration ? For performance analysis ?


Best Regard,
Jeff Zhang


From: yogendra reddy <yo...@gmail.com>>
Reply-To: "user@tez.apache.org<ma...@tez.apache.org>" <us...@tez.apache.org>>
Date: Thursday, April 16, 2015 at 12:24 PM
To: "user@tez.apache.org<ma...@tez.apache.org>" <us...@tez.apache.org>>
Subject: Tez job configuration file

mapreduce stores job-conf.xml for each job run in mapreduce.jobhistory.done-dir. Is there a similar file for tez job. If yes, where is this stored?

Thanks,
Yogendra