You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Jan Morlock <ja...@googlemail.com> on 2016/10/11 23:05:12 UTC
Origin of failed tasks
Hi,
currently failed tasks occur during the execution of my Hive/Tez job.
However in the end, the overall job succeeds. Is it possible to find out
afterwards about the origin of those failed tasks (without using the Tez
UI) just by analyzing the output log files?
Best regards
Jan
RE: Origin of failed tasks
Posted by HuXi <hu...@hotmail.com>.
> Subject: Re: Origin of failed tasks
> From: hitesh@apache.org
> Date: Wed, 12 Oct 2016 08:57:12 -0700
> To: user@tez.apache.org
>
> If you have the logs for the application master, you can try the following:
>
> grep “[HISTORY]” | grep “TASK_ATTEMPT_FINISHED”
>
> This will give you info on any failed task attempts.
>
> The AM logs have history events being published to them. You can do grep “[HISTORY]” | grep “<entity type>_<event type>” where entity type is one of DAG, VERTEX, TASK, TASK_ATTEMPT and event type is STARTED or FINISHED.
>
> The logs are also split into diff files. e.g.
> The AM logs use a syslog_dag… format to split across dags.
> Task/Container logs use syslog_attempt* format to split out logs for different task attempts.
>
> If you have YARN timeline enabled, you can use the analyzers to do more analysis on the dag specific data. These are more related to perf tuning and not failure diagnostics though.
>
> thanks
> ― Hitesh
>
>
> > On Oct 11, 2016, at 5:09 PM, Allan Wilson <wi...@gmail.com> wrote:
> >
> > Use the yarn logs command. That's your only chance without the TEZ UI. I setup the TEZ UI
> > In our shop and it is really nice.
> >
> > Allan
> > Sent from my iPhone
> >
> >> On Oct 11, 2016, at 5:05 PM, Jan Morlock <ja...@googlemail.com> wrote:
> >>
> >> Hi,
> >>
> >> currently failed tasks occur during the execution of my Hive/Tez job.
> >> However in the end, the overall job succeeds. Is it possible to find out
> >> afterwards about the origin of those failed tasks (without using the Tez
> >> UI) just by analyzing the output log files?
> >>
> >> Best regards
> >> Jan
>
Re: Origin of failed tasks
Posted by Hitesh Shah <hi...@apache.org>.
If you have the logs for the application master, you can try the following:
grep “[HISTORY]” | grep “TASK_ATTEMPT_FINISHED”
This will give you info on any failed task attempts.
The AM logs have history events being published to them. You can do grep “[HISTORY]” | grep “<entity type>_<event type>” where entity type is one of DAG, VERTEX, TASK, TASK_ATTEMPT and event type is STARTED or FINISHED.
The logs are also split into diff files. e.g.
The AM logs use a syslog_dag… format to split across dags.
Task/Container logs use syslog_attempt* format to split out logs for different task attempts.
If you have YARN timeline enabled, you can use the analyzers to do more analysis on the dag specific data. These are more related to perf tuning and not failure diagnostics though.
thanks
— Hitesh
> On Oct 11, 2016, at 5:09 PM, Allan Wilson <wi...@gmail.com> wrote:
>
> Use the yarn logs command. That's your only chance without the TEZ UI. I setup the TEZ UI
> In our shop and it is really nice.
>
> Allan
> Sent from my iPhone
>
>> On Oct 11, 2016, at 5:05 PM, Jan Morlock <ja...@googlemail.com> wrote:
>>
>> Hi,
>>
>> currently failed tasks occur during the execution of my Hive/Tez job.
>> However in the end, the overall job succeeds. Is it possible to find out
>> afterwards about the origin of those failed tasks (without using the Tez
>> UI) just by analyzing the output log files?
>>
>> Best regards
>> Jan
Re: Origin of failed tasks
Posted by Allan Wilson <wi...@gmail.com>.
Use the yarn logs command. That's your only chance without the TEZ UI. I setup the TEZ UI
In our shop and it is really nice.
Allan
Sent from my iPhone
> On Oct 11, 2016, at 5:05 PM, Jan Morlock <ja...@googlemail.com> wrote:
>
> Hi,
>
> currently failed tasks occur during the execution of my Hive/Tez job.
> However in the end, the overall job succeeds. Is it possible to find out
> afterwards about the origin of those failed tasks (without using the Tez
> UI) just by analyzing the output log files?
>
> Best regards
> Jan