You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by Tsuyoshi OZAWA <oz...@gmail.com> on 2014/08/18 10:26:01 UTC
OrderedWordCount slow down or hung-up for a time
Hi,
I'm trying some jobs in examples of tez. I found that sometimes jobs
get hung-up on my distributed environment.
Because of this behavior, the jobs on tez get slow down and sometimes
get slower than original MapReduce jobs(e.g. if I run
OrderedWordCount, WordCount + Sort in MapReduce is faster than
OrderedWordCount on Tez sometimes). Is this correct behavior? Do you
know how can we solve it or tune Tez program? I attached logs at the
time as follows:
<--- log1 - job start up time -->
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 0% TotalTasks: -1 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
-- next print is 10 minutes later --
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 1.04% TotalTasks: 96 Succeeded: 1 Running: 52 Failed: 0
Killed: 0
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 3.37% TotalTasks: 89 Succeeded: 3 Running: 52
Failed: 0 Killed: 0
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
<--- log1 -->
<--- log2 -->
14/08/18 16:35:15 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 92.71% TotalTasks: 96 Succeeded: 89 Running: 6 Failed: 0
Killed: 0
14/08/18 16:35:15 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 100% TotalTasks: 89 Succeeded: 89 Running: 0
Failed: 0 Killed: 0
14/08/18 12:19:25 INFO rpc.DAGClientRPCImpl: VertexStatus:
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
14/08/18 16:35:15 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 6 Failed: 0
Killed: 0
--- next print is 10 minutes later --
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 93.75% TotalTasks: 96 Succeeded: 90 Running: 5 Failed: 0
Killed: 0
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 100% TotalTasks: 89 Succeeded: 89 Running: 0
Failed: 0 Killed: 0
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 16.67% TotalTasks: 6 Succeeded: 1 Running: 5
Failed: 0 Killed: 0
<--- log2 -->
Thanks,
- Tsuyoshi
RE: OrderedWordCount slow down or hung-up for a time
Posted by Bikas Saha <bi...@hortonworks.com>.
If you can repro the case, then can you please open a jira and attach the
logs.
Thanks
Bikas
-----Original Message-----
From: Tsuyoshi OZAWA [mailto:ozawa.tsuyoshi@gmail.com]
Sent: Monday, August 18, 2014 1:26 AM
To: user@tez.apache.org; dev@tez.apache.org
Subject: OrderedWordCount slow down or hung-up for a time
Hi,
I'm trying some jobs in examples of tez. I found that sometimes jobs get
hung-up on my distributed environment.
Because of this behavior, the jobs on tez get slow down and sometimes get
slower than original MapReduce jobs(e.g. if I run OrderedWordCount,
WordCount + Sort in MapReduce is faster than OrderedWordCount on Tez
sometimes). Is this correct behavior? Do you know how can we solve it or
tune Tez program? I attached logs at the time as follows:
<--- log1 - job start up time -->
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 0% TotalTasks: -1 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
-- next print is 10 minutes later --
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 1.04% TotalTasks: 96 Succeeded: 1 Running: 52 Failed: 0
Killed: 0
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 3.37% TotalTasks: 89 Succeeded: 3 Running: 52
Failed: 0 Killed: 0
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
<--- log1 -->
<--- log2 -->
14/08/18 16:35:15 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 92.71% TotalTasks: 96 Succeeded: 89 Running: 6 Failed: 0
Killed: 0
14/08/18 16:35:15 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 100% TotalTasks: 89 Succeeded: 89 Running: 0
Failed: 0 Killed: 0
14/08/18 12:19:25 INFO rpc.DAGClientRPCImpl: VertexStatus:
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
14/08/18 16:35:15 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 6 Failed: 0
Killed: 0
--- next print is 10 minutes later --
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 93.75% TotalTasks: 96 Succeeded: 90 Running: 5 Failed: 0
Killed: 0
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 100% TotalTasks: 89 Succeeded: 89 Running: 0
Failed: 0 Killed: 0
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 16.67% TotalTasks: 6 Succeeded: 1 Running: 5
Failed: 0 Killed: 0
<--- log2 -->
Thanks,
- Tsuyoshi
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RE: OrderedWordCount slow down or hung-up for a time
Posted by Bikas Saha <bi...@hortonworks.com>.
If you can repro the case, then can you please open a jira and attach the
logs.
Thanks
Bikas
-----Original Message-----
From: Tsuyoshi OZAWA [mailto:ozawa.tsuyoshi@gmail.com]
Sent: Monday, August 18, 2014 1:26 AM
To: user@tez.apache.org; dev@tez.apache.org
Subject: OrderedWordCount slow down or hung-up for a time
Hi,
I'm trying some jobs in examples of tez. I found that sometimes jobs get
hung-up on my distributed environment.
Because of this behavior, the jobs on tez get slow down and sometimes get
slower than original MapReduce jobs(e.g. if I run OrderedWordCount,
WordCount + Sort in MapReduce is faster than OrderedWordCount on Tez
sometimes). Is this correct behavior? Do you know how can we solve it or
tune Tez program? I attached logs at the time as follows:
<--- log1 - job start up time -->
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 0% TotalTasks: -1 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
14/08/18 16:02:36 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
-- next print is 10 minutes later --
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 1.04% TotalTasks: 96 Succeeded: 1 Running: 52 Failed: 0
Killed: 0
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 3.37% TotalTasks: 89 Succeeded: 3 Running: 52
Failed: 0 Killed: 0
14/08/18 16:15:57 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
<--- log1 -->
<--- log2 -->
14/08/18 16:35:15 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 92.71% TotalTasks: 96 Succeeded: 89 Running: 6 Failed: 0
Killed: 0
14/08/18 16:35:15 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 100% TotalTasks: 89 Succeeded: 89 Running: 0
Failed: 0 Killed: 0
14/08/18 12:19:25 INFO rpc.DAGClientRPCImpl: VertexStatus:
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
14/08/18 16:35:15 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 0% TotalTasks: 6 Succeeded: 0 Running: 6 Failed: 0
Killed: 0
--- next print is 10 minutes later --
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: DAG: State: RUNNING
Progress: 93.75% TotalTasks: 96 Succeeded: 90 Running: 5 Failed: 0
Killed: 0
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0
Killed: 0
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Tokenizer Progress: 100% TotalTasks: 89 Succeeded: 89 Running: 0
Failed: 0 Killed: 0
14/08/18 16:53:28 INFO rpc.DAGClientRPCImpl: VertexStatus: VertexName:
Summation Progress: 16.67% TotalTasks: 6 Succeeded: 1 Running: 5
Failed: 0 Killed: 0
<--- log2 -->
Thanks,
- Tsuyoshi
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.