You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Bikas Saha <bi...@hortonworks.com> on 2014/07/15 18:29:38 UTC
RE: Error when submit a hbase rowcounter job on Tez
What build of Tez and YARN are you using?
Bikas
*From:* 袁奎(震河) [mailto:zhenhe.yk@taobao.com]
*Sent:* Tuesday, July 15, 2014 1:15 AM
*To:* user@tez.incubator.apache.org
*Subject:* Error when submit a hbase rowcounter job on Tez
I deploy tez on my gateway,config the tea-site.xml as the Install/Deploy
Instructions <http://tez.incubator.apache.org/install.html>
command:
hadoop jar ~/hbase/hbase.jar rowcounter table_name
then I submit a rowcounter job, it fails as below.
I have no idea what’s wrong, anyone have any idea?
-----------------------------------------------------------
14/07/15 16:10:49 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1404997729523_0326
14/07/15 16:10:49 INFO client.YARNRunner: Number of stages: 1
14/07/15 16:10:49 INFO client.TezClient: Submitting DAG to YARN,
applicationId=application_1404997729523_0326
14/07/15 16:10:49 INFO impl.YarnClientImpl: Submitted application
application_1404997729523_0326 to ResourceManager at r101072037.sqa.zmf/
10.101.72.37:18040
14/07/15 16:10:49 INFO client.RMProxy: Connecting to ResourceManager at
r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO client.RMProxy: Connecting to ResourceManager at
r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO mapreduce.Job: The url to track the job:
r101072037.sqa.zmf:18088/proxy/application_1404997729523_0326/
14/07/15 16:10:49 INFO mapreduce.Job: Running job: job_1404997729523_0326
14/07/15 16:10:55 INFO mapreduce.Job: Job job_1404997729523_0326 running in
uber mode : false
14/07/15 16:10:55 INFO mapreduce.Job: map 0% reduce 0%
14/07/15 16:11:03 INFO mapreduce.Job: Job job_1404997729523_0326 failed
with state FAILED due to: Vertex failed, vertexName=initialmap,
vertexId=vertex_1404997729523_0326_1_00, diagnostics=[Task failed,
taskId=task_1404997729523_0326_1_00_000000,
diagnostics=[AttemptID:attempt_1404997729523_0326_1_00_000000_0
Info:Container launch failed for container_1404997729523_0326_01_000004 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application,
AttemptID:attempt_1404997729523_0326_1_00_000000_1 Info:Container launch
failed for container_1404997729523_0326_01_000007 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application,
AttemptID:attempt_1404997729523_0326_1_00_000000_2 Info:Container launch
failed for container_1404997729523_0326_01_000012 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application,
AttemptID:attempt_1404997729523_0326_1_00_000000_3 Info:Container launch
failed for container_1404997729523_0326_01_000017 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
], Vertex failed as one or more tasks failed. failedTasks:1]. DAG failed
due to vertex failure. failedVertices:1 killedVertices:0
14/07/15 16:11:03 INFO mapreduce.Job: Counters: 0
✉
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RE: Error when submit a hbase rowcounter job on Tez
Posted by Bikas Saha <bi...@hortonworks.com>.
Its probably a YARN/CDH bug in 2.2 that got fixed in 2.3.
Bikas
*From:* yuankui [mailto:zhenhe.yk@taobao.com]
*Sent:* Wednesday, July 16, 2014 3:08 AM
*To:* user@tez.incubator.apache.org
*Subject:* Re: Error when submit a hbase rowcounter job on Tez
thank Bikas!
I upgrade my hadoop version to 2.3.0-cdh5,then It’s Ok
I guess there 2.2.0-cdh5-beta1 does not work well with tea-0.4.1
✉
在 2014年7月16日，上午10:00，Bikas Saha <bi...@hortonworks.com> 写道：
Are you compiling tez against Apache Hadoop 2.2.0 or cdh-hadoop? It may be
there is some difference between them.
Try compiling tez against cdh-hadoop and using that.
This error is unexpected and so it may be some mismatch between the
different versions. Or it may be a bug in Hadoop 2.2.
This code is not part of Tez. It is a yarn library code.
Bikas
*From:* yuankui [mailto:zhenhe.yk@taobao.com]
*Sent:* Tuesday, July 15, 2014 6:45 PM
*To:* user@tez.incubator.apache.org
*Subject:* Re: Error when submit a hbase rowcounter job on Tez
I use tez-0.4.1-incubating & I compile tez in hadoop-2.2.0 and runing on
hadoop-2.2.0-cdh5.0.0-beta-1
✉
在 2014年7月16日，上午12:29，Bikas Saha <bi...@hortonworks.com> 写道：
What build of Tez and YARN are you using?
Bikas
*From:* 袁奎(震河) [mailto:zhenhe.yk@taobao.com]
*Sent:* Tuesday, July 15, 2014 1:15 AM
*To:* user@tez.incubator.apache.org
*Subject:* Error when submit a hbase rowcounter job on Tez
I deploy tez on my gateway,config the tea-site.xml as the Install/Deploy
Instructions <http://tez.incubator.apache.org/install.html>
command:
hadoop jar ~/hbase/hbase.jar rowcounter table_name
then I submit a rowcounter job, it fails as below.
I have no idea what’s wrong, anyone have any idea?
-----------------------------------------------------------
14/07/15 16:10:49 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1404997729523_0326
14/07/15 16:10:49 INFO client.YARNRunner: Number of stages: 1
14/07/15 16:10:49 INFO client.TezClient: Submitting DAG to YARN,
applicationId=application_1404997729523_0326
14/07/15 16:10:49 INFO impl.YarnClientImpl: Submitted application
application_1404997729523_0326 to ResourceManager at r101072037.sqa.zmf/
10.101.72.37:18040
14/07/15 16:10:49 INFO client.RMProxy: Connecting to ResourceManager at
r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO client.RMProxy: Connecting to ResourceManager at
r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO mapreduce.Job: The url to track the job:
r101072037.sqa.zmf:18088/proxy/application_1404997729523_0326/
14/07/15 16:10:49 INFO mapreduce.Job: Running job: job_1404997729523_0326
14/07/15 16:10:55 INFO mapreduce.Job: Job job_1404997729523_0326 running in
uber mode : false
14/07/15 16:10:55 INFO mapreduce.Job: map 0% reduce 0%
14/07/15 16:11:03 INFO mapreduce.Job: Job job_1404997729523_0326 failed
with state FAILED due to: Vertex failed, vertexName=initialmap,
vertexId=vertex_1404997729523_0326_1_00, diagnostics=[Task failed,
taskId=task_1404997729523_0326_1_00_000000,
diagnostics=[AttemptID:attempt_1404997729523_0326_1_00_000000_0
Info:Container launch failed for container_1404997729523_0326_01_000004 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application,
AttemptID:attempt_1404997729523_0326_1_00_000000_1 Info:Container launch
failed for container_1404997729523_0326_01_000007 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application,
AttemptID:attempt_1404997729523_0326_1_00_000000_2 Info:Container launch
failed for container_1404997729523_0326_01_000012 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application,
AttemptID:attempt_1404997729523_0326_1_00_000000_3 Info:Container launch
failed for container_1404997729523_0326_01_000017 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
], Vertex failed as one or more tasks failed. failedTasks:1]. DAG failed
due to vertex failure. failedVertices:1 killedVertices:0
14/07/15 16:11:03 INFO mapreduce.Job: Counters: 0
✉
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Error when submit a hbase rowcounter job on Tez
Posted by yuankui <zh...@taobao.com>.
thank Bikas!
I upgrade my hadoop version to 2.3.0-cdh5,then It’s Ok
I guess there 2.2.0-cdh5-beta1 does not work well with tea-0.4.1
✉
在 2014年7月16日，上午10:00，Bikas Saha <bi...@hortonworks.com> 写道：
Are you compiling tez against Apache Hadoop 2.2.0 or cdh-hadoop? It may be there is some difference between them.
Try compiling tez against cdh-hadoop and using that.
This error is unexpected and so it may be some mismatch between the different versions. Or it may be a bug in Hadoop 2.2.
This code is not part of Tez. It is a yarn library code.
Bikas
From: yuankui [mailto:zhenhe.yk@taobao.com]
Sent: Tuesday, July 15, 2014 6:45 PM
To: user@tez.incubator.apache.org
Subject: Re: Error when submit a hbase rowcounter job on Tez
I use tez-0.4.1-incubating & I compile tez in hadoop-2.2.0 and runing on hadoop-2.2.0-cdh5.0.0-beta-1
✉
在 2014年7月16日，上午12:29，Bikas Saha <bi...@hortonworks.com> 写道：
What build of Tez and YARN are you using?
Bikas
From: 袁奎(震河) [mailto:zhenhe.yk@taobao.com]
Sent: Tuesday, July 15, 2014 1:15 AM
To: user@tez.incubator.apache.org
Subject: Error when submit a hbase rowcounter job on Tez
I deploy tez on my gateway,config the tea-site.xml as the Install/Deploy Instructions
command:
hadoop jar ~/hbase/hbase.jar rowcounter table_name
then I submit a rowcounter job, it fails as below.
I have no idea what’s wrong, anyone have any idea?
-----------------------------------------------------------
14/07/15 16:10:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1404997729523_0326
14/07/15 16:10:49 INFO client.YARNRunner: Number of stages: 1
14/07/15 16:10:49 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1404997729523_0326
14/07/15 16:10:49 INFO impl.YarnClientImpl: Submitted application application_1404997729523_0326 to ResourceManager at r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO client.RMProxy: Connecting to ResourceManager at r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO client.RMProxy: Connecting to ResourceManager at r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO mapreduce.Job: The url to track the job: r101072037.sqa.zmf:18088/proxy/application_1404997729523_0326/
14/07/15 16:10:49 INFO mapreduce.Job: Running job: job_1404997729523_0326
14/07/15 16:10:55 INFO mapreduce.Job: Job job_1404997729523_0326 running in uber mode : false
14/07/15 16:10:55 INFO mapreduce.Job: map 0% reduce 0%
14/07/15 16:11:03 INFO mapreduce.Job: Job job_1404997729523_0326 failed with state FAILED due to: Vertex failed, vertexName=initialmap, vertexId=vertex_1404997729523_0326_1_00, diagnostics=[Task failed, taskId=task_1404997729523_0326_1_00_000000, diagnostics=[AttemptID:attempt_1404997729523_0326_1_00_000000_0 Info:Container launch failed for container_1404997729523_0326_01_000004 : java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application, AttemptID:attempt_1404997729523_0326_1_00_000000_1 Info:Container launch failed for container_1404997729523_0326_01_000007 : java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application, AttemptID:attempt_1404997729523_0326_1_00_000000_2 Info:Container launch failed for container_1404997729523_0326_01_000012 : java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application, AttemptID:attempt_1404997729523_0326_1_00_000000_3 Info:Container launch failed for container_1404997729523_0326_01_000017 : java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
], Vertex failed as one or more tasks failed. failedTasks:1]. DAG failed due to vertex failure. failedVertices:1 killedVertices:0
14/07/15 16:11:03 INFO mapreduce.Job: Counters: 0
✉
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Error when submit a hbase rowcounter job on Tez
Posted by Bikas Saha <bi...@hortonworks.com>.
Are you compiling tez against Apache Hadoop 2.2.0 or cdh-hadoop? It may be
there is some difference between them.
Try compiling tez against cdh-hadoop and using that.
This error is unexpected and so it may be some mismatch between the
different versions. Or it may be a bug in Hadoop 2.2.
This code is not part of Tez. It is a yarn library code.
Bikas
*From:* yuankui [mailto:zhenhe.yk@taobao.com]
*Sent:* Tuesday, July 15, 2014 6:45 PM
*To:* user@tez.incubator.apache.org
*Subject:* Re: Error when submit a hbase rowcounter job on Tez
I use tez-0.4.1-incubating & I compile tez in hadoop-2.2.0 and runing on
hadoop-2.2.0-cdh5.0.0-beta-1
✉
在 2014年7月16日，上午12:29，Bikas Saha <bi...@hortonworks.com> 写道：
What build of Tez and YARN are you using?
Bikas
*From:* 袁奎(震河) [mailto:zhenhe.yk@taobao.com]
*Sent:* Tuesday, July 15, 2014 1:15 AM
*To:* user@tez.incubator.apache.org
*Subject:* Error when submit a hbase rowcounter job on Tez
I deploy tez on my gateway,config the tea-site.xml as the Install/Deploy
Instructions <http://tez.incubator.apache.org/install.html>
command:
hadoop jar ~/hbase/hbase.jar rowcounter table_name
then I submit a rowcounter job, it fails as below.
I have no idea what’s wrong, anyone have any idea?
-----------------------------------------------------------
14/07/15 16:10:49 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1404997729523_0326
14/07/15 16:10:49 INFO client.YARNRunner: Number of stages: 1
14/07/15 16:10:49 INFO client.TezClient: Submitting DAG to YARN,
applicationId=application_1404997729523_0326
14/07/15 16:10:49 INFO impl.YarnClientImpl: Submitted application
application_1404997729523_0326 to ResourceManager at r101072037.sqa.zmf/
10.101.72.37:18040
14/07/15 16:10:49 INFO client.RMProxy: Connecting to ResourceManager at
r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO client.RMProxy: Connecting to ResourceManager at
r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO mapreduce.Job: The url to track the job:
r101072037.sqa.zmf:18088/proxy/application_1404997729523_0326/
14/07/15 16:10:49 INFO mapreduce.Job: Running job: job_1404997729523_0326
14/07/15 16:10:55 INFO mapreduce.Job: Job job_1404997729523_0326 running in
uber mode : false
14/07/15 16:10:55 INFO mapreduce.Job: map 0% reduce 0%
14/07/15 16:11:03 INFO mapreduce.Job: Job job_1404997729523_0326 failed
with state FAILED due to: Vertex failed, vertexName=initialmap,
vertexId=vertex_1404997729523_0326_1_00, diagnostics=[Task failed,
taskId=task_1404997729523_0326_1_00_000000,
diagnostics=[AttemptID:attempt_1404997729523_0326_1_00_000000_0
Info:Container launch failed for container_1404997729523_0326_01_000004 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application,
AttemptID:attempt_1404997729523_0326_1_00_000000_1 Info:Container launch
failed for container_1404997729523_0326_01_000007 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application,
AttemptID:attempt_1404997729523_0326_1_00_000000_2 Info:Container launch
failed for container_1404997729523_0326_01_000012 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application,
AttemptID:attempt_1404997729523_0326_1_00_000000_3 Info:Container launch
failed for container_1404997729523_0326_01_000017 :
java.lang.NullPointerException
at
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
], Vertex failed as one or more tasks failed. failedTasks:1]. DAG failed
due to vertex failure. failedVertices:1 killedVertices:0
14/07/15 16:11:03 INFO mapreduce.Job: Counters: 0
✉
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Error when submit a hbase rowcounter job on Tez
Posted by yuankui <zh...@taobao.com>.
I use tez-0.4.1-incubating & I compile tez in hadoop-2.2.0 and runing on hadoop-2.2.0-cdh5.0.0-beta-1
✉
在 2014年7月16日，上午12:29，Bikas Saha <bi...@hortonworks.com> 写道：
What build of Tez and YARN are you using?
Bikas
From: 袁奎(震河) [mailto:zhenhe.yk@taobao.com]
Sent: Tuesday, July 15, 2014 1:15 AM
To: user@tez.incubator.apache.org
Subject: Error when submit a hbase rowcounter job on Tez
I deploy tez on my gateway,config the tea-site.xml as the Install/Deploy Instructions
command:
hadoop jar ~/hbase/hbase.jar rowcounter table_name
then I submit a rowcounter job, it fails as below.
I have no idea what’s wrong, anyone have any idea?
-----------------------------------------------------------
14/07/15 16:10:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1404997729523_0326
14/07/15 16:10:49 INFO client.YARNRunner: Number of stages: 1
14/07/15 16:10:49 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1404997729523_0326
14/07/15 16:10:49 INFO impl.YarnClientImpl: Submitted application application_1404997729523_0326 to ResourceManager at r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO client.RMProxy: Connecting to ResourceManager at r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO client.RMProxy: Connecting to ResourceManager at r101072037.sqa.zmf/10.101.72.37:18040
14/07/15 16:10:49 INFO mapreduce.Job: The url to track the job: r101072037.sqa.zmf:18088/proxy/application_1404997729523_0326/
14/07/15 16:10:49 INFO mapreduce.Job: Running job: job_1404997729523_0326
14/07/15 16:10:55 INFO mapreduce.Job: Job job_1404997729523_0326 running in uber mode : false
14/07/15 16:10:55 INFO mapreduce.Job: map 0% reduce 0%
14/07/15 16:11:03 INFO mapreduce.Job: Job job_1404997729523_0326 failed with state FAILED due to: Vertex failed, vertexName=initialmap, vertexId=vertex_1404997729523_0326_1_00, diagnostics=[Task failed, taskId=task_1404997729523_0326_1_00_000000, diagnostics=[AttemptID:attempt_1404997729523_0326_1_00_000000_0 Info:Container launch failed for container_1404997729523_0326_01_000004 : java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application, AttemptID:attempt_1404997729523_0326_1_00_000000_1 Info:Container launch failed for container_1404997729523_0326_01_000007 : java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application, AttemptID:attempt_1404997729523_0326_1_00_000000_2 Info:Container launch failed for container_1404997729523_0326_01_000012 : java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container released by application, AttemptID:attempt_1404997729523_0326_1_00_000000_3 Info:Container launch failed for container_1404997729523_0326_01_000017 : java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:115)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:352)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:148)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
], Vertex failed as one or more tasks failed. failedTasks:1]. DAG failed due to vertex failure. failedVertices:1 killedVertices:0
14/07/15 16:11:03 INFO mapreduce.Job: Counters: 0
✉
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.