You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Jonathan Turner Eagles (Jira)" <ji...@apache.org> on 2021/04/23 21:02:00 UTC
[jira] [Commented] (TEZ-3421) cdh 5.5.2 cann't use tez 0.8.4
[ https://issues.apache.org/jira/browse/TEZ-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331035#comment-17331035 ]
Jonathan Turner Eagles commented on TEZ-3421:
---------------------------------------------
[~ft20082], Is this jira still relevant? Trying to close out old jiras.
> cdh 5.5.2 cann't use tez 0.8.4
> ------------------------------
>
> Key: TEZ-3421
> URL: https://issues.apache.org/jira/browse/TEZ-3421
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.7.1, 0.8.4
> Environment: cdh 5.5.2, hive 1.1.0, tez 0.8.4
> Reporter: Jake.zhang
> Priority: Major
>
> following the install guide, and it don't work.
> {code}
> 2016-08-31 15:31:20,614 [INFO] [IPC Server handler 0 on 43797] |app.DAGAppMaster|: Running DAG: root_20160831153131_0c02cc1a-1447-4ce9-a4b1-f66c2b8e1354:2
> 2016-08-31 15:31:20,849 [INFO] [IPC Server handler 0 on 43797] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1470895429219_0037_1][Event:DAG_SUBMITTED]: dagID=dag_1470895429219_0037_1, submitTime=1472628680567
> 2016-08-31 15:31:20,886 [INFO] [IPC Server handler 0 on 43797] |impl.VertexImpl|: Default container context for vertex_1470895429219_0037_1_00 [Map 1]=LocalResources: [[ name=tezlib, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/user/tez/tez-0.8.4.tar.gz" } size: 48523006 timestamp: 1472625394859 type: ARCHIVE visibility: PUBLIC],[ name=hbase-client.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-client.jar" } size: 1294765 timestamp: 1472625755599 type: FILE visibility: PRIVATE],[ name=hbase-server.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-server.jar" } size: 4299872 timestamp: 1472625755903 type: FILE visibility: PRIVATE],[ name=hive-hbase-handler-1.1.0-cdh5.8.0.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hive-hbase-handler-1.1.0-cdh5.8.0.jar" } size: 118398 timestamp: 1472625755443 type: FILE visibility: PRIVATE],[ name=postgresql-9.0-801.jdbc4.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/postgresql-9.0-801.jdbc4.jar" } size: 539705 timestamp: 1472625756348 type: FILE visibility: PRIVATE],[ name=hbase-common.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-common.jar" } size: 581623 timestamp: 1472625755634 type: FILE visibility: PRIVATE],[ name=htrace-core-3.2.0-incubating.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core-3.2.0-incubating.jar" } size: 1483913 timestamp: 1472625756002 type: FILE visibility: PRIVATE],[ name=hbase-hadoop-compat.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-hadoop-compat.jar" } size: 39226 timestamp: 1472625755671 type: FILE visibility: PRIVATE],[ name=htrace-core4-4.0.1-incubating.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core4-4.0.1-incubating.jar" } size: 1485102 timestamp: 1472625756117 type: FILE visibility: PRIVATE],[ name=hbase-protocol.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-protocol.jar" } size: 4378452 timestamp: 1472625755802 type: FILE visibility: PRIVATE],[ name=htrace-core.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core.jar" } size: 1483913 timestamp: 1472625756052 type: FILE visibility: PRIVATE],[ name=hbase-hadoop2-compat.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-hadoop2-compat.jar" } size: 97555 timestamp: 1472625755713 type: FILE visibility: PRIVATE],[ name=tez-conf.pb, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/.tez/application_1470895429219_0037/tez-conf.pb" } size: 66799 timestamp: 1472628677372 type: FILE visibility: APPLICATION],[ name=hive-exec-1.1.0-cdh5.8.0-a7a75fba1322d9f2c8d2af1c5461acb91e9006758e4a4b1a3f83f2a0bd646623.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/user/root/.hiveJars/hive-exec-1.1.0-cdh5.8.0-a7a75fba1322d9f2c8d2af1c5461acb91e9006758e4a4b1a3f83f2a0bd646623.jar" } size: 19306196 timestamp: 1472610430690 type: FILE visibility: PRIVATE],], environment: [[ SHELL=/bin/bash ],[ CLASSPATH=$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR: ],[ HADOOP_ROOT_LOGGER=INFO,CLA ],[ LD_LIBRARY_PATH=$PWD:$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.net.preferIPv4Stack=true -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA , vertex: vertex_1470895429219_0037_1_00 [Map 1], Default Resources=<memory:0, vCores:1>
> 2016-08-31 15:31:20,886 [INFO] [IPC Server handler 0 on 43797] |impl.VertexImpl|: Setting 1 additional inputs for vertexvertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,887 [INFO] [IPC Server handler 0 on 43797] |impl.VertexImpl|: Running vertex: vertex_1470895429219_0037_1_00 [Map 1] : TaskScheduler=0:TezYarn, ContainerLauncher=0:TezYarn, TaskCommunicator=0:TezYarn
> 2016-08-31 15:31:20,894 [INFO] [IPC Server handler 0 on 43797] |impl.VertexImpl|: Default container context for vertex_1470895429219_0037_1_01 [Reducer 2]=LocalResources: [[ name=tezlib, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/user/tez/tez-0.8.4.tar.gz" } size: 48523006 timestamp: 1472625394859 type: ARCHIVE visibility: PUBLIC],[ name=hbase-client.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-client.jar" } size: 1294765 timestamp: 1472625755599 type: FILE visibility: PRIVATE],[ name=hbase-server.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-server.jar" } size: 4299872 timestamp: 1472625755903 type: FILE visibility: PRIVATE],[ name=hive-hbase-handler-1.1.0-cdh5.8.0.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hive-hbase-handler-1.1.0-cdh5.8.0.jar" } size: 118398 timestamp: 1472625755443 type: FILE visibility: PRIVATE],[ name=postgresql-9.0-801.jdbc4.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/postgresql-9.0-801.jdbc4.jar" } size: 539705 timestamp: 1472625756348 type: FILE visibility: PRIVATE],[ name=hbase-common.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-common.jar" } size: 581623 timestamp: 1472625755634 type: FILE visibility: PRIVATE],[ name=htrace-core-3.2.0-incubating.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core-3.2.0-incubating.jar" } size: 1483913 timestamp: 1472625756002 type: FILE visibility: PRIVATE],[ name=hbase-hadoop-compat.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-hadoop-compat.jar" } size: 39226 timestamp: 1472625755671 type: FILE visibility: PRIVATE],[ name=htrace-core4-4.0.1-incubating.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core4-4.0.1-incubating.jar" } size: 1485102 timestamp: 1472625756117 type: FILE visibility: PRIVATE],[ name=hbase-protocol.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-protocol.jar" } size: 4378452 timestamp: 1472625755802 type: FILE visibility: PRIVATE],[ name=htrace-core.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core.jar" } size: 1483913 timestamp: 1472625756052 type: FILE visibility: PRIVATE],[ name=hbase-hadoop2-compat.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-hadoop2-compat.jar" } size: 97555 timestamp: 1472625755713 type: FILE visibility: PRIVATE],[ name=tez-conf.pb, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/.tez/application_1470895429219_0037/tez-conf.pb" } size: 66799 timestamp: 1472628677372 type: FILE visibility: APPLICATION],[ name=hive-exec-1.1.0-cdh5.8.0-a7a75fba1322d9f2c8d2af1c5461acb91e9006758e4a4b1a3f83f2a0bd646623.jar, value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file: "/user/root/.hiveJars/hive-exec-1.1.0-cdh5.8.0-a7a75fba1322d9f2c8d2af1c5461acb91e9006758e4a4b1a3f83f2a0bd646623.jar" } size: 19306196 timestamp: 1472610430690 type: FILE visibility: PRIVATE],], environment: [[ SHELL=/bin/bash ],[ CLASSPATH=$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR: ],[ HADOOP_ROOT_LOGGER=INFO,CLA ],[ LD_LIBRARY_PATH=$PWD:$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.net.preferIPv4Stack=true -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA , vertex: vertex_1470895429219_0037_1_01 [Reducer 2], Default Resources=<memory:0, vCores:1>
> 2016-08-31 15:31:20,894 [INFO] [IPC Server handler 0 on 43797] |impl.VertexImpl|: Setting 1 additional outputs for vertex vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:20,895 [INFO] [IPC Server handler 0 on 43797] |impl.VertexImpl|: Running vertex: vertex_1470895429219_0037_1_01 [Reducer 2] : TaskScheduler=0:TezYarn, ContainerLauncher=0:TezYarn, TaskCommunicator=0:TezYarn
> 2016-08-31 15:31:20,900 [INFO] [IPC Server handler 0 on 43797] |impl.DAGImpl|: Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrder
> 2016-08-31 15:31:20,901 [INFO] [IPC Server handler 0 on 43797] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1470895429219_0037_1][Event:DAG_INITIALIZED]: dagID=dag_1470895429219_0037_1, initTime=1472628680901
> 2016-08-31 15:31:20,901 [INFO] [IPC Server handler 0 on 43797] |impl.DAGImpl|: dag_1470895429219_0037_1 transitioned from NEW to INITED due to event DAG_INIT
> 2016-08-31 15:31:20,901 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Added additional resources : [[]] to classpath
> 2016-08-31 15:31:20,903 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1470895429219_0037_1][Event:DAG_STARTED]: dagID=dag_1470895429219_0037_1, startTime=1472628680901
> 2016-08-31 15:31:20,903 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1470895429219_0037_1 transitioned from INITED to RUNNING due to event DAG_START
> 2016-08-31 15:31:20,903 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Root Inputs exist for Vertex: Map 1 : {tab={InputName=tab}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInputLegacy, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator, hasPayload=false}}
> 2016-08-31 15:31:20,903 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Starting root input initializer for input: tab, with class: [org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator]
> 2016-08-31 15:31:20,903 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting vertexManager to RootInputVertexManager for vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,907 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,907 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex will initialize from input initializer. vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,908 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Starting 1 inputInitializers for vertex vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,916 [INFO] [InputInitializer {Map 1} #0] |dag.RootInputInitializerManager|: Starting InputInitializer for Input: tab on vertex vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,924 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1470895429219_0037_1_00 [Map 1] transitioned from NEW to INITIALIZING due to event V_INIT
> 2016-08-31 15:31:20,924 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting vertexManager to ShuffleVertexManager for vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:20,924 [INFO] [InputInitializer {Map 1} #0] |Configuration.deprecation|: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
> 2016-08-31 15:31:20,950 [INFO] [Dispatcher thread {Central}] |vertexmanager.ShuffleVertexManager|: Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
> 2016-08-31 15:31:20,950 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Creating 1 tasks for vertex: vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:20,958 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Directly initializing vertex: vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:20,959 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1470895429219_0037_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1470895429219_0037_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
> 2016-08-31 15:31:20,959 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting up committers for vertex vertex_1470895429219_0037_1_01 [Reducer 2], numAdditionalOutputs=1
> 2016-08-31 15:31:20,959 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1470895429219_0037_1][Event:VERTEX_INITIALIZED]: vertexName=Reducer 2, vertexId=vertex_1470895429219_0037_1_01, initRequestedTime=1472628680924, initedTime=1472628680959, numTasks=1, processorName=org.apache.hadoop.hive.ql.exec.tez.ReduceTezProcessor, additionalInputsCount=0, initGeneratedEventsCount=0, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl }
> 2016-08-31 15:31:20,959 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1470895429219_0037_1_01 [Reducer 2] transitioned from NEW to INITED due to event V_INIT
> 2016-08-31 15:31:20,976 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: PLAN PATH = hdfs://name2.test.kingnetdc.com:8020/tmp/hive/root/c933a804-f7ea-4a31-ae8e-ba47378e84b2/hive_2016-08-31_15-31-16_521_8741335194912077181-1/root/_tez_scratch_dir/0822a3c4-f015-4485-91d7-168ec4f3eacc/map.xml
> 2016-08-31 15:31:20,976 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: ***************non-local mode***************
> 2016-08-31 15:31:20,976 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: local path = hdfs://name2.test.kingnetdc.com:8020/tmp/hive/root/c933a804-f7ea-4a31-ae8e-ba47378e84b2/hive_2016-08-31_15-31-16_521_8741335194912077181-1/root/_tez_scratch_dir/0822a3c4-f015-4485-91d7-168ec4f3eacc/map.xml
> 2016-08-31 15:31:20,981 [INFO] [InputInitializer {Map 1} #0] |log.PerfLogger|: <PERFLOG method=deserializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
> 2016-08-31 15:31:20,982 [INFO] [InputInitializer {Map 1} #0] |exec.Utilities|: Deserializing MapWork via kryo
> 2016-08-31 15:31:21,082 [INFO] [InputInitializer {Map 1} #0] |log.PerfLogger|: </PERFLOG method=deserializePlan start=1472628680981 end=1472628681082 duration=101 from=org.apache.hadoop.hive.ql.exec.Utilities>
> 2016-08-31 15:31:21,085 [INFO] [InputInitializer {Map 1} #0] |dag.RootInputInitializerManager|: Failed InputInitializer for Input: tab on vertex vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:21,086 [ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input: tab initializer failed, vertex=vertex_1470895429219_0037_1_00 [Map 1]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException: java.lang.ArithmeticException: / by zero
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:327)
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1140)
> at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
> at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:150)
> at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:135)
> at com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91)
> at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:380)
> at java.util.concurrent.FutureTask.setException(FutureTask.java:247)
> at java.util.concurrent.FutureTask.run(FutureTask.java:267)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ArithmeticException: / by zero
> at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:123)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ... 3 more
> 2016-08-31 15:31:21,094 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1470895429219_0037_1][Event:VERTEX_FINISHED]: vertexName=Map 1, vertexId=vertex_1470895429219_0037_1_00, initRequestedTime=1472628680903, initedTime=0, startRequestedTime=1472628680924, startedTime=0, finishTime=1472628681089, timeTaken=1472628681089, status=FAILED, diagnostics=Vertex vertex_1470895429219_0037_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE
> Vertex Input: tab initializer failed, vertex=vertex_1470895429219_0037_1_00 [Map 1], java.lang.ArithmeticException: / by zero
> at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:123)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> , counters=Counters: 0, vertexStats=firstTaskStartTime=-1, firstTasksToStart=[ ], lastTaskFinishTime=-1, lastTasksToFinish=[ ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=0}, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl }
> 2016-08-31 15:31:21,094 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1470895429219_0037_1_00 [Map 1] transitioned from INITIALIZING to FAILED due to event V_ROOT_INPUT_FAILED
> 2016-08-31 15:31:21,095 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Vertex vertex_1470895429219_0037_1_00 [Map 1] completed., numCompletedVertices=1, numSuccessfulVertices=0, numFailedVertices=1, numKilledVertices=0, numVertices=2
> 2016-08-31 15:31:21,095 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=0, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
> 2016-08-31 15:31:21,095 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1470895429219_0037_1 transitioned from RUNNING to TERMINATING due to event DAG_VERTEX_COMPLETED
> 2016-08-31 15:31:21,095 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Invoking committer abort for vertex, vertexId=vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:21,097 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1470895429219_0037_1][Event:VERTEX_FINISHED]: vertexName=Reducer 2, vertexId=vertex_1470895429219_0037_1_01, initRequestedTime=1472628680924, initedTime=1472628680959, startRequestedTime=0, startedTime=0, finishTime=1472628681095, timeTaken=1472628681095, status=KILLED, diagnostics=Vertex received Kill in INITED state.
> Vertex vertex_1470895429219_0037_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=0, firstTasksToStart=[ task_1470895429219_0037_1_01_000000 ], lastTaskFinishTime=-1, lastTasksToFinish=[ ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=0}, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl, taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService, taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl }
> 2016-08-31 15:31:21,098 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1470895429219_0037_1_01 [Reducer 2] transitioned from INITED to KILLED due to event V_TERMINATE
> 2016-08-31 15:31:21,098 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Vertex vertex_1470895429219_0037_1_01 [Reducer 2] completed., numCompletedVertices=2, numSuccessfulVertices=0, numFailedVertices=1, numKilledVertices=1, numVertices=2
> 2016-08-31 15:31:21,098 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=0, numFailedVertices=1, numKilledVertices=1, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
> 2016-08-31 15:31:21,098 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
> 2016-08-31 15:31:21,098 [INFO] [Dispatcher thread {Central}] |counters.Limits|: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
> 2016-08-31 15:31:21,111 [INFO] [Dispatcher thread {Central}] |recovery.RecoveryService|: DAG completed, dagId=dag_1470895429219_0037_1, queueSize=0
> {code}
> {code}
> 2016-08-31 15:31:21,125 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1470895429219_0037_1][Event:DAG_FINISHED]: dagId=dag_1470895429219_0037_1, startTime=1472628680901, finishTime=1472628681098, timeTaken=197, status=FAILED, diagnostics=Vertex failed, vertexName=Map 1, vertexId=vertex_1470895429219_0037_1_00, diagnostics=[Vertex vertex_1470895429219_0037_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tab initializer failed, vertex=vertex_1470895429219_0037_1_00 [Map 1], java.lang.ArithmeticException: / by zero
> at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:123)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ]
> Vertex killed, vertexName=Reducer 2, vertexId=vertex_1470895429219_0037_1_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1470895429219_0037_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]
> DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1, counters=Counters: 2, org.apache.tez.common.counters.DAGCounter, AM_CPU_MILLISECONDS=450, AM_GC_TIME_MILLIS=0
> 2016-08-31 15:31:21,125 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Ignoring multiple aborts for vertex: vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:21,125 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Ignoring multiple aborts for vertex: vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:21,125 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: DAG: dag_1470895429219_0037_1 finished with state: FAILED
> 2016-08-31 15:31:21,125 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1470895429219_0037_1 transitioned from TERMINATING to FAILED due to event DAG_VERTEX_COMPLETED
> 2016-08-31 15:31:21,126 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: DAG completed, dagId=dag_1470895429219_0037_1, dagState=FAILED
> 2016-08-31 15:31:21,126 [INFO] [Dispatcher thread {Central}] |common.TezUtilsInternal|: Redirecting log file based on addend: dag_1470895429219_0037_1_post
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)