[jira] [Commented] (TEZ-3884) Hadoop3-beta1 fixes for Tez tests

["Thai Bui (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371812#comment-16371812 ] 

Thai Bui commented on TEZ-3884:
-------------------------------

[~ewohlstadter] I was working on getting tez 0.9.1 + Hive 3.x to use hadoop 3.0.0 and encountered a dependency conflict between guava 11.x and hadoop 3 guava 19. I supposed we should upgrade Guava as well? 

Here's the dependency conflict detail causing DAGAppMaster to shutdown immediately after it had started 
{noformat}
2018-02-21 06:51:00,829 [FATAL] [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6] |yarn.YarnUncaughtExceptionHandler|: Thread Thread[org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6,5,main] threw an Error.  Shutting down now...
java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createUnstarted()Lcom/google/common/base/Stopwatch;
	at org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:176)
	at java.lang.Thread.run(Thread.java:748)
2018-02-21 06:51:00,831 [DEBUG] [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6] |util.ExitUtil|: Exiting with status -1: ExitException
-1: ExitException
	at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
	at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:292)
	at org.apache.hadoop.yarn.YarnUncaughtExceptionHandler.uncaughtException(YarnUncaughtExceptionHandler.java:65)
	at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
	at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
	at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
2018-02-21 06:51:00,831 [INFO] [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@3cec92b6] |util.ExitUtil|: Exiting with status -1: ExitException
2018-02-21 06:51:00,832 [DEBUG] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |impl.MetricsSystemImpl|: JvmMetrics, JVM related metrics etc.
2018-02-21 06:51:00,833 [DEBUG] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |impl.MetricsConfig|: poking parent 'PropertiesConfiguration' for key: source.source.start_mbeans
2018-02-21 06:51:00,833 [DEBUG] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |impl.MetricsConfig|: poking parent 'MetricsConfig' for key: source.start_mbeans
2018-02-21 06:51:00,833 [DEBUG] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |impl.MetricsConfig|: poking parent 'PropertiesConfiguration' for key: *.source.start_mbeans
2018-02-21 06:51:00,833 [INFO] [pool-1-thread-1] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked
2018-02-21 06:51:00,833 [INFO] [pool-1-thread-1] |app.DAGAppMaster|: DAGAppMaster received a signal. Signaling TaskScheduler
2018-02-21 06:51:00,833 [INFO] [pool-1-thread-1] |rm.TaskSchedulerManager|: TaskScheduler notified that iSignalled was : true{noformat}
Noticed that DAGAppMaster received a KILL signal due to Guava dependency conflict in JvmPauseMonitor. This caused the DAGAppMaster to shutdown immediately after it had started
{noformat}
2018-02-21 06:51:36,280 [DEBUG] [main] |service.AbstractService|: Service org.apache.tez.dag.app.DAGAppMaster is started
2018-02-21 06:51:36,280 [DEBUG] [pool-1-thread-1] |service.AbstractService|: Service: org.apache.tez.dag.app.DAGAppMaster entered state STOPPED
2018-02-21 06:51:36,280 [DEBUG] [IPC Parameter Sending Thread #0] |ipc.Client|: IPC Client (484199463) connection to ip-10-8-118-171/10.8.118.171:18030 from root sending #11 org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate
2018-02-21 06:51:36,280 [DEBUG] [IPC Client (484199463) connection to ip-10-8-118-171/10.8.118.171:18030 from root] |ipc.Client|: IPC Client (484199463) connection to ip-10-8-118-171/10.8.118.171:18030 from root: starting, having connections 1
2018-02-21 06:51:36,280 [INFO] [pool-1-thread-1] |rm.YarnTaskSchedulerService|: Initiating stop of YarnTaskScheduler
2018-02-21 06:51:36,281 [INFO] [pool-1-thread-1] |rm.YarnTaskSchedulerService|: Releasing held containers
2018-02-21 06:51:36,281 [INFO] [pool-1-thread-1] |rm.YarnTaskSchedulerService|: Removing all pending taskRequests{noformat}
I'm working on upgrading Guava and perhaps various other issue and will keep it posted here. Let me know what you guys think. Thanks!

> Hadoop3-beta1 fixes for Tez tests
> ---------------------------------
>
>                 Key: TEZ-3884
>                 URL: https://issues.apache.org/jira/browse/TEZ-3884
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.9.1
>            Reporter: Gopal V
>            Priority: Minor
>         Attachments: TEZ-3884.patch.1
>
>
> {code}
> [ERROR] /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[48,30] cannot find symbol
> [ERROR] symbol:   class DistributedFileSystem
> [ERROR] location: package org.apache.hadoop.hdfs
> [ERROR] /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[680,50] cannot find symbol
> [ERROR] symbol:   class DistributedFileSystem
> [ERROR] location: class org.apache.tez.client.TestTezClientUtils
> [ERROR] /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/common/TestTezCommonUtils.java:[62,42] cannot access org.apache.hadoop.hdfs.DistributedFileSystem
> [ERROR] class file for org.apache.hadoop.hdfs.DistributedFileSystem not found
> [ERROR] -> [Help 1]
> [ERROR] 
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)