[GitHub] [hive] abstractdog commented on pull request #3279: HIVE-24484: Upgrade Hadoop to 3.3.3

[GitBox <gi...@apache.org>]

abstractdog commented on PR #3279:
URL: https://github.com/apache/hive/pull/3279#issuecomment-1141946273

   > I've deployed
   > 
   > * hadoop-3.3.3
   > * tez 0.10.1
   > * hive from the PR
   >   running a simple insert failed with:
   > 
   > ```
   > Caused by: java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;
   > 	at org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:569)
   > 	at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:550)
   > 	at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:117)
   > 	at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:425)
   > 	at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:341)
   > 	at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:432)
   > 	at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:428)
   > 	at org.apache.tez.dag.app.web.WebUIService.serviceStart(WebUIService.java:94)
   > 	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
   > 	at org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1800)
   > 	at org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1821)
   > 2022-05-31 09:17:19,422 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked
   > ```
   > 
   > maybe I've missed something - but it seems like the tez dagappmaster has issues running with the jetty because of hadoop 3.3.3
   > 
   > I've these settings:
   > 
   > ```
   > tez/tez-site/tez.lib.uris ${fs.defaultFS}/apps/tez/tez.tar.gz
   > tez/tez-site/tez.use.cluster.hadoop-libs true
   > ```
   > 
   > @abstractdog is hadoop-3.3.3 supported with tez-0.10.1?
   
   thanks @kgyrtkirk for trying this out, I've just created TEZ-4420, as tez is on hadoop 3.3.1, and I'm not sure about compatibility
   
   checked tez.tar.gz contents before and after bump and I got:
   
   ```
   hadoop 3.3.1
   
   tar tf tez-dist/target/tez-0.10.2-SNAPSHOT.tar.gz | grep jetty
   
   lib/jetty-server-9.4.40.v20210413.jar
   lib/jetty-http-9.4.40.v20210413.jar
   lib/jetty-io-9.4.40.v20210413.jar
   lib/jetty-util-9.4.40.v20210413.jar
   lib/jetty-servlet-9.4.40.v20210413.jar
   lib/jetty-security-9.4.40.v20210413.jar
   lib/jetty-util-ajax-9.4.40.v20210413.jar
   lib/jetty-webapp-9.4.40.v20210413.jar
   lib/jetty-xml-9.4.40.v20210413.jar
   lib/jetty-client-9.4.40.v20210413.jar
   
   hadoop 3.3.3
   
   tar tf tez-dist/target/tez-0.10.2-SNAPSHOT.tar.gz | grep jetty
   
   lib/jetty-server-9.4.43.v20210629.jar
   lib/jetty-http-9.4.43.v20210629.jar
   lib/jetty-io-9.4.43.v20210629.jar
   lib/jetty-util-9.4.43.v20210629.jar
   lib/jetty-servlet-9.4.43.v20210629.jar
   lib/jetty-security-9.4.43.v20210629.jar
   lib/jetty-util-ajax-9.4.43.v20210629.jar
   lib/jetty-webapp-9.4.43.v20210629.jar
   lib/jetty-xml-9.4.43.v20210629.jar
   lib/jetty-client-9.4.43.v20210629.jar
   ``` 
   
   tez packs jetty from hadoop, so hadoop upgrade means jetty upgrade too, so there is a chance that an old tez.tar.gz can clash with new hadoop
   
   I've just uplodaded the new tez.tar.gz, is there a chance you can give it a try?
   https://drive.google.com/file/d/18RMfh40s6kKdFt77E7HpS-j4EJhd2DKi/view?usp=sharing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org