[GitHub] [hive] belugabehr commented on pull request #1742: HIVE-24484: Upgrade Hadoop to 3.3.0

[GitBox <gi...@apache.org>]

belugabehr commented on pull request #1742:
URL: https://github.com/apache/hive/pull/1742#issuecomment-751765866


   @wangyum Stuck.
   
   There are two big issues here:
   
   1. Hive integration tests fire up Druid, Kafka, HDFS, LLAP, etc. all in the same JVM and their 3rd party dependencies are all over the place. Using a higher version of a dependency breaks one product, but using a lower version breaks the other.  To make this work well, there probably needs to be a way to launch each service in their own JVM class loader.  In lieu of that, I've been trying to move the ball closer to the goal post and getting dependencies closer together.
   
   https://github.com/apache/druid/pull/10683
   HIVE-24542
   
   
   2. In HDFS 3.3.0, Hadoop team introduced `ProtobufRpcEngine2` in addition to `ProtobufRpcEngine` (sigh).  Some of the Hive LLAP stuff is using this Hadoop Protobuf RPC engine (`ProtobufRpcEngine`).  There's some `static` logic in the protocol engines that prohibits loading both RPC engines into the same JVM at the same time, I'm not sure why.  HDFS was migrated to `ProtobufRpcEngine2`.  So, again, in the integration tests, when the HDFS mini cluster is loaded, version 2 of the RPC engine is loaded into the JVM.  When LLAP is later loaded, it fails to start because version 1 cannot be registered at the same time.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org