You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/01/08 21:09:57 UTC
[GitHub] [hive] vyaslav commented on pull request #1742: HIVE-24484: Upgrade Hadoop to 3.3.0
vyaslav commented on pull request #1742:
URL: https://github.com/apache/hive/pull/1742#issuecomment-756999283
> @wangyum Stuck.
>
> There are two big issues here:
>
> 1. Hive integration tests fire up Druid, Kafka, HDFS, LLAP, etc. all in the same JVM and their 3rd party dependencies are all over the place. Using a higher version of a dependency breaks one product, but using a lower version breaks the other. To make this work well, there probably needs to be a way to launch each service in their own JVM class loader. In lieu of that, I've been trying to move the ball closer to the goal post and getting dependencies closer together.
>
> [apache/druid#10683](https://github.com/apache/druid/pull/10683)
> [HIVE-24542](https://issues.apache.org/jira/browse/HIVE-24542)
>
> 1. In HDFS 3.3.0, Hadoop team introduced `ProtobufRpcEngine2` in addition to `ProtobufRpcEngine` (sigh). Some of the Hive LLAP stuff is using this Hadoop Protobuf RPC engine (`ProtobufRpcEngine`). There's some `static` logic in the protocol engines that prohibits loading both RPC engines into the same JVM at the same time, I'm not sure why. HDFS was migrated to `ProtobufRpcEngine2`. So, again, in the integration tests, when the HDFS mini cluster is loaded, version 2 of the RPC engine is loaded into the JVM. When LLAP is later loaded, it fails to start because version 1 cannot be registered at the same time.
Regarding 1st, I faced the same issues in my PR for upgrade to 3.1.3 - https://github.com/apache/hive/pull/1638
But, regarding 2nd I'am curious if it would be hard to replace `ProtobufRpcEngine` with `ProtobufRpcEngine2` in Hive itself. As I understand they have upgraded from PB2 to PB3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org