Testing Hive 4.0.0-SNAPSHOT with Hadoop 3.2.1 and Spark 3.0.0-preview2

[Jagat Singh <ja...@gmail.com>]

Hello everyone,

I was playing with the latest source code of Hive, my goal is to make
Hadoop, Hive, Spark to work with the latest version of each other.

Locally, I made and ran full tests to make it run with

Hadoop 3.2.1
Spark 3.0.0-preview2
Tez 0.9.2

I ran the following maven command to ensure tests run successfully and did
any changes required.

mvn clean package -Pdist

1)
I was just wondering I can create a Jira to share these changes? I read the
contributors guide here
https://cwiki.apache.org/confluence/display/Hive/HowToContribute and it
says to ask on this mailing list before creating any Jira. If yes, should I
create separate Jira for Hadoop 3.2.1, Spark 3.0.0-preview2, Tez 0.9.2
changes?

2)
How should I do further testing to ensure things are working correctly? I
am trying to look at itests as well, for example below, but it does not
give any meaningful results. Maybe I am doing something wrong?

mvn test -q -Pitests -Dtest=TestSparkCliDriver

Thanks for reading.

Regards,

Jagat Singh

Re: Testing Hive 4.0.0-SNAPSHOT with Hadoop 3.2.1 and Spark 3.0.0-preview2

[David Lavati <dl...@cloudera.com.INVALID>]

Hi Jagat,

Thanks for your initiative. Here's my two cents:

For the Hadoop bump feel free to use/assign to yourself one of the existing
jiras which are inactive:
https://issues.apache.org/jira/browse/HIVE-22307
This was blocked by guava though, so once you open a PR, the tests will
reveal whether your solution takes care of it.

I don't see a Tez upgrade ticket yet and don't know whether there are any
blockers for it, so if you're confident with it, you could go ahead and
create a Jira+Github PR.

As for Spark, part of the community recently expressed their opinion to
remove Hive on Spark support:
https://mail-archives.apache.org/mod_mbox/hive-dev/202006.mbox/browser

Cheers,
David


On Sun, Jun 14, 2020 at 11:24 AM Jagat Singh <ja...@gmail.com> wrote:

> Hello everyone,
>
> I was playing with the latest source code of Hive, my goal is to make
> Hadoop, Hive, Spark to work with the latest version of each other.
>
> Locally, I made and ran full tests to make it run with
>
> Hadoop 3.2.1
> Spark 3.0.0-preview2
> Tez 0.9.2
>
> I ran the following maven command to ensure tests run successfully and did
> any changes required.
>
> mvn clean package -Pdist
>
> 1)
> I was just wondering I can create a Jira to share these changes? I read the
> contributors guide here
> https://cwiki.apache.org/confluence/display/Hive/HowToContribute and it
> says to ask on this mailing list before creating any Jira. If yes, should I
> create separate Jira for Hadoop 3.2.1, Spark 3.0.0-preview2, Tez 0.9.2
> changes?
>
> 2)
> How should I do further testing to ensure things are working correctly? I
> am trying to look at itests as well, for example below, but it does not
> give any meaningful results. Maybe I am doing something wrong?
>
> mvn test -q -Pitests -Dtest=TestSparkCliDriver
>
> Thanks for reading.
>
> Regards,
>
> Jagat Singh
>