[discuss] jdk8 support

[Mohit Sabharwal <mo...@cloudera.com>]

Hi folks,

Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
In order to support Java 8, we fixed quite a bunch of tests in
HIVE-8607 (*) early last year. However, since our pre-commit tests run
on JDK7 only, any JDK8 test failures are getting ignored. As a result,
the count has crept up
<http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/>
back
from zero to 125.

Some options to address this:

(1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
and 2.x: This will further slow down the pre-commit run.

(2) Alternate pre-commit test runs between JDK7 and JDK8 (in
both 1.x and 2.x): It's a cheap hack. More messy, though failures
won't get ignored.

(3) Drop support for JDK7 in Hive 2.x, i.e. build and test Hive 2.x on
JDK8 only. For Hive 1.x, continue running JDK7 pre-commit tests.
The pros are:
- Simple test matrix.
- No need to slow down test run or maintain version specific golden files
(**).
- 2.x looks like the logical place to move to JDK8.
- Users transitioning to JDK8 for all other services do not have to
maintain
multiple java versions on the cluster.

Option (3) looks most attractive to me.

Moving to JDK8 also lines us up better for Java 9, which is on the
horizon (Oracle will end public updates for Java 8 in Sep 2017)

Around 100 of the latest crop of failures are due to one cause
(HIVE-13409).
I can take a pass at triaging the rest if there is consensus around what
to do overall.

Thanks,
Mohit

(*) Most test failures are due to JDK8 using a different hash function for
HashMap  compared to JDK7. This results in (mostly benign, but hard
to debug) ordering differences in q-file output related to serialization
order of map entries, numbering of stages in query plan, etc.

(**) In some cases, hash function related ordering differences originate
in external libraries like Avro, antlr, json ,Thrift's map, etc. for which
code
changes are either more involved or led to more test failures.  To address
this,
we added support for version specific golden files (HIVE-9109). Currently,
there
are ~40 golden files with ".java1.8.out" extension.

Re: [discuss] jdk8 support

[Gopal Vijayaraghavan <go...@apache.org>]

+1 from me, it's been more than a year since JDK7 EOL.

This needs a separate voting thread away from [discuss].

Cheers,
Gopal

On 6/4/16, 11:14 AM, "Lars Francke" <la...@gmail.com> wrote:

>+1 for option 3 to drop Java 7 and start using Java8 features in code as
>well
>
>On Fri, Jun 3, 2016 at 9:38 PM, Reuben Kuhnert
><re...@cloudera.com>
>wrote:
>
>> +1 Drop Java7 support, start using JDK8 features.
>>
>> On Fri, Jun 3, 2016 at 12:39 PM, Prasanth Jayachandran <
>> pjayachandran@hortonworks.com> wrote:
>>
>> > +1 for using jdk8 as minimum required version.
>> >
>> > Thanks
>> > Prasanth
>> >
>> >
>> >
>> >
>> > On Fri, Jun 3, 2016 at 10:21 AM -0700, "Siddharth Seth" <
>> sseth@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> > +1. Drop Java7 support, and start using JDK8 features.
>> >
>> > On Fri, Jun 3, 2016 at 8:13 AM, Ashutosh Chauhan
>><ha...@apache.org>
>> > wrote:
>> >
>> > > What I meant was that we start compiling using jdk8 as well. That
>>will
>> > > allow devs to use jdk8 only features (lambda functions etc.)
>> > >
>> > > On Fri, Jun 3, 2016 at 8:10 AM, Sergio Pena
>><se...@cloudera.com>
>> > > wrote:
>> > >
>> > > > Hey Ashutosh,
>> > > >
>> > > > I switched to JDK8 in master last weekend. Jenkins has been
>>running
>> all
>> > > > tests in Java8 successfully for a week now.
>> > > > There are still a few tests we need to fix, but so far is looking
>> good.
>> > > >
>> > > > Sergio
>> > > >
>> > > > On Fri, Jun 3, 2016 at 12:02 AM, Ashutosh Chauhan <
>> > hashutosh@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Now that branch-2.1 has been cut, I think its an opportune time
>>to
>> > drop
>> > > > > support of jdk7 altogether in master. That is we compile using
>>jdk8
>> > as
>> > > > > well. What do others think?
>> > > > >
>> > > > > Ashutosh
>> > > > >
>> > > > > On Fri, May 27, 2016 at 7:41 PM, Sergio Pena <
>> > sergio.pena@cloudera.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > I did the change to JDK8 on Ptest now.
>> > > > > >
>> > > > > > Please let me tknow if there are issues with Java8.
>> > > > > >
>> > > > > > - Sergio
>> > > > > >
>> > > > > > On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <
>> > > sergio.pena@cloudera.com
>> > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Thanks Mohit.
>> > > > > > >
>> > > > > > > I will plan to do the JDK8 change on Jenkins today EOD and
>> > monitor
>> > > > any
>> > > > > > > issues through the weekend.
>> > > > > > >
>> > > > > > > - Sergio
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <
>> > > mohit@cloudera.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > >> Update on moving Hive2 tests to JDK8: I've addressed almost
>> all
>> > > test
>> > > > > > >> failures in HIVE-13547 on java8 branch. There is one
>>remaining
>> > > > > > >> open item (HIVE-13834) which is currently assigned. Given
>> > current
>> > > > > > >> state of flaky test runs, there might be few more.
>> > > > > > >>
>> > > > > > >> I will work with Sergio to merge the test fixes to master
>>and
>> > > switch
>> > > > > > >> the Hive2 pre-commit job to use JDK8, hopefully sometime
>> > tomorrow.
>> > > > > > >>
>> > > > > > >> After Hive2 tests switch, if your patch sees ordering
>>related
>> > test
>> > > > > > >> failures in pre-commit run, it's likely JDK version related
>> and
>> > > > you'll
>> > > > > > >> need to build & re-run the test using JDK8. Number of such
>> tests
>> > > > > > >> should be relatively small.
>> > > > > > >>
>> > > > > > >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <
>> > > > mohit@cloudera.com
>> > > > > >
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >> > Created HIVE-13547 to track switching 2x tests to JDK8.
>> > > > > > >> >
>> > > > > > >> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <
>> > > > > > sergio.pena@cloudera.com
>> > > > > > >> >
>> > > > > > >> > wrote:
>> > > > > > >> >
>> > > > > > >> >> I agree with such change as JDK7 is not longer
>>supported.
>> > > > > > >> >>
>> > > > > > >> >> Changes on Jenkins and Hive PTest shouldn't be hard. We
>> just
>> > > need
>> > > > > to
>> > > > > > >> >> replace the path from java7 to java8. But I think we
>>should
>> > fix
>> > > > all
>> > > > > > >> JDK8
>> > > > > > >> >> issues or most of them before doing the change or we
>>will
>> end
>> > > up
>> > > > > > >> having a
>> > > > > > >> >> lot of failures on all JIRAs running pre-commit tests.
>> > > > > > >> >>
>> > > > > > >> >> +1 with the change.
>> > > > > > >> >>
>> > > > > > >> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <
>> > > > sseth@apache.org>
>> > > > > > >> wrote:
>> > > > > > >> >>
>> > > > > > >> >> > Option 3 sounds good. I'd ideally like to make JDK8
>>the
>> > > minimum
>> > > > > > >> >> requirement
>> > > > > > >> >> > soon as well.
>> > > > > > >> >> >
>> > > > > > >> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <
>> > > > szehon@cloudera.com>
>> > > > > > >> wrote:
>> > > > > > >> >> >
>> > > > > > >> >> > > Sounds like a good plan, +1
>> > > > > > >> >> > >
>> > > > > > >> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
>> > > > > > >> mohit@cloudera.com>
>> > > > > > >> >> > > wrote:
>> > > > > > >> >> > >
>> > > > > > >> >> > > > Thanks, Ashutosh. Makes sense to keep the source
>>and
>> > > target
>> > > > > as
>> > > > > > >> 1.7
>> > > > > > >> >> > since
>> > > > > > >> >> > > > we're not using any JDK8 specific features yet.
>>So,
>> > > option
>> > > > > (3)
>> > > > > > >> >> > > essentially
>> > > > > > >> >> > > > just means using JDK8 exclusively to build & test
>> > Hive2.
>> > > > > > >> >> > > >
>> > > > > > >> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan
>><
>> > > > > > >> >> > hashutosh@apache.org>
>> > > > > > >> >> > > > wrote:
>> > > > > > >> >> > > >
>> > > > > > >> >> > > > > Hi Mohit,
>> > > > > > >> >> > > > >
>> > > > > > >> >> > > > > I also think option 3 makes sense. We should
>>strive
>> > to
>> > > > keep
>> > > > > > >> test
>> > > > > > >> >> > matrix
>> > > > > > >> >> > > > > small so that we can do fast QA runs for dev
>> patches.
>> > > > > > >> >> > > > > We can just use jdk7 to build & test Hive1 and
>>jdk8
>> > to
>> > > > > build
>> > > > > > &
>> > > > > > >> >> test
>> > > > > > >> >> > > > Hive2.
>> > > > > > >> >> > > > > However, I am not sure of explicitly dropping
>> support
>> > > > > > >> altogether
>> > > > > > >> >> for
>> > > > > > >> >> > > jdk7
>> > > > > > >> >> > > > > on Hive2. We should make sure that in pom.xml
>>java
>> > > > source &
>> > > > > > >> target
>> > > > > > >> >> > > > > compatibility is still 1.7 (which already is the
>> case
>> > > > > > >> currently)
>> > > > > > >> >> so
>> > > > > > >> >> > > that
>> > > > > > >> >> > > > > Hive2 is still compatible with jdk7. Unit tests
>>as
>> I
>> > > said
>> > > > > we
>> > > > > > >> can
>> > > > > > >> >> run
>> > > > > > >> >> > on
>> > > > > > >> >> > > > > jdk8.
>> > > > > > >> >> > > > >
>> > > > > > >> >> > > > > Thanks,
>> > > > > > >> >> > > > > Ashutosh
>> > > > > > >> >> > > > >
>> > > > > > >> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal
>><
>> > > > > > >> >> mohit@cloudera.com>
>> > > > > > >> >> > > > > wrote:
>> > > > > > >> >> > > > >
>> > > > > > >> >> > > > > > Hi folks,
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > Oracle EOL'ed (ended public updates) for Java
>>7
>> in
>> > > > April
>> > > > > > >> 2015.
>> > > > > > >> >> > > > > > In order to support Java 8, we fixed quite a
>> bunch
>> > of
>> > > > > tests
>> > > > > > >> in
>> > > > > > >> >> > > > > > HIVE-8607 (*) early last year. However, since
>>our
>> > > > > > pre-commit
>> > > > > > >> >> tests
>> > > > > > >> >> > > run
>> > > > > > >> >> > > > > > on JDK7 only, any JDK8 test failures are
>>getting
>> > > > ignored.
>> > > > > > As
>> > > > > > >> a
>> > > > > > >> >> > > result,
>> > > > > > >> >> > > > > > the count has crept up
>> > > > > > >> >> > > > > > <
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > >
>> > > > > > >> >> > > >
>> > > > > > >> >> > >
>> > > > > > >> >> >
>> > > > > > >> >>
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> 
>>http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-
>>JAVA8/
>> > > > > > >> >> > > > > > >
>> > > > > > >> >> > > > > > back
>> > > > > > >> >> > > > > > from zero to 125.
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > Some options to address this:
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > (1) Run pre-commit tests on both JDK7 and
>>JDK8,
>> in
>> > > both
>> > > > > 1.x
>> > > > > > >> >> > > > > > and 2.x: This will further slow down the
>> pre-commit
>> > > > run.
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > (2) Alternate pre-commit test runs between
>>JDK7
>> and
>> > > > JDK8
>> > > > > > (in
>> > > > > > >> >> > > > > > both 1.x and 2.x): It's a cheap hack. More
>>messy,
>> > > > though
>> > > > > > >> >> failures
>> > > > > > >> >> > > > > > won't get ignored.
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e.
>>build
>> > and
>> > > > > test
>> > > > > > >> Hive
>> > > > > > >> >> 2.x
>> > > > > > >> >> > > on
>> > > > > > >> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7
>> > > > pre-commit
>> > > > > > >> tests.
>> > > > > > >> >> > > > > > The pros are:
>> > > > > > >> >> > > > > > - Simple test matrix.
>> > > > > > >> >> > > > > > - No need to slow down test run or maintain
>> version
>> > > > > > specific
>> > > > > > >> >> golden
>> > > > > > >> >> > > > files
>> > > > > > >> >> > > > > > (**).
>> > > > > > >> >> > > > > > - 2.x looks like the logical place to move to
>> JDK8.
>> > > > > > >> >> > > > > > - Users transitioning to JDK8 for all other
>> > services
>> > > do
>> > > > > not
>> > > > > > >> >> have to
>> > > > > > >> >> > > > > > maintain
>> > > > > > >> >> > > > > > multiple java versions on the cluster.
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > Option (3) looks most attractive to me.
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > Moving to JDK8 also lines us up better for
>>Java
>> 9,
>> > > > which
>> > > > > is
>> > > > > > >> on
>> > > > > > >> >> the
>> > > > > > >> >> > > > > > horizon (Oracle will end public updates for
>>Java
>> 8
>> > in
>> > > > Sep
>> > > > > > >> 2017)
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > Around 100 of the latest crop of failures are
>>due
>> > to
>> > > > one
>> > > > > > >> cause
>> > > > > > >> >> > > > > > (HIVE-13409).
>> > > > > > >> >> > > > > > I can take a pass at triaging the rest if
>>there
>> is
>> > > > > > consensus
>> > > > > > >> >> around
>> > > > > > >> >> > > > what
>> > > > > > >> >> > > > > > to do overall.
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > Thanks,
>> > > > > > >> >> > > > > > Mohit
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > (*) Most test failures are due to JDK8 using a
>> > > > different
>> > > > > > hash
>> > > > > > >> >> > > function
>> > > > > > >> >> > > > > for
>> > > > > > >> >> > > > > > HashMap  compared to JDK7. This results in
>> (mostly
>> > > > > benign,
>> > > > > > >> but
>> > > > > > >> >> hard
>> > > > > > >> >> > > > > > to debug) ordering differences in q-file
>>output
>> > > related
>> > > > > to
>> > > > > > >> >> > > > serialization
>> > > > > > >> >> > > > > > order of map entries, numbering of stages in
>> query
>> > > > plan,
>> > > > > > etc.
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > > > (**) In some cases, hash function related
>> ordering
>> > > > > > >> differences
>> > > > > > >> >> > > > originate
>> > > > > > >> >> > > > > > in external libraries like Avro, antlr, json
>> > > ,Thrift's
>> > > > > map,
>> > > > > > >> etc.
>> > > > > > >> >> > for
>> > > > > > >> >> > > > > which
>> > > > > > >> >> > > > > > code
>> > > > > > >> >> > > > > > changes are either more involved or led to
>>more
>> > test
>> > > > > > >> failures.
>> > > > > > >> >> To
>> > > > > > >> >> > > > > address
>> > > > > > >> >> > > > > > this,
>> > > > > > >> >> > > > > > we added support for version specific golden
>> files
>> > > > > > >> (HIVE-9109).
>> > > > > > >> >> > > > > Currently,
>> > > > > > >> >> > > > > > there
>> > > > > > >> >> > > > > > are ~40 golden files with ".java1.8.out"
>> extension.
>> > > > > > >> >> > > > > >
>> > > > > > >> >> > > > >
>> > > > > > >> >> > > >
>> > > > > > >> >> > >
>> > > > > > >> >> >
>> > > > > > >> >>
>> > > > > > >> >
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: [discuss] jdk8 support

[Lars Francke <la...@gmail.com>]

+1 for option 3 to drop Java 7 and start using Java8 features in code as
well

On Fri, Jun 3, 2016 at 9:38 PM, Reuben Kuhnert <re...@cloudera.com>
wrote:

> +1 Drop Java7 support, start using JDK8 features.
>
> On Fri, Jun 3, 2016 at 12:39 PM, Prasanth Jayachandran <
> pjayachandran@hortonworks.com> wrote:
>
> > +1 for using jdk8 as minimum required version.
> >
> > Thanks
> > Prasanth
> >
> >
> >
> >
> > On Fri, Jun 3, 2016 at 10:21 AM -0700, "Siddharth Seth" <
> sseth@apache.org
> > <ma...@apache.org>> wrote:
> >
> > +1. Drop Java7 support, and start using JDK8 features.
> >
> > On Fri, Jun 3, 2016 at 8:13 AM, Ashutosh Chauhan <ha...@apache.org>
> > wrote:
> >
> > > What I meant was that we start compiling using jdk8 as well. That will
> > > allow devs to use jdk8 only features (lambda functions etc.)
> > >
> > > On Fri, Jun 3, 2016 at 8:10 AM, Sergio Pena <se...@cloudera.com>
> > > wrote:
> > >
> > > > Hey Ashutosh,
> > > >
> > > > I switched to JDK8 in master last weekend. Jenkins has been running
> all
> > > > tests in Java8 successfully for a week now.
> > > > There are still a few tests we need to fix, but so far is looking
> good.
> > > >
> > > > Sergio
> > > >
> > > > On Fri, Jun 3, 2016 at 12:02 AM, Ashutosh Chauhan <
> > hashutosh@apache.org>
> > > > wrote:
> > > >
> > > > > Now that branch-2.1 has been cut, I think its an opportune time to
> > drop
> > > > > support of jdk7 altogether in master. That is we compile using jdk8
> > as
> > > > > well. What do others think?
> > > > >
> > > > > Ashutosh
> > > > >
> > > > > On Fri, May 27, 2016 at 7:41 PM, Sergio Pena <
> > sergio.pena@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > I did the change to JDK8 on Ptest now.
> > > > > >
> > > > > > Please let me tknow if there are issues with Java8.
> > > > > >
> > > > > > - Sergio
> > > > > >
> > > > > > On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <
> > > sergio.pena@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Mohit.
> > > > > > >
> > > > > > > I will plan to do the JDK8 change on Jenkins today EOD and
> > monitor
> > > > any
> > > > > > > issues through the weekend.
> > > > > > >
> > > > > > > - Sergio
> > > > > > >
> > > > > > >
> > > > > > > On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <
> > > mohit@cloudera.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Update on moving Hive2 tests to JDK8: I've addressed almost
> all
> > > test
> > > > > > >> failures in HIVE-13547 on java8 branch. There is one remaining
> > > > > > >> open item (HIVE-13834) which is currently assigned. Given
> > current
> > > > > > >> state of flaky test runs, there might be few more.
> > > > > > >>
> > > > > > >> I will work with Sergio to merge the test fixes to master and
> > > switch
> > > > > > >> the Hive2 pre-commit job to use JDK8, hopefully sometime
> > tomorrow.
> > > > > > >>
> > > > > > >> After Hive2 tests switch, if your patch sees ordering related
> > test
> > > > > > >> failures in pre-commit run, it's likely JDK version related
> and
> > > > you'll
> > > > > > >> need to build & re-run the test using JDK8. Number of such
> tests
> > > > > > >> should be relatively small.
> > > > > > >>
> > > > > > >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <
> > > > mohit@cloudera.com
> > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Created HIVE-13547 to track switching 2x tests to JDK8.
> > > > > > >> >
> > > > > > >> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <
> > > > > > sergio.pena@cloudera.com
> > > > > > >> >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> >> I agree with such change as JDK7 is not longer supported.
> > > > > > >> >>
> > > > > > >> >> Changes on Jenkins and Hive PTest shouldn't be hard. We
> just
> > > need
> > > > > to
> > > > > > >> >> replace the path from java7 to java8. But I think we should
> > fix
> > > > all
> > > > > > >> JDK8
> > > > > > >> >> issues or most of them before doing the change or we will
> end
> > > up
> > > > > > >> having a
> > > > > > >> >> lot of failures on all JIRAs running pre-commit tests.
> > > > > > >> >>
> > > > > > >> >> +1 with the change.
> > > > > > >> >>
> > > > > > >> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <
> > > > sseth@apache.org>
> > > > > > >> wrote:
> > > > > > >> >>
> > > > > > >> >> > Option 3 sounds good. I'd ideally like to make JDK8 the
> > > minimum
> > > > > > >> >> requirement
> > > > > > >> >> > soon as well.
> > > > > > >> >> >
> > > > > > >> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <
> > > > szehon@cloudera.com>
> > > > > > >> wrote:
> > > > > > >> >> >
> > > > > > >> >> > > Sounds like a good plan, +1
> > > > > > >> >> > >
> > > > > > >> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
> > > > > > >> mohit@cloudera.com>
> > > > > > >> >> > > wrote:
> > > > > > >> >> > >
> > > > > > >> >> > > > Thanks, Ashutosh. Makes sense to keep the source and
> > > target
> > > > > as
> > > > > > >> 1.7
> > > > > > >> >> > since
> > > > > > >> >> > > > we're not using any JDK8 specific features yet. So,
> > > option
> > > > > (3)
> > > > > > >> >> > > essentially
> > > > > > >> >> > > > just means using JDK8 exclusively to build & test
> > Hive2.
> > > > > > >> >> > > >
> > > > > > >> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> > > > > > >> >> > hashutosh@apache.org>
> > > > > > >> >> > > > wrote:
> > > > > > >> >> > > >
> > > > > > >> >> > > > > Hi Mohit,
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > I also think option 3 makes sense. We should strive
> > to
> > > > keep
> > > > > > >> test
> > > > > > >> >> > matrix
> > > > > > >> >> > > > > small so that we can do fast QA runs for dev
> patches.
> > > > > > >> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8
> > to
> > > > > build
> > > > > > &
> > > > > > >> >> test
> > > > > > >> >> > > > Hive2.
> > > > > > >> >> > > > > However, I am not sure of explicitly dropping
> support
> > > > > > >> altogether
> > > > > > >> >> for
> > > > > > >> >> > > jdk7
> > > > > > >> >> > > > > on Hive2. We should make sure that in pom.xml java
> > > > source &
> > > > > > >> target
> > > > > > >> >> > > > > compatibility is still 1.7 (which already is the
> case
> > > > > > >> currently)
> > > > > > >> >> so
> > > > > > >> >> > > that
> > > > > > >> >> > > > > Hive2 is still compatible with jdk7. Unit tests as
> I
> > > said
> > > > > we
> > > > > > >> can
> > > > > > >> >> run
> > > > > > >> >> > on
> > > > > > >> >> > > > > jdk8.
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > Thanks,
> > > > > > >> >> > > > > Ashutosh
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
> > > > > > >> >> mohit@cloudera.com>
> > > > > > >> >> > > > > wrote:
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > > Hi folks,
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7
> in
> > > > April
> > > > > > >> 2015.
> > > > > > >> >> > > > > > In order to support Java 8, we fixed quite a
> bunch
> > of
> > > > > tests
> > > > > > >> in
> > > > > > >> >> > > > > > HIVE-8607 (*) early last year. However, since our
> > > > > > pre-commit
> > > > > > >> >> tests
> > > > > > >> >> > > run
> > > > > > >> >> > > > > > on JDK7 only, any JDK8 test failures are getting
> > > > ignored.
> > > > > > As
> > > > > > >> a
> > > > > > >> >> > > result,
> > > > > > >> >> > > > > > the count has crept up
> > > > > > >> >> > > > > > <
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > >
> > > > > > >> >> > > >
> > > > > > >> >> > >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > > > > > >> >> > > > > > >
> > > > > > >> >> > > > > > back
> > > > > > >> >> > > > > > from zero to 125.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > Some options to address this:
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8,
> in
> > > both
> > > > > 1.x
> > > > > > >> >> > > > > > and 2.x: This will further slow down the
> pre-commit
> > > > run.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > (2) Alternate pre-commit test runs between JDK7
> and
> > > > JDK8
> > > > > > (in
> > > > > > >> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy,
> > > > though
> > > > > > >> >> failures
> > > > > > >> >> > > > > > won't get ignored.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build
> > and
> > > > > test
> > > > > > >> Hive
> > > > > > >> >> 2.x
> > > > > > >> >> > > on
> > > > > > >> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7
> > > > pre-commit
> > > > > > >> tests.
> > > > > > >> >> > > > > > The pros are:
> > > > > > >> >> > > > > > - Simple test matrix.
> > > > > > >> >> > > > > > - No need to slow down test run or maintain
> version
> > > > > > specific
> > > > > > >> >> golden
> > > > > > >> >> > > > files
> > > > > > >> >> > > > > > (**).
> > > > > > >> >> > > > > > - 2.x looks like the logical place to move to
> JDK8.
> > > > > > >> >> > > > > > - Users transitioning to JDK8 for all other
> > services
> > > do
> > > > > not
> > > > > > >> >> have to
> > > > > > >> >> > > > > > maintain
> > > > > > >> >> > > > > > multiple java versions on the cluster.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > Option (3) looks most attractive to me.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > Moving to JDK8 also lines us up better for Java
> 9,
> > > > which
> > > > > is
> > > > > > >> on
> > > > > > >> >> the
> > > > > > >> >> > > > > > horizon (Oracle will end public updates for Java
> 8
> > in
> > > > Sep
> > > > > > >> 2017)
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > Around 100 of the latest crop of failures are due
> > to
> > > > one
> > > > > > >> cause
> > > > > > >> >> > > > > > (HIVE-13409).
> > > > > > >> >> > > > > > I can take a pass at triaging the rest if there
> is
> > > > > > consensus
> > > > > > >> >> around
> > > > > > >> >> > > > what
> > > > > > >> >> > > > > > to do overall.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > Thanks,
> > > > > > >> >> > > > > > Mohit
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > (*) Most test failures are due to JDK8 using a
> > > > different
> > > > > > hash
> > > > > > >> >> > > function
> > > > > > >> >> > > > > for
> > > > > > >> >> > > > > > HashMap  compared to JDK7. This results in
> (mostly
> > > > > benign,
> > > > > > >> but
> > > > > > >> >> hard
> > > > > > >> >> > > > > > to debug) ordering differences in q-file output
> > > related
> > > > > to
> > > > > > >> >> > > > serialization
> > > > > > >> >> > > > > > order of map entries, numbering of stages in
> query
> > > > plan,
> > > > > > etc.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > (**) In some cases, hash function related
> ordering
> > > > > > >> differences
> > > > > > >> >> > > > originate
> > > > > > >> >> > > > > > in external libraries like Avro, antlr, json
> > > ,Thrift's
> > > > > map,
> > > > > > >> etc.
> > > > > > >> >> > for
> > > > > > >> >> > > > > which
> > > > > > >> >> > > > > > code
> > > > > > >> >> > > > > > changes are either more involved or led to more
> > test
> > > > > > >> failures.
> > > > > > >> >> To
> > > > > > >> >> > > > > address
> > > > > > >> >> > > > > > this,
> > > > > > >> >> > > > > > we added support for version specific golden
> files
> > > > > > >> (HIVE-9109).
> > > > > > >> >> > > > > Currently,
> > > > > > >> >> > > > > > there
> > > > > > >> >> > > > > > are ~40 golden files with ".java1.8.out"
> extension.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > >
> > > > > > >> >> > > >
> > > > > > >> >> > >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [discuss] jdk8 support

[Reuben Kuhnert <re...@cloudera.com>]

+1 Drop Java7 support, start using JDK8 features.

On Fri, Jun 3, 2016 at 12:39 PM, Prasanth Jayachandran <
pjayachandran@hortonworks.com> wrote:

> +1 for using jdk8 as minimum required version.
>
> Thanks
> Prasanth
>
>
>
>
> On Fri, Jun 3, 2016 at 10:21 AM -0700, "Siddharth Seth" <sseth@apache.org
> <ma...@apache.org>> wrote:
>
> +1. Drop Java7 support, and start using JDK8 features.
>
> On Fri, Jun 3, 2016 at 8:13 AM, Ashutosh Chauhan <ha...@apache.org>
> wrote:
>
> > What I meant was that we start compiling using jdk8 as well. That will
> > allow devs to use jdk8 only features (lambda functions etc.)
> >
> > On Fri, Jun 3, 2016 at 8:10 AM, Sergio Pena <se...@cloudera.com>
> > wrote:
> >
> > > Hey Ashutosh,
> > >
> > > I switched to JDK8 in master last weekend. Jenkins has been running all
> > > tests in Java8 successfully for a week now.
> > > There are still a few tests we need to fix, but so far is looking good.
> > >
> > > Sergio
> > >
> > > On Fri, Jun 3, 2016 at 12:02 AM, Ashutosh Chauhan <
> hashutosh@apache.org>
> > > wrote:
> > >
> > > > Now that branch-2.1 has been cut, I think its an opportune time to
> drop
> > > > support of jdk7 altogether in master. That is we compile using jdk8
> as
> > > > well. What do others think?
> > > >
> > > > Ashutosh
> > > >
> > > > On Fri, May 27, 2016 at 7:41 PM, Sergio Pena <
> sergio.pena@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > I did the change to JDK8 on Ptest now.
> > > > >
> > > > > Please let me tknow if there are issues with Java8.
> > > > >
> > > > > - Sergio
> > > > >
> > > > > On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <
> > sergio.pena@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Mohit.
> > > > > >
> > > > > > I will plan to do the JDK8 change on Jenkins today EOD and
> monitor
> > > any
> > > > > > issues through the weekend.
> > > > > >
> > > > > > - Sergio
> > > > > >
> > > > > >
> > > > > > On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <
> > mohit@cloudera.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Update on moving Hive2 tests to JDK8: I've addressed almost all
> > test
> > > > > >> failures in HIVE-13547 on java8 branch. There is one remaining
> > > > > >> open item (HIVE-13834) which is currently assigned. Given
> current
> > > > > >> state of flaky test runs, there might be few more.
> > > > > >>
> > > > > >> I will work with Sergio to merge the test fixes to master and
> > switch
> > > > > >> the Hive2 pre-commit job to use JDK8, hopefully sometime
> tomorrow.
> > > > > >>
> > > > > >> After Hive2 tests switch, if your patch sees ordering related
> test
> > > > > >> failures in pre-commit run, it's likely JDK version related and
> > > you'll
> > > > > >> need to build & re-run the test using JDK8. Number of such tests
> > > > > >> should be relatively small.
> > > > > >>
> > > > > >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <
> > > mohit@cloudera.com
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Created HIVE-13547 to track switching 2x tests to JDK8.
> > > > > >> >
> > > > > >> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <
> > > > > sergio.pena@cloudera.com
> > > > > >> >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> I agree with such change as JDK7 is not longer supported.
> > > > > >> >>
> > > > > >> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just
> > need
> > > > to
> > > > > >> >> replace the path from java7 to java8. But I think we should
> fix
> > > all
> > > > > >> JDK8
> > > > > >> >> issues or most of them before doing the change or we will end
> > up
> > > > > >> having a
> > > > > >> >> lot of failures on all JIRAs running pre-commit tests.
> > > > > >> >>
> > > > > >> >> +1 with the change.
> > > > > >> >>
> > > > > >> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <
> > > sseth@apache.org>
> > > > > >> wrote:
> > > > > >> >>
> > > > > >> >> > Option 3 sounds good. I'd ideally like to make JDK8 the
> > minimum
> > > > > >> >> requirement
> > > > > >> >> > soon as well.
> > > > > >> >> >
> > > > > >> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <
> > > szehon@cloudera.com>
> > > > > >> wrote:
> > > > > >> >> >
> > > > > >> >> > > Sounds like a good plan, +1
> > > > > >> >> > >
> > > > > >> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
> > > > > >> mohit@cloudera.com>
> > > > > >> >> > > wrote:
> > > > > >> >> > >
> > > > > >> >> > > > Thanks, Ashutosh. Makes sense to keep the source and
> > target
> > > > as
> > > > > >> 1.7
> > > > > >> >> > since
> > > > > >> >> > > > we're not using any JDK8 specific features yet. So,
> > option
> > > > (3)
> > > > > >> >> > > essentially
> > > > > >> >> > > > just means using JDK8 exclusively to build & test
> Hive2.
> > > > > >> >> > > >
> > > > > >> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> > > > > >> >> > hashutosh@apache.org>
> > > > > >> >> > > > wrote:
> > > > > >> >> > > >
> > > > > >> >> > > > > Hi Mohit,
> > > > > >> >> > > > >
> > > > > >> >> > > > > I also think option 3 makes sense. We should strive
> to
> > > keep
> > > > > >> test
> > > > > >> >> > matrix
> > > > > >> >> > > > > small so that we can do fast QA runs for dev patches.
> > > > > >> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8
> to
> > > > build
> > > > > &
> > > > > >> >> test
> > > > > >> >> > > > Hive2.
> > > > > >> >> > > > > However, I am not sure of explicitly dropping support
> > > > > >> altogether
> > > > > >> >> for
> > > > > >> >> > > jdk7
> > > > > >> >> > > > > on Hive2. We should make sure that in pom.xml java
> > > source &
> > > > > >> target
> > > > > >> >> > > > > compatibility is still 1.7 (which already is the case
> > > > > >> currently)
> > > > > >> >> so
> > > > > >> >> > > that
> > > > > >> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I
> > said
> > > > we
> > > > > >> can
> > > > > >> >> run
> > > > > >> >> > on
> > > > > >> >> > > > > jdk8.
> > > > > >> >> > > > >
> > > > > >> >> > > > > Thanks,
> > > > > >> >> > > > > Ashutosh
> > > > > >> >> > > > >
> > > > > >> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
> > > > > >> >> mohit@cloudera.com>
> > > > > >> >> > > > > wrote:
> > > > > >> >> > > > >
> > > > > >> >> > > > > > Hi folks,
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in
> > > April
> > > > > >> 2015.
> > > > > >> >> > > > > > In order to support Java 8, we fixed quite a bunch
> of
> > > > tests
> > > > > >> in
> > > > > >> >> > > > > > HIVE-8607 (*) early last year. However, since our
> > > > > pre-commit
> > > > > >> >> tests
> > > > > >> >> > > run
> > > > > >> >> > > > > > on JDK7 only, any JDK8 test failures are getting
> > > ignored.
> > > > > As
> > > > > >> a
> > > > > >> >> > > result,
> > > > > >> >> > > > > > the count has crept up
> > > > > >> >> > > > > > <
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > back
> > > > > >> >> > > > > > from zero to 125.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Some options to address this:
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in
> > both
> > > > 1.x
> > > > > >> >> > > > > > and 2.x: This will further slow down the pre-commit
> > > run.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > (2) Alternate pre-commit test runs between JDK7 and
> > > JDK8
> > > > > (in
> > > > > >> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy,
> > > though
> > > > > >> >> failures
> > > > > >> >> > > > > > won't get ignored.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build
> and
> > > > test
> > > > > >> Hive
> > > > > >> >> 2.x
> > > > > >> >> > > on
> > > > > >> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7
> > > pre-commit
> > > > > >> tests.
> > > > > >> >> > > > > > The pros are:
> > > > > >> >> > > > > > - Simple test matrix.
> > > > > >> >> > > > > > - No need to slow down test run or maintain version
> > > > > specific
> > > > > >> >> golden
> > > > > >> >> > > > files
> > > > > >> >> > > > > > (**).
> > > > > >> >> > > > > > - 2.x looks like the logical place to move to JDK8.
> > > > > >> >> > > > > > - Users transitioning to JDK8 for all other
> services
> > do
> > > > not
> > > > > >> >> have to
> > > > > >> >> > > > > > maintain
> > > > > >> >> > > > > > multiple java versions on the cluster.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Option (3) looks most attractive to me.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Moving to JDK8 also lines us up better for Java 9,
> > > which
> > > > is
> > > > > >> on
> > > > > >> >> the
> > > > > >> >> > > > > > horizon (Oracle will end public updates for Java 8
> in
> > > Sep
> > > > > >> 2017)
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Around 100 of the latest crop of failures are due
> to
> > > one
> > > > > >> cause
> > > > > >> >> > > > > > (HIVE-13409).
> > > > > >> >> > > > > > I can take a pass at triaging the rest if there is
> > > > > consensus
> > > > > >> >> around
> > > > > >> >> > > > what
> > > > > >> >> > > > > > to do overall.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Thanks,
> > > > > >> >> > > > > > Mohit
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > (*) Most test failures are due to JDK8 using a
> > > different
> > > > > hash
> > > > > >> >> > > function
> > > > > >> >> > > > > for
> > > > > >> >> > > > > > HashMap  compared to JDK7. This results in (mostly
> > > > benign,
> > > > > >> but
> > > > > >> >> hard
> > > > > >> >> > > > > > to debug) ordering differences in q-file output
> > related
> > > > to
> > > > > >> >> > > > serialization
> > > > > >> >> > > > > > order of map entries, numbering of stages in query
> > > plan,
> > > > > etc.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > (**) In some cases, hash function related ordering
> > > > > >> differences
> > > > > >> >> > > > originate
> > > > > >> >> > > > > > in external libraries like Avro, antlr, json
> > ,Thrift's
> > > > map,
> > > > > >> etc.
> > > > > >> >> > for
> > > > > >> >> > > > > which
> > > > > >> >> > > > > > code
> > > > > >> >> > > > > > changes are either more involved or led to more
> test
> > > > > >> failures.
> > > > > >> >> To
> > > > > >> >> > > > > address
> > > > > >> >> > > > > > this,
> > > > > >> >> > > > > > we added support for version specific golden files
> > > > > >> (HIVE-9109).
> > > > > >> >> > > > > Currently,
> > > > > >> >> > > > > > there
> > > > > >> >> > > > > > are ~40 golden files with ".java1.8.out" extension.
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [discuss] jdk8 support

[Prasanth Jayachandran <pj...@hortonworks.com>]

+1 for using jdk8 as minimum required version.

Thanks
Prasanth




On Fri, Jun 3, 2016 at 10:21 AM -0700, "Siddharth Seth" <ss...@apache.org>> wrote:

+1. Drop Java7 support, and start using JDK8 features.

On Fri, Jun 3, 2016 at 8:13 AM, Ashutosh Chauhan <ha...@apache.org>
wrote:

> What I meant was that we start compiling using jdk8 as well. That will
> allow devs to use jdk8 only features (lambda functions etc.)
>
> On Fri, Jun 3, 2016 at 8:10 AM, Sergio Pena <se...@cloudera.com>
> wrote:
>
> > Hey Ashutosh,
> >
> > I switched to JDK8 in master last weekend. Jenkins has been running all
> > tests in Java8 successfully for a week now.
> > There are still a few tests we need to fix, but so far is looking good.
> >
> > Sergio
> >
> > On Fri, Jun 3, 2016 at 12:02 AM, Ashutosh Chauhan <ha...@apache.org>
> > wrote:
> >
> > > Now that branch-2.1 has been cut, I think its an opportune time to drop
> > > support of jdk7 altogether in master. That is we compile using jdk8 as
> > > well. What do others think?
> > >
> > > Ashutosh
> > >
> > > On Fri, May 27, 2016 at 7:41 PM, Sergio Pena <sergio.pena@cloudera.com
> >
> > > wrote:
> > >
> > > > I did the change to JDK8 on Ptest now.
> > > >
> > > > Please let me tknow if there are issues with Java8.
> > > >
> > > > - Sergio
> > > >
> > > > On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <
> sergio.pena@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Thanks Mohit.
> > > > >
> > > > > I will plan to do the JDK8 change on Jenkins today EOD and monitor
> > any
> > > > > issues through the weekend.
> > > > >
> > > > > - Sergio
> > > > >
> > > > >
> > > > > On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <
> mohit@cloudera.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Update on moving Hive2 tests to JDK8: I've addressed almost all
> test
> > > > >> failures in HIVE-13547 on java8 branch. There is one remaining
> > > > >> open item (HIVE-13834) which is currently assigned. Given current
> > > > >> state of flaky test runs, there might be few more.
> > > > >>
> > > > >> I will work with Sergio to merge the test fixes to master and
> switch
> > > > >> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
> > > > >>
> > > > >> After Hive2 tests switch, if your patch sees ordering related test
> > > > >> failures in pre-commit run, it's likely JDK version related and
> > you'll
> > > > >> need to build & re-run the test using JDK8. Number of such tests
> > > > >> should be relatively small.
> > > > >>
> > > > >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <
> > mohit@cloudera.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Created HIVE-13547 to track switching 2x tests to JDK8.
> > > > >> >
> > > > >> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <
> > > > sergio.pena@cloudera.com
> > > > >> >
> > > > >> > wrote:
> > > > >> >
> > > > >> >> I agree with such change as JDK7 is not longer supported.
> > > > >> >>
> > > > >> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just
> need
> > > to
> > > > >> >> replace the path from java7 to java8. But I think we should fix
> > all
> > > > >> JDK8
> > > > >> >> issues or most of them before doing the change or we will end
> up
> > > > >> having a
> > > > >> >> lot of failures on all JIRAs running pre-commit tests.
> > > > >> >>
> > > > >> >> +1 with the change.
> > > > >> >>
> > > > >> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <
> > sseth@apache.org>
> > > > >> wrote:
> > > > >> >>
> > > > >> >> > Option 3 sounds good. I'd ideally like to make JDK8 the
> minimum
> > > > >> >> requirement
> > > > >> >> > soon as well.
> > > > >> >> >
> > > > >> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <
> > szehon@cloudera.com>
> > > > >> wrote:
> > > > >> >> >
> > > > >> >> > > Sounds like a good plan, +1
> > > > >> >> > >
> > > > >> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
> > > > >> mohit@cloudera.com>
> > > > >> >> > > wrote:
> > > > >> >> > >
> > > > >> >> > > > Thanks, Ashutosh. Makes sense to keep the source and
> target
> > > as
> > > > >> 1.7
> > > > >> >> > since
> > > > >> >> > > > we're not using any JDK8 specific features yet. So,
> option
> > > (3)
> > > > >> >> > > essentially
> > > > >> >> > > > just means using JDK8 exclusively to build & test Hive2.
> > > > >> >> > > >
> > > > >> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> > > > >> >> > hashutosh@apache.org>
> > > > >> >> > > > wrote:
> > > > >> >> > > >
> > > > >> >> > > > > Hi Mohit,
> > > > >> >> > > > >
> > > > >> >> > > > > I also think option 3 makes sense. We should strive to
> > keep
> > > > >> test
> > > > >> >> > matrix
> > > > >> >> > > > > small so that we can do fast QA runs for dev patches.
> > > > >> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to
> > > build
> > > > &
> > > > >> >> test
> > > > >> >> > > > Hive2.
> > > > >> >> > > > > However, I am not sure of explicitly dropping support
> > > > >> altogether
> > > > >> >> for
> > > > >> >> > > jdk7
> > > > >> >> > > > > on Hive2. We should make sure that in pom.xml java
> > source &
> > > > >> target
> > > > >> >> > > > > compatibility is still 1.7 (which already is the case
> > > > >> currently)
> > > > >> >> so
> > > > >> >> > > that
> > > > >> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I
> said
> > > we
> > > > >> can
> > > > >> >> run
> > > > >> >> > on
> > > > >> >> > > > > jdk8.
> > > > >> >> > > > >
> > > > >> >> > > > > Thanks,
> > > > >> >> > > > > Ashutosh
> > > > >> >> > > > >
> > > > >> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
> > > > >> >> mohit@cloudera.com>
> > > > >> >> > > > > wrote:
> > > > >> >> > > > >
> > > > >> >> > > > > > Hi folks,
> > > > >> >> > > > > >
> > > > >> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in
> > April
> > > > >> 2015.
> > > > >> >> > > > > > In order to support Java 8, we fixed quite a bunch of
> > > tests
> > > > >> in
> > > > >> >> > > > > > HIVE-8607 (*) early last year. However, since our
> > > > pre-commit
> > > > >> >> tests
> > > > >> >> > > run
> > > > >> >> > > > > > on JDK7 only, any JDK8 test failures are getting
> > ignored.
> > > > As
> > > > >> a
> > > > >> >> > > result,
> > > > >> >> > > > > > the count has crept up
> > > > >> >> > > > > > <
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > > > >> >> > > > > > >
> > > > >> >> > > > > > back
> > > > >> >> > > > > > from zero to 125.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Some options to address this:
> > > > >> >> > > > > >
> > > > >> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in
> both
> > > 1.x
> > > > >> >> > > > > > and 2.x: This will further slow down the pre-commit
> > run.
> > > > >> >> > > > > >
> > > > >> >> > > > > > (2) Alternate pre-commit test runs between JDK7 and
> > JDK8
> > > > (in
> > > > >> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy,
> > though
> > > > >> >> failures
> > > > >> >> > > > > > won't get ignored.
> > > > >> >> > > > > >
> > > > >> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and
> > > test
> > > > >> Hive
> > > > >> >> 2.x
> > > > >> >> > > on
> > > > >> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7
> > pre-commit
> > > > >> tests.
> > > > >> >> > > > > > The pros are:
> > > > >> >> > > > > > - Simple test matrix.
> > > > >> >> > > > > > - No need to slow down test run or maintain version
> > > > specific
> > > > >> >> golden
> > > > >> >> > > > files
> > > > >> >> > > > > > (**).
> > > > >> >> > > > > > - 2.x looks like the logical place to move to JDK8.
> > > > >> >> > > > > > - Users transitioning to JDK8 for all other services
> do
> > > not
> > > > >> >> have to
> > > > >> >> > > > > > maintain
> > > > >> >> > > > > > multiple java versions on the cluster.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Option (3) looks most attractive to me.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Moving to JDK8 also lines us up better for Java 9,
> > which
> > > is
> > > > >> on
> > > > >> >> the
> > > > >> >> > > > > > horizon (Oracle will end public updates for Java 8 in
> > Sep
> > > > >> 2017)
> > > > >> >> > > > > >
> > > > >> >> > > > > > Around 100 of the latest crop of failures are due to
> > one
> > > > >> cause
> > > > >> >> > > > > > (HIVE-13409).
> > > > >> >> > > > > > I can take a pass at triaging the rest if there is
> > > > consensus
> > > > >> >> around
> > > > >> >> > > > what
> > > > >> >> > > > > > to do overall.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Thanks,
> > > > >> >> > > > > > Mohit
> > > > >> >> > > > > >
> > > > >> >> > > > > > (*) Most test failures are due to JDK8 using a
> > different
> > > > hash
> > > > >> >> > > function
> > > > >> >> > > > > for
> > > > >> >> > > > > > HashMap  compared to JDK7. This results in (mostly
> > > benign,
> > > > >> but
> > > > >> >> hard
> > > > >> >> > > > > > to debug) ordering differences in q-file output
> related
> > > to
> > > > >> >> > > > serialization
> > > > >> >> > > > > > order of map entries, numbering of stages in query
> > plan,
> > > > etc.
> > > > >> >> > > > > >
> > > > >> >> > > > > > (**) In some cases, hash function related ordering
> > > > >> differences
> > > > >> >> > > > originate
> > > > >> >> > > > > > in external libraries like Avro, antlr, json
> ,Thrift's
> > > map,
> > > > >> etc.
> > > > >> >> > for
> > > > >> >> > > > > which
> > > > >> >> > > > > > code
> > > > >> >> > > > > > changes are either more involved or led to more test
> > > > >> failures.
> > > > >> >> To
> > > > >> >> > > > > address
> > > > >> >> > > > > > this,
> > > > >> >> > > > > > we added support for version specific golden files
> > > > >> (HIVE-9109).
> > > > >> >> > > > > Currently,
> > > > >> >> > > > > > there
> > > > >> >> > > > > > are ~40 golden files with ".java1.8.out" extension.
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [discuss] jdk8 support

[Siddharth Seth <ss...@apache.org>]

+1. Drop Java7 support, and start using JDK8 features.

On Fri, Jun 3, 2016 at 8:13 AM, Ashutosh Chauhan <ha...@apache.org>
wrote:

> What I meant was that we start compiling using jdk8 as well. That will
> allow devs to use jdk8 only features (lambda functions etc.)
>
> On Fri, Jun 3, 2016 at 8:10 AM, Sergio Pena <se...@cloudera.com>
> wrote:
>
> > Hey Ashutosh,
> >
> > I switched to JDK8 in master last weekend. Jenkins has been running all
> > tests in Java8 successfully for a week now.
> > There are still a few tests we need to fix, but so far is looking good.
> >
> > Sergio
> >
> > On Fri, Jun 3, 2016 at 12:02 AM, Ashutosh Chauhan <ha...@apache.org>
> > wrote:
> >
> > > Now that branch-2.1 has been cut, I think its an opportune time to drop
> > > support of jdk7 altogether in master. That is we compile using jdk8 as
> > > well. What do others think?
> > >
> > > Ashutosh
> > >
> > > On Fri, May 27, 2016 at 7:41 PM, Sergio Pena <sergio.pena@cloudera.com
> >
> > > wrote:
> > >
> > > > I did the change to JDK8 on Ptest now.
> > > >
> > > > Please let me tknow if there are issues with Java8.
> > > >
> > > > - Sergio
> > > >
> > > > On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <
> sergio.pena@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Thanks Mohit.
> > > > >
> > > > > I will plan to do the JDK8 change on Jenkins today EOD and monitor
> > any
> > > > > issues through the weekend.
> > > > >
> > > > > - Sergio
> > > > >
> > > > >
> > > > > On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <
> mohit@cloudera.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Update on moving Hive2 tests to JDK8: I've addressed almost all
> test
> > > > >> failures in HIVE-13547 on java8 branch. There is one remaining
> > > > >> open item (HIVE-13834) which is currently assigned. Given current
> > > > >> state of flaky test runs, there might be few more.
> > > > >>
> > > > >> I will work with Sergio to merge the test fixes to master and
> switch
> > > > >> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
> > > > >>
> > > > >> After Hive2 tests switch, if your patch sees ordering related test
> > > > >> failures in pre-commit run, it's likely JDK version related and
> > you'll
> > > > >> need to build & re-run the test using JDK8. Number of such tests
> > > > >> should be relatively small.
> > > > >>
> > > > >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <
> > mohit@cloudera.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Created HIVE-13547 to track switching 2x tests to JDK8.
> > > > >> >
> > > > >> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <
> > > > sergio.pena@cloudera.com
> > > > >> >
> > > > >> > wrote:
> > > > >> >
> > > > >> >> I agree with such change as JDK7 is not longer supported.
> > > > >> >>
> > > > >> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just
> need
> > > to
> > > > >> >> replace the path from java7 to java8. But I think we should fix
> > all
> > > > >> JDK8
> > > > >> >> issues or most of them before doing the change or we will end
> up
> > > > >> having a
> > > > >> >> lot of failures on all JIRAs running pre-commit tests.
> > > > >> >>
> > > > >> >> +1 with the change.
> > > > >> >>
> > > > >> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <
> > sseth@apache.org>
> > > > >> wrote:
> > > > >> >>
> > > > >> >> > Option 3 sounds good. I'd ideally like to make JDK8 the
> minimum
> > > > >> >> requirement
> > > > >> >> > soon as well.
> > > > >> >> >
> > > > >> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <
> > szehon@cloudera.com>
> > > > >> wrote:
> > > > >> >> >
> > > > >> >> > > Sounds like a good plan, +1
> > > > >> >> > >
> > > > >> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
> > > > >> mohit@cloudera.com>
> > > > >> >> > > wrote:
> > > > >> >> > >
> > > > >> >> > > > Thanks, Ashutosh. Makes sense to keep the source and
> target
> > > as
> > > > >> 1.7
> > > > >> >> > since
> > > > >> >> > > > we're not using any JDK8 specific features yet. So,
> option
> > > (3)
> > > > >> >> > > essentially
> > > > >> >> > > > just means using JDK8 exclusively to build & test Hive2.
> > > > >> >> > > >
> > > > >> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> > > > >> >> > hashutosh@apache.org>
> > > > >> >> > > > wrote:
> > > > >> >> > > >
> > > > >> >> > > > > Hi Mohit,
> > > > >> >> > > > >
> > > > >> >> > > > > I also think option 3 makes sense. We should strive to
> > keep
> > > > >> test
> > > > >> >> > matrix
> > > > >> >> > > > > small so that we can do fast QA runs for dev patches.
> > > > >> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to
> > > build
> > > > &
> > > > >> >> test
> > > > >> >> > > > Hive2.
> > > > >> >> > > > > However, I am not sure of explicitly dropping support
> > > > >> altogether
> > > > >> >> for
> > > > >> >> > > jdk7
> > > > >> >> > > > > on Hive2. We should make sure that in pom.xml java
> > source &
> > > > >> target
> > > > >> >> > > > > compatibility is still 1.7 (which already is the case
> > > > >> currently)
> > > > >> >> so
> > > > >> >> > > that
> > > > >> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I
> said
> > > we
> > > > >> can
> > > > >> >> run
> > > > >> >> > on
> > > > >> >> > > > > jdk8.
> > > > >> >> > > > >
> > > > >> >> > > > > Thanks,
> > > > >> >> > > > > Ashutosh
> > > > >> >> > > > >
> > > > >> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
> > > > >> >> mohit@cloudera.com>
> > > > >> >> > > > > wrote:
> > > > >> >> > > > >
> > > > >> >> > > > > > Hi folks,
> > > > >> >> > > > > >
> > > > >> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in
> > April
> > > > >> 2015.
> > > > >> >> > > > > > In order to support Java 8, we fixed quite a bunch of
> > > tests
> > > > >> in
> > > > >> >> > > > > > HIVE-8607 (*) early last year. However, since our
> > > > pre-commit
> > > > >> >> tests
> > > > >> >> > > run
> > > > >> >> > > > > > on JDK7 only, any JDK8 test failures are getting
> > ignored.
> > > > As
> > > > >> a
> > > > >> >> > > result,
> > > > >> >> > > > > > the count has crept up
> > > > >> >> > > > > > <
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > > > >> >> > > > > > >
> > > > >> >> > > > > > back
> > > > >> >> > > > > > from zero to 125.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Some options to address this:
> > > > >> >> > > > > >
> > > > >> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in
> both
> > > 1.x
> > > > >> >> > > > > > and 2.x: This will further slow down the pre-commit
> > run.
> > > > >> >> > > > > >
> > > > >> >> > > > > > (2) Alternate pre-commit test runs between JDK7 and
> > JDK8
> > > > (in
> > > > >> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy,
> > though
> > > > >> >> failures
> > > > >> >> > > > > > won't get ignored.
> > > > >> >> > > > > >
> > > > >> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and
> > > test
> > > > >> Hive
> > > > >> >> 2.x
> > > > >> >> > > on
> > > > >> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7
> > pre-commit
> > > > >> tests.
> > > > >> >> > > > > > The pros are:
> > > > >> >> > > > > > - Simple test matrix.
> > > > >> >> > > > > > - No need to slow down test run or maintain version
> > > > specific
> > > > >> >> golden
> > > > >> >> > > > files
> > > > >> >> > > > > > (**).
> > > > >> >> > > > > > - 2.x looks like the logical place to move to JDK8.
> > > > >> >> > > > > > - Users transitioning to JDK8 for all other services
> do
> > > not
> > > > >> >> have to
> > > > >> >> > > > > > maintain
> > > > >> >> > > > > > multiple java versions on the cluster.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Option (3) looks most attractive to me.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Moving to JDK8 also lines us up better for Java 9,
> > which
> > > is
> > > > >> on
> > > > >> >> the
> > > > >> >> > > > > > horizon (Oracle will end public updates for Java 8 in
> > Sep
> > > > >> 2017)
> > > > >> >> > > > > >
> > > > >> >> > > > > > Around 100 of the latest crop of failures are due to
> > one
> > > > >> cause
> > > > >> >> > > > > > (HIVE-13409).
> > > > >> >> > > > > > I can take a pass at triaging the rest if there is
> > > > consensus
> > > > >> >> around
> > > > >> >> > > > what
> > > > >> >> > > > > > to do overall.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Thanks,
> > > > >> >> > > > > > Mohit
> > > > >> >> > > > > >
> > > > >> >> > > > > > (*) Most test failures are due to JDK8 using a
> > different
> > > > hash
> > > > >> >> > > function
> > > > >> >> > > > > for
> > > > >> >> > > > > > HashMap  compared to JDK7. This results in (mostly
> > > benign,
> > > > >> but
> > > > >> >> hard
> > > > >> >> > > > > > to debug) ordering differences in q-file output
> related
> > > to
> > > > >> >> > > > serialization
> > > > >> >> > > > > > order of map entries, numbering of stages in query
> > plan,
> > > > etc.
> > > > >> >> > > > > >
> > > > >> >> > > > > > (**) In some cases, hash function related ordering
> > > > >> differences
> > > > >> >> > > > originate
> > > > >> >> > > > > > in external libraries like Avro, antlr, json
> ,Thrift's
> > > map,
> > > > >> etc.
> > > > >> >> > for
> > > > >> >> > > > > which
> > > > >> >> > > > > > code
> > > > >> >> > > > > > changes are either more involved or led to more test
> > > > >> failures.
> > > > >> >> To
> > > > >> >> > > > > address
> > > > >> >> > > > > > this,
> > > > >> >> > > > > > we added support for version specific golden files
> > > > >> (HIVE-9109).
> > > > >> >> > > > > Currently,
> > > > >> >> > > > > > there
> > > > >> >> > > > > > are ~40 golden files with ".java1.8.out" extension.
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [discuss] jdk8 support

[Ashutosh Chauhan <ha...@apache.org>]

What I meant was that we start compiling using jdk8 as well. That will
allow devs to use jdk8 only features (lambda functions etc.)

On Fri, Jun 3, 2016 at 8:10 AM, Sergio Pena <se...@cloudera.com>
wrote:

> Hey Ashutosh,
>
> I switched to JDK8 in master last weekend. Jenkins has been running all
> tests in Java8 successfully for a week now.
> There are still a few tests we need to fix, but so far is looking good.
>
> Sergio
>
> On Fri, Jun 3, 2016 at 12:02 AM, Ashutosh Chauhan <ha...@apache.org>
> wrote:
>
> > Now that branch-2.1 has been cut, I think its an opportune time to drop
> > support of jdk7 altogether in master. That is we compile using jdk8 as
> > well. What do others think?
> >
> > Ashutosh
> >
> > On Fri, May 27, 2016 at 7:41 PM, Sergio Pena <se...@cloudera.com>
> > wrote:
> >
> > > I did the change to JDK8 on Ptest now.
> > >
> > > Please let me tknow if there are issues with Java8.
> > >
> > > - Sergio
> > >
> > > On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <sergio.pena@cloudera.com
> >
> > > wrote:
> > >
> > > > Thanks Mohit.
> > > >
> > > > I will plan to do the JDK8 change on Jenkins today EOD and monitor
> any
> > > > issues through the weekend.
> > > >
> > > > - Sergio
> > > >
> > > >
> > > > On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <mohit@cloudera.com
> >
> > > > wrote:
> > > >
> > > >> Update on moving Hive2 tests to JDK8: I've addressed almost all test
> > > >> failures in HIVE-13547 on java8 branch. There is one remaining
> > > >> open item (HIVE-13834) which is currently assigned. Given current
> > > >> state of flaky test runs, there might be few more.
> > > >>
> > > >> I will work with Sergio to merge the test fixes to master and switch
> > > >> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
> > > >>
> > > >> After Hive2 tests switch, if your patch sees ordering related test
> > > >> failures in pre-commit run, it's likely JDK version related and
> you'll
> > > >> need to build & re-run the test using JDK8. Number of such tests
> > > >> should be relatively small.
> > > >>
> > > >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <
> mohit@cloudera.com
> > >
> > > >> wrote:
> > > >>
> > > >> > Created HIVE-13547 to track switching 2x tests to JDK8.
> > > >> >
> > > >> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <
> > > sergio.pena@cloudera.com
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> >> I agree with such change as JDK7 is not longer supported.
> > > >> >>
> > > >> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just need
> > to
> > > >> >> replace the path from java7 to java8. But I think we should fix
> all
> > > >> JDK8
> > > >> >> issues or most of them before doing the change or we will end up
> > > >> having a
> > > >> >> lot of failures on all JIRAs running pre-commit tests.
> > > >> >>
> > > >> >> +1 with the change.
> > > >> >>
> > > >> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <
> sseth@apache.org>
> > > >> wrote:
> > > >> >>
> > > >> >> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
> > > >> >> requirement
> > > >> >> > soon as well.
> > > >> >> >
> > > >> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <
> szehon@cloudera.com>
> > > >> wrote:
> > > >> >> >
> > > >> >> > > Sounds like a good plan, +1
> > > >> >> > >
> > > >> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
> > > >> mohit@cloudera.com>
> > > >> >> > > wrote:
> > > >> >> > >
> > > >> >> > > > Thanks, Ashutosh. Makes sense to keep the source and target
> > as
> > > >> 1.7
> > > >> >> > since
> > > >> >> > > > we're not using any JDK8 specific features yet. So, option
> > (3)
> > > >> >> > > essentially
> > > >> >> > > > just means using JDK8 exclusively to build & test Hive2.
> > > >> >> > > >
> > > >> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> > > >> >> > hashutosh@apache.org>
> > > >> >> > > > wrote:
> > > >> >> > > >
> > > >> >> > > > > Hi Mohit,
> > > >> >> > > > >
> > > >> >> > > > > I also think option 3 makes sense. We should strive to
> keep
> > > >> test
> > > >> >> > matrix
> > > >> >> > > > > small so that we can do fast QA runs for dev patches.
> > > >> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to
> > build
> > > &
> > > >> >> test
> > > >> >> > > > Hive2.
> > > >> >> > > > > However, I am not sure of explicitly dropping support
> > > >> altogether
> > > >> >> for
> > > >> >> > > jdk7
> > > >> >> > > > > on Hive2. We should make sure that in pom.xml java
> source &
> > > >> target
> > > >> >> > > > > compatibility is still 1.7 (which already is the case
> > > >> currently)
> > > >> >> so
> > > >> >> > > that
> > > >> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I said
> > we
> > > >> can
> > > >> >> run
> > > >> >> > on
> > > >> >> > > > > jdk8.
> > > >> >> > > > >
> > > >> >> > > > > Thanks,
> > > >> >> > > > > Ashutosh
> > > >> >> > > > >
> > > >> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
> > > >> >> mohit@cloudera.com>
> > > >> >> > > > > wrote:
> > > >> >> > > > >
> > > >> >> > > > > > Hi folks,
> > > >> >> > > > > >
> > > >> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in
> April
> > > >> 2015.
> > > >> >> > > > > > In order to support Java 8, we fixed quite a bunch of
> > tests
> > > >> in
> > > >> >> > > > > > HIVE-8607 (*) early last year. However, since our
> > > pre-commit
> > > >> >> tests
> > > >> >> > > run
> > > >> >> > > > > > on JDK7 only, any JDK8 test failures are getting
> ignored.
> > > As
> > > >> a
> > > >> >> > > result,
> > > >> >> > > > > > the count has crept up
> > > >> >> > > > > > <
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > > >> >> > > > > > >
> > > >> >> > > > > > back
> > > >> >> > > > > > from zero to 125.
> > > >> >> > > > > >
> > > >> >> > > > > > Some options to address this:
> > > >> >> > > > > >
> > > >> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both
> > 1.x
> > > >> >> > > > > > and 2.x: This will further slow down the pre-commit
> run.
> > > >> >> > > > > >
> > > >> >> > > > > > (2) Alternate pre-commit test runs between JDK7 and
> JDK8
> > > (in
> > > >> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy,
> though
> > > >> >> failures
> > > >> >> > > > > > won't get ignored.
> > > >> >> > > > > >
> > > >> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and
> > test
> > > >> Hive
> > > >> >> 2.x
> > > >> >> > > on
> > > >> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7
> pre-commit
> > > >> tests.
> > > >> >> > > > > > The pros are:
> > > >> >> > > > > > - Simple test matrix.
> > > >> >> > > > > > - No need to slow down test run or maintain version
> > > specific
> > > >> >> golden
> > > >> >> > > > files
> > > >> >> > > > > > (**).
> > > >> >> > > > > > - 2.x looks like the logical place to move to JDK8.
> > > >> >> > > > > > - Users transitioning to JDK8 for all other services do
> > not
> > > >> >> have to
> > > >> >> > > > > > maintain
> > > >> >> > > > > > multiple java versions on the cluster.
> > > >> >> > > > > >
> > > >> >> > > > > > Option (3) looks most attractive to me.
> > > >> >> > > > > >
> > > >> >> > > > > > Moving to JDK8 also lines us up better for Java 9,
> which
> > is
> > > >> on
> > > >> >> the
> > > >> >> > > > > > horizon (Oracle will end public updates for Java 8 in
> Sep
> > > >> 2017)
> > > >> >> > > > > >
> > > >> >> > > > > > Around 100 of the latest crop of failures are due to
> one
> > > >> cause
> > > >> >> > > > > > (HIVE-13409).
> > > >> >> > > > > > I can take a pass at triaging the rest if there is
> > > consensus
> > > >> >> around
> > > >> >> > > > what
> > > >> >> > > > > > to do overall.
> > > >> >> > > > > >
> > > >> >> > > > > > Thanks,
> > > >> >> > > > > > Mohit
> > > >> >> > > > > >
> > > >> >> > > > > > (*) Most test failures are due to JDK8 using a
> different
> > > hash
> > > >> >> > > function
> > > >> >> > > > > for
> > > >> >> > > > > > HashMap  compared to JDK7. This results in (mostly
> > benign,
> > > >> but
> > > >> >> hard
> > > >> >> > > > > > to debug) ordering differences in q-file output related
> > to
> > > >> >> > > > serialization
> > > >> >> > > > > > order of map entries, numbering of stages in query
> plan,
> > > etc.
> > > >> >> > > > > >
> > > >> >> > > > > > (**) In some cases, hash function related ordering
> > > >> differences
> > > >> >> > > > originate
> > > >> >> > > > > > in external libraries like Avro, antlr, json ,Thrift's
> > map,
> > > >> etc.
> > > >> >> > for
> > > >> >> > > > > which
> > > >> >> > > > > > code
> > > >> >> > > > > > changes are either more involved or led to more test
> > > >> failures.
> > > >> >> To
> > > >> >> > > > > address
> > > >> >> > > > > > this,
> > > >> >> > > > > > we added support for version specific golden files
> > > >> (HIVE-9109).
> > > >> >> > > > > Currently,
> > > >> >> > > > > > there
> > > >> >> > > > > > are ~40 golden files with ".java1.8.out" extension.
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: [discuss] jdk8 support

[Sergio Pena <se...@cloudera.com>]

Hey Ashutosh,

I switched to JDK8 in master last weekend. Jenkins has been running all
tests in Java8 successfully for a week now.
There are still a few tests we need to fix, but so far is looking good.

Sergio

On Fri, Jun 3, 2016 at 12:02 AM, Ashutosh Chauhan <ha...@apache.org>
wrote:

> Now that branch-2.1 has been cut, I think its an opportune time to drop
> support of jdk7 altogether in master. That is we compile using jdk8 as
> well. What do others think?
>
> Ashutosh
>
> On Fri, May 27, 2016 at 7:41 PM, Sergio Pena <se...@cloudera.com>
> wrote:
>
> > I did the change to JDK8 on Ptest now.
> >
> > Please let me tknow if there are issues with Java8.
> >
> > - Sergio
> >
> > On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <se...@cloudera.com>
> > wrote:
> >
> > > Thanks Mohit.
> > >
> > > I will plan to do the JDK8 change on Jenkins today EOD and monitor any
> > > issues through the weekend.
> > >
> > > - Sergio
> > >
> > >
> > > On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <mo...@cloudera.com>
> > > wrote:
> > >
> > >> Update on moving Hive2 tests to JDK8: I've addressed almost all test
> > >> failures in HIVE-13547 on java8 branch. There is one remaining
> > >> open item (HIVE-13834) which is currently assigned. Given current
> > >> state of flaky test runs, there might be few more.
> > >>
> > >> I will work with Sergio to merge the test fixes to master and switch
> > >> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
> > >>
> > >> After Hive2 tests switch, if your patch sees ordering related test
> > >> failures in pre-commit run, it's likely JDK version related and you'll
> > >> need to build & re-run the test using JDK8. Number of such tests
> > >> should be relatively small.
> > >>
> > >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <mohit@cloudera.com
> >
> > >> wrote:
> > >>
> > >> > Created HIVE-13547 to track switching 2x tests to JDK8.
> > >> >
> > >> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <
> > sergio.pena@cloudera.com
> > >> >
> > >> > wrote:
> > >> >
> > >> >> I agree with such change as JDK7 is not longer supported.
> > >> >>
> > >> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just need
> to
> > >> >> replace the path from java7 to java8. But I think we should fix all
> > >> JDK8
> > >> >> issues or most of them before doing the change or we will end up
> > >> having a
> > >> >> lot of failures on all JIRAs running pre-commit tests.
> > >> >>
> > >> >> +1 with the change.
> > >> >>
> > >> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <ss...@apache.org>
> > >> wrote:
> > >> >>
> > >> >> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
> > >> >> requirement
> > >> >> > soon as well.
> > >> >> >
> > >> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <sz...@cloudera.com>
> > >> wrote:
> > >> >> >
> > >> >> > > Sounds like a good plan, +1
> > >> >> > >
> > >> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
> > >> mohit@cloudera.com>
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > > Thanks, Ashutosh. Makes sense to keep the source and target
> as
> > >> 1.7
> > >> >> > since
> > >> >> > > > we're not using any JDK8 specific features yet. So, option
> (3)
> > >> >> > > essentially
> > >> >> > > > just means using JDK8 exclusively to build & test Hive2.
> > >> >> > > >
> > >> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> > >> >> > hashutosh@apache.org>
> > >> >> > > > wrote:
> > >> >> > > >
> > >> >> > > > > Hi Mohit,
> > >> >> > > > >
> > >> >> > > > > I also think option 3 makes sense. We should strive to keep
> > >> test
> > >> >> > matrix
> > >> >> > > > > small so that we can do fast QA runs for dev patches.
> > >> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to
> build
> > &
> > >> >> test
> > >> >> > > > Hive2.
> > >> >> > > > > However, I am not sure of explicitly dropping support
> > >> altogether
> > >> >> for
> > >> >> > > jdk7
> > >> >> > > > > on Hive2. We should make sure that in pom.xml java source &
> > >> target
> > >> >> > > > > compatibility is still 1.7 (which already is the case
> > >> currently)
> > >> >> so
> > >> >> > > that
> > >> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I said
> we
> > >> can
> > >> >> run
> > >> >> > on
> > >> >> > > > > jdk8.
> > >> >> > > > >
> > >> >> > > > > Thanks,
> > >> >> > > > > Ashutosh
> > >> >> > > > >
> > >> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
> > >> >> mohit@cloudera.com>
> > >> >> > > > > wrote:
> > >> >> > > > >
> > >> >> > > > > > Hi folks,
> > >> >> > > > > >
> > >> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in April
> > >> 2015.
> > >> >> > > > > > In order to support Java 8, we fixed quite a bunch of
> tests
> > >> in
> > >> >> > > > > > HIVE-8607 (*) early last year. However, since our
> > pre-commit
> > >> >> tests
> > >> >> > > run
> > >> >> > > > > > on JDK7 only, any JDK8 test failures are getting ignored.
> > As
> > >> a
> > >> >> > > result,
> > >> >> > > > > > the count has crept up
> > >> >> > > > > > <
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > >> >> > > > > > >
> > >> >> > > > > > back
> > >> >> > > > > > from zero to 125.
> > >> >> > > > > >
> > >> >> > > > > > Some options to address this:
> > >> >> > > > > >
> > >> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both
> 1.x
> > >> >> > > > > > and 2.x: This will further slow down the pre-commit run.
> > >> >> > > > > >
> > >> >> > > > > > (2) Alternate pre-commit test runs between JDK7 and JDK8
> > (in
> > >> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy, though
> > >> >> failures
> > >> >> > > > > > won't get ignored.
> > >> >> > > > > >
> > >> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and
> test
> > >> Hive
> > >> >> 2.x
> > >> >> > > on
> > >> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit
> > >> tests.
> > >> >> > > > > > The pros are:
> > >> >> > > > > > - Simple test matrix.
> > >> >> > > > > > - No need to slow down test run or maintain version
> > specific
> > >> >> golden
> > >> >> > > > files
> > >> >> > > > > > (**).
> > >> >> > > > > > - 2.x looks like the logical place to move to JDK8.
> > >> >> > > > > > - Users transitioning to JDK8 for all other services do
> not
> > >> >> have to
> > >> >> > > > > > maintain
> > >> >> > > > > > multiple java versions on the cluster.
> > >> >> > > > > >
> > >> >> > > > > > Option (3) looks most attractive to me.
> > >> >> > > > > >
> > >> >> > > > > > Moving to JDK8 also lines us up better for Java 9, which
> is
> > >> on
> > >> >> the
> > >> >> > > > > > horizon (Oracle will end public updates for Java 8 in Sep
> > >> 2017)
> > >> >> > > > > >
> > >> >> > > > > > Around 100 of the latest crop of failures are due to one
> > >> cause
> > >> >> > > > > > (HIVE-13409).
> > >> >> > > > > > I can take a pass at triaging the rest if there is
> > consensus
> > >> >> around
> > >> >> > > > what
> > >> >> > > > > > to do overall.
> > >> >> > > > > >
> > >> >> > > > > > Thanks,
> > >> >> > > > > > Mohit
> > >> >> > > > > >
> > >> >> > > > > > (*) Most test failures are due to JDK8 using a different
> > hash
> > >> >> > > function
> > >> >> > > > > for
> > >> >> > > > > > HashMap  compared to JDK7. This results in (mostly
> benign,
> > >> but
> > >> >> hard
> > >> >> > > > > > to debug) ordering differences in q-file output related
> to
> > >> >> > > > serialization
> > >> >> > > > > > order of map entries, numbering of stages in query plan,
> > etc.
> > >> >> > > > > >
> > >> >> > > > > > (**) In some cases, hash function related ordering
> > >> differences
> > >> >> > > > originate
> > >> >> > > > > > in external libraries like Avro, antlr, json ,Thrift's
> map,
> > >> etc.
> > >> >> > for
> > >> >> > > > > which
> > >> >> > > > > > code
> > >> >> > > > > > changes are either more involved or led to more test
> > >> failures.
> > >> >> To
> > >> >> > > > > address
> > >> >> > > > > > this,
> > >> >> > > > > > we added support for version specific golden files
> > >> (HIVE-9109).
> > >> >> > > > > Currently,
> > >> >> > > > > > there
> > >> >> > > > > > are ~40 golden files with ".java1.8.out" extension.
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>

Re: [discuss] jdk8 support

[Ashutosh Chauhan <ha...@apache.org>]

Now that branch-2.1 has been cut, I think its an opportune time to drop
support of jdk7 altogether in master. That is we compile using jdk8 as
well. What do others think?

Ashutosh

On Fri, May 27, 2016 at 7:41 PM, Sergio Pena <se...@cloudera.com>
wrote:

> I did the change to JDK8 on Ptest now.
>
> Please let me tknow if there are issues with Java8.
>
> - Sergio
>
> On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <se...@cloudera.com>
> wrote:
>
> > Thanks Mohit.
> >
> > I will plan to do the JDK8 change on Jenkins today EOD and monitor any
> > issues through the weekend.
> >
> > - Sergio
> >
> >
> > On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <mo...@cloudera.com>
> > wrote:
> >
> >> Update on moving Hive2 tests to JDK8: I've addressed almost all test
> >> failures in HIVE-13547 on java8 branch. There is one remaining
> >> open item (HIVE-13834) which is currently assigned. Given current
> >> state of flaky test runs, there might be few more.
> >>
> >> I will work with Sergio to merge the test fixes to master and switch
> >> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
> >>
> >> After Hive2 tests switch, if your patch sees ordering related test
> >> failures in pre-commit run, it's likely JDK version related and you'll
> >> need to build & re-run the test using JDK8. Number of such tests
> >> should be relatively small.
> >>
> >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <mo...@cloudera.com>
> >> wrote:
> >>
> >> > Created HIVE-13547 to track switching 2x tests to JDK8.
> >> >
> >> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <
> sergio.pena@cloudera.com
> >> >
> >> > wrote:
> >> >
> >> >> I agree with such change as JDK7 is not longer supported.
> >> >>
> >> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
> >> >> replace the path from java7 to java8. But I think we should fix all
> >> JDK8
> >> >> issues or most of them before doing the change or we will end up
> >> having a
> >> >> lot of failures on all JIRAs running pre-commit tests.
> >> >>
> >> >> +1 with the change.
> >> >>
> >> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <ss...@apache.org>
> >> wrote:
> >> >>
> >> >> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
> >> >> requirement
> >> >> > soon as well.
> >> >> >
> >> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <sz...@cloudera.com>
> >> wrote:
> >> >> >
> >> >> > > Sounds like a good plan, +1
> >> >> > >
> >> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
> >> mohit@cloudera.com>
> >> >> > > wrote:
> >> >> > >
> >> >> > > > Thanks, Ashutosh. Makes sense to keep the source and target as
> >> 1.7
> >> >> > since
> >> >> > > > we're not using any JDK8 specific features yet. So, option (3)
> >> >> > > essentially
> >> >> > > > just means using JDK8 exclusively to build & test Hive2.
> >> >> > > >
> >> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> >> >> > hashutosh@apache.org>
> >> >> > > > wrote:
> >> >> > > >
> >> >> > > > > Hi Mohit,
> >> >> > > > >
> >> >> > > > > I also think option 3 makes sense. We should strive to keep
> >> test
> >> >> > matrix
> >> >> > > > > small so that we can do fast QA runs for dev patches.
> >> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to build
> &
> >> >> test
> >> >> > > > Hive2.
> >> >> > > > > However, I am not sure of explicitly dropping support
> >> altogether
> >> >> for
> >> >> > > jdk7
> >> >> > > > > on Hive2. We should make sure that in pom.xml java source &
> >> target
> >> >> > > > > compatibility is still 1.7 (which already is the case
> >> currently)
> >> >> so
> >> >> > > that
> >> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I said we
> >> can
> >> >> run
> >> >> > on
> >> >> > > > > jdk8.
> >> >> > > > >
> >> >> > > > > Thanks,
> >> >> > > > > Ashutosh
> >> >> > > > >
> >> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
> >> >> mohit@cloudera.com>
> >> >> > > > > wrote:
> >> >> > > > >
> >> >> > > > > > Hi folks,
> >> >> > > > > >
> >> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in April
> >> 2015.
> >> >> > > > > > In order to support Java 8, we fixed quite a bunch of tests
> >> in
> >> >> > > > > > HIVE-8607 (*) early last year. However, since our
> pre-commit
> >> >> tests
> >> >> > > run
> >> >> > > > > > on JDK7 only, any JDK8 test failures are getting ignored.
> As
> >> a
> >> >> > > result,
> >> >> > > > > > the count has crept up
> >> >> > > > > > <
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> >> >> > > > > > >
> >> >> > > > > > back
> >> >> > > > > > from zero to 125.
> >> >> > > > > >
> >> >> > > > > > Some options to address this:
> >> >> > > > > >
> >> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
> >> >> > > > > > and 2.x: This will further slow down the pre-commit run.
> >> >> > > > > >
> >> >> > > > > > (2) Alternate pre-commit test runs between JDK7 and JDK8
> (in
> >> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy, though
> >> >> failures
> >> >> > > > > > won't get ignored.
> >> >> > > > > >
> >> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test
> >> Hive
> >> >> 2.x
> >> >> > > on
> >> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit
> >> tests.
> >> >> > > > > > The pros are:
> >> >> > > > > > - Simple test matrix.
> >> >> > > > > > - No need to slow down test run or maintain version
> specific
> >> >> golden
> >> >> > > > files
> >> >> > > > > > (**).
> >> >> > > > > > - 2.x looks like the logical place to move to JDK8.
> >> >> > > > > > - Users transitioning to JDK8 for all other services do not
> >> >> have to
> >> >> > > > > > maintain
> >> >> > > > > > multiple java versions on the cluster.
> >> >> > > > > >
> >> >> > > > > > Option (3) looks most attractive to me.
> >> >> > > > > >
> >> >> > > > > > Moving to JDK8 also lines us up better for Java 9, which is
> >> on
> >> >> the
> >> >> > > > > > horizon (Oracle will end public updates for Java 8 in Sep
> >> 2017)
> >> >> > > > > >
> >> >> > > > > > Around 100 of the latest crop of failures are due to one
> >> cause
> >> >> > > > > > (HIVE-13409).
> >> >> > > > > > I can take a pass at triaging the rest if there is
> consensus
> >> >> around
> >> >> > > > what
> >> >> > > > > > to do overall.
> >> >> > > > > >
> >> >> > > > > > Thanks,
> >> >> > > > > > Mohit
> >> >> > > > > >
> >> >> > > > > > (*) Most test failures are due to JDK8 using a different
> hash
> >> >> > > function
> >> >> > > > > for
> >> >> > > > > > HashMap  compared to JDK7. This results in (mostly benign,
> >> but
> >> >> hard
> >> >> > > > > > to debug) ordering differences in q-file output related to
> >> >> > > > serialization
> >> >> > > > > > order of map entries, numbering of stages in query plan,
> etc.
> >> >> > > > > >
> >> >> > > > > > (**) In some cases, hash function related ordering
> >> differences
> >> >> > > > originate
> >> >> > > > > > in external libraries like Avro, antlr, json ,Thrift's map,
> >> etc.
> >> >> > for
> >> >> > > > > which
> >> >> > > > > > code
> >> >> > > > > > changes are either more involved or led to more test
> >> failures.
> >> >> To
> >> >> > > > > address
> >> >> > > > > > this,
> >> >> > > > > > we added support for version specific golden files
> >> (HIVE-9109).
> >> >> > > > > Currently,
> >> >> > > > > > there
> >> >> > > > > > are ~40 golden files with ".java1.8.out" extension.
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>

Re: [discuss] jdk8 support

[Sergio Pena <se...@cloudera.com>]

I did the change to JDK8 on Ptest now.

Please let me tknow if there are issues with Java8.

- Sergio

On Fri, May 27, 2016 at 2:45 PM, Sergio Pena <se...@cloudera.com>
wrote:

> Thanks Mohit.
>
> I will plan to do the JDK8 change on Jenkins today EOD and monitor any
> issues through the weekend.
>
> - Sergio
>
>
> On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <mo...@cloudera.com>
> wrote:
>
>> Update on moving Hive2 tests to JDK8: I've addressed almost all test
>> failures in HIVE-13547 on java8 branch. There is one remaining
>> open item (HIVE-13834) which is currently assigned. Given current
>> state of flaky test runs, there might be few more.
>>
>> I will work with Sergio to merge the test fixes to master and switch
>> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
>>
>> After Hive2 tests switch, if your patch sees ordering related test
>> failures in pre-commit run, it's likely JDK version related and you'll
>> need to build & re-run the test using JDK8. Number of such tests
>> should be relatively small.
>>
>> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <mo...@cloudera.com>
>> wrote:
>>
>> > Created HIVE-13547 to track switching 2x tests to JDK8.
>> >
>> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <sergio.pena@cloudera.com
>> >
>> > wrote:
>> >
>> >> I agree with such change as JDK7 is not longer supported.
>> >>
>> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
>> >> replace the path from java7 to java8. But I think we should fix all
>> JDK8
>> >> issues or most of them before doing the change or we will end up
>> having a
>> >> lot of failures on all JIRAs running pre-commit tests.
>> >>
>> >> +1 with the change.
>> >>
>> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <ss...@apache.org>
>> wrote:
>> >>
>> >> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
>> >> requirement
>> >> > soon as well.
>> >> >
>> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <sz...@cloudera.com>
>> wrote:
>> >> >
>> >> > > Sounds like a good plan, +1
>> >> > >
>> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
>> mohit@cloudera.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Thanks, Ashutosh. Makes sense to keep the source and target as
>> 1.7
>> >> > since
>> >> > > > we're not using any JDK8 specific features yet. So, option (3)
>> >> > > essentially
>> >> > > > just means using JDK8 exclusively to build & test Hive2.
>> >> > > >
>> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
>> >> > hashutosh@apache.org>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Hi Mohit,
>> >> > > > >
>> >> > > > > I also think option 3 makes sense. We should strive to keep
>> test
>> >> > matrix
>> >> > > > > small so that we can do fast QA runs for dev patches.
>> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to build &
>> >> test
>> >> > > > Hive2.
>> >> > > > > However, I am not sure of explicitly dropping support
>> altogether
>> >> for
>> >> > > jdk7
>> >> > > > > on Hive2. We should make sure that in pom.xml java source &
>> target
>> >> > > > > compatibility is still 1.7 (which already is the case
>> currently)
>> >> so
>> >> > > that
>> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I said we
>> can
>> >> run
>> >> > on
>> >> > > > > jdk8.
>> >> > > > >
>> >> > > > > Thanks,
>> >> > > > > Ashutosh
>> >> > > > >
>> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
>> >> mohit@cloudera.com>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > Hi folks,
>> >> > > > > >
>> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in April
>> 2015.
>> >> > > > > > In order to support Java 8, we fixed quite a bunch of tests
>> in
>> >> > > > > > HIVE-8607 (*) early last year. However, since our pre-commit
>> >> tests
>> >> > > run
>> >> > > > > > on JDK7 only, any JDK8 test failures are getting ignored. As
>> a
>> >> > > result,
>> >> > > > > > the count has crept up
>> >> > > > > > <
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
>> >> > > > > > >
>> >> > > > > > back
>> >> > > > > > from zero to 125.
>> >> > > > > >
>> >> > > > > > Some options to address this:
>> >> > > > > >
>> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
>> >> > > > > > and 2.x: This will further slow down the pre-commit run.
>> >> > > > > >
>> >> > > > > > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
>> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy, though
>> >> failures
>> >> > > > > > won't get ignored.
>> >> > > > > >
>> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test
>> Hive
>> >> 2.x
>> >> > > on
>> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit
>> tests.
>> >> > > > > > The pros are:
>> >> > > > > > - Simple test matrix.
>> >> > > > > > - No need to slow down test run or maintain version specific
>> >> golden
>> >> > > > files
>> >> > > > > > (**).
>> >> > > > > > - 2.x looks like the logical place to move to JDK8.
>> >> > > > > > - Users transitioning to JDK8 for all other services do not
>> >> have to
>> >> > > > > > maintain
>> >> > > > > > multiple java versions on the cluster.
>> >> > > > > >
>> >> > > > > > Option (3) looks most attractive to me.
>> >> > > > > >
>> >> > > > > > Moving to JDK8 also lines us up better for Java 9, which is
>> on
>> >> the
>> >> > > > > > horizon (Oracle will end public updates for Java 8 in Sep
>> 2017)
>> >> > > > > >
>> >> > > > > > Around 100 of the latest crop of failures are due to one
>> cause
>> >> > > > > > (HIVE-13409).
>> >> > > > > > I can take a pass at triaging the rest if there is consensus
>> >> around
>> >> > > > what
>> >> > > > > > to do overall.
>> >> > > > > >
>> >> > > > > > Thanks,
>> >> > > > > > Mohit
>> >> > > > > >
>> >> > > > > > (*) Most test failures are due to JDK8 using a different hash
>> >> > > function
>> >> > > > > for
>> >> > > > > > HashMap  compared to JDK7. This results in (mostly benign,
>> but
>> >> hard
>> >> > > > > > to debug) ordering differences in q-file output related to
>> >> > > > serialization
>> >> > > > > > order of map entries, numbering of stages in query plan, etc.
>> >> > > > > >
>> >> > > > > > (**) In some cases, hash function related ordering
>> differences
>> >> > > > originate
>> >> > > > > > in external libraries like Avro, antlr, json ,Thrift's map,
>> etc.
>> >> > for
>> >> > > > > which
>> >> > > > > > code
>> >> > > > > > changes are either more involved or led to more test
>> failures.
>> >> To
>> >> > > > > address
>> >> > > > > > this,
>> >> > > > > > we added support for version specific golden files
>> (HIVE-9109).
>> >> > > > > Currently,
>> >> > > > > > there
>> >> > > > > > are ~40 golden files with ".java1.8.out" extension.
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>

Re: [discuss] jdk8 support

[Sergio Pena <se...@cloudera.com>]

Thanks Mohit.

I will plan to do the JDK8 change on Jenkins today EOD and monitor any
issues through the weekend.

- Sergio


On Fri, May 27, 2016 at 2:58 AM, Mohit Sabharwal <mo...@cloudera.com> wrote:

> Update on moving Hive2 tests to JDK8: I've addressed almost all test
> failures in HIVE-13547 on java8 branch. There is one remaining
> open item (HIVE-13834) which is currently assigned. Given current
> state of flaky test runs, there might be few more.
>
> I will work with Sergio to merge the test fixes to master and switch
> the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.
>
> After Hive2 tests switch, if your patch sees ordering related test
> failures in pre-commit run, it's likely JDK version related and you'll
> need to build & re-run the test using JDK8. Number of such tests
> should be relatively small.
>
> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <mo...@cloudera.com>
> wrote:
>
> > Created HIVE-13547 to track switching 2x tests to JDK8.
> >
> > On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <se...@cloudera.com>
> > wrote:
> >
> >> I agree with such change as JDK7 is not longer supported.
> >>
> >> Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
> >> replace the path from java7 to java8. But I think we should fix all JDK8
> >> issues or most of them before doing the change or we will end up having
> a
> >> lot of failures on all JIRAs running pre-commit tests.
> >>
> >> +1 with the change.
> >>
> >> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> >>
> >> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
> >> requirement
> >> > soon as well.
> >> >
> >> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <sz...@cloudera.com>
> wrote:
> >> >
> >> > > Sounds like a good plan, +1
> >> > >
> >> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <
> mohit@cloudera.com>
> >> > > wrote:
> >> > >
> >> > > > Thanks, Ashutosh. Makes sense to keep the source and target as 1.7
> >> > since
> >> > > > we're not using any JDK8 specific features yet. So, option (3)
> >> > > essentially
> >> > > > just means using JDK8 exclusively to build & test Hive2.
> >> > > >
> >> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> >> > hashutosh@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Mohit,
> >> > > > >
> >> > > > > I also think option 3 makes sense. We should strive to keep test
> >> > matrix
> >> > > > > small so that we can do fast QA runs for dev patches.
> >> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to build &
> >> test
> >> > > > Hive2.
> >> > > > > However, I am not sure of explicitly dropping support altogether
> >> for
> >> > > jdk7
> >> > > > > on Hive2. We should make sure that in pom.xml java source &
> target
> >> > > > > compatibility is still 1.7 (which already is the case currently)
> >> so
> >> > > that
> >> > > > > Hive2 is still compatible with jdk7. Unit tests as I said we can
> >> run
> >> > on
> >> > > > > jdk8.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Ashutosh
> >> > > > >
> >> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
> >> mohit@cloudera.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi folks,
> >> > > > > >
> >> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
> >> > > > > > In order to support Java 8, we fixed quite a bunch of tests in
> >> > > > > > HIVE-8607 (*) early last year. However, since our pre-commit
> >> tests
> >> > > run
> >> > > > > > on JDK7 only, any JDK8 test failures are getting ignored. As a
> >> > > result,
> >> > > > > > the count has crept up
> >> > > > > > <
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> >> > > > > > >
> >> > > > > > back
> >> > > > > > from zero to 125.
> >> > > > > >
> >> > > > > > Some options to address this:
> >> > > > > >
> >> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
> >> > > > > > and 2.x: This will further slow down the pre-commit run.
> >> > > > > >
> >> > > > > > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
> >> > > > > > both 1.x and 2.x): It's a cheap hack. More messy, though
> >> failures
> >> > > > > > won't get ignored.
> >> > > > > >
> >> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test
> Hive
> >> 2.x
> >> > > on
> >> > > > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit
> tests.
> >> > > > > > The pros are:
> >> > > > > > - Simple test matrix.
> >> > > > > > - No need to slow down test run or maintain version specific
> >> golden
> >> > > > files
> >> > > > > > (**).
> >> > > > > > - 2.x looks like the logical place to move to JDK8.
> >> > > > > > - Users transitioning to JDK8 for all other services do not
> >> have to
> >> > > > > > maintain
> >> > > > > > multiple java versions on the cluster.
> >> > > > > >
> >> > > > > > Option (3) looks most attractive to me.
> >> > > > > >
> >> > > > > > Moving to JDK8 also lines us up better for Java 9, which is on
> >> the
> >> > > > > > horizon (Oracle will end public updates for Java 8 in Sep
> 2017)
> >> > > > > >
> >> > > > > > Around 100 of the latest crop of failures are due to one cause
> >> > > > > > (HIVE-13409).
> >> > > > > > I can take a pass at triaging the rest if there is consensus
> >> around
> >> > > > what
> >> > > > > > to do overall.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Mohit
> >> > > > > >
> >> > > > > > (*) Most test failures are due to JDK8 using a different hash
> >> > > function
> >> > > > > for
> >> > > > > > HashMap  compared to JDK7. This results in (mostly benign, but
> >> hard
> >> > > > > > to debug) ordering differences in q-file output related to
> >> > > > serialization
> >> > > > > > order of map entries, numbering of stages in query plan, etc.
> >> > > > > >
> >> > > > > > (**) In some cases, hash function related ordering differences
> >> > > > originate
> >> > > > > > in external libraries like Avro, antlr, json ,Thrift's map,
> etc.
> >> > for
> >> > > > > which
> >> > > > > > code
> >> > > > > > changes are either more involved or led to more test failures.
> >> To
> >> > > > > address
> >> > > > > > this,
> >> > > > > > we added support for version specific golden files
> (HIVE-9109).
> >> > > > > Currently,
> >> > > > > > there
> >> > > > > > are ~40 golden files with ".java1.8.out" extension.
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: [discuss] jdk8 support

[Mohit Sabharwal <mo...@cloudera.com>]

Update on moving Hive2 tests to JDK8: I've addressed almost all test
failures in HIVE-13547 on java8 branch. There is one remaining
open item (HIVE-13834) which is currently assigned. Given current
state of flaky test runs, there might be few more.

I will work with Sergio to merge the test fixes to master and switch
the Hive2 pre-commit job to use JDK8, hopefully sometime tomorrow.

After Hive2 tests switch, if your patch sees ordering related test
failures in pre-commit run, it's likely JDK version related and you'll
need to build & re-run the test using JDK8. Number of such tests
should be relatively small.

On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal <mo...@cloudera.com>
wrote:

> Created HIVE-13547 to track switching 2x tests to JDK8.
>
> On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <se...@cloudera.com>
> wrote:
>
>> I agree with such change as JDK7 is not longer supported.
>>
>> Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
>> replace the path from java7 to java8. But I think we should fix all JDK8
>> issues or most of them before doing the change or we will end up having a
>> lot of failures on all JIRAs running pre-commit tests.
>>
>> +1 with the change.
>>
>> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <ss...@apache.org> wrote:
>>
>> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
>> requirement
>> > soon as well.
>> >
>> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <sz...@cloudera.com> wrote:
>> >
>> > > Sounds like a good plan, +1
>> > >
>> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <mo...@cloudera.com>
>> > > wrote:
>> > >
>> > > > Thanks, Ashutosh. Makes sense to keep the source and target as 1.7
>> > since
>> > > > we're not using any JDK8 specific features yet. So, option (3)
>> > > essentially
>> > > > just means using JDK8 exclusively to build & test Hive2.
>> > > >
>> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
>> > hashutosh@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Hi Mohit,
>> > > > >
>> > > > > I also think option 3 makes sense. We should strive to keep test
>> > matrix
>> > > > > small so that we can do fast QA runs for dev patches.
>> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to build &
>> test
>> > > > Hive2.
>> > > > > However, I am not sure of explicitly dropping support altogether
>> for
>> > > jdk7
>> > > > > on Hive2. We should make sure that in pom.xml java source & target
>> > > > > compatibility is still 1.7 (which already is the case currently)
>> so
>> > > that
>> > > > > Hive2 is still compatible with jdk7. Unit tests as I said we can
>> run
>> > on
>> > > > > jdk8.
>> > > > >
>> > > > > Thanks,
>> > > > > Ashutosh
>> > > > >
>> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
>> mohit@cloudera.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi folks,
>> > > > > >
>> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
>> > > > > > In order to support Java 8, we fixed quite a bunch of tests in
>> > > > > > HIVE-8607 (*) early last year. However, since our pre-commit
>> tests
>> > > run
>> > > > > > on JDK7 only, any JDK8 test failures are getting ignored. As a
>> > > result,
>> > > > > > the count has crept up
>> > > > > > <
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
>> > > > > > >
>> > > > > > back
>> > > > > > from zero to 125.
>> > > > > >
>> > > > > > Some options to address this:
>> > > > > >
>> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
>> > > > > > and 2.x: This will further slow down the pre-commit run.
>> > > > > >
>> > > > > > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
>> > > > > > both 1.x and 2.x): It's a cheap hack. More messy, though
>> failures
>> > > > > > won't get ignored.
>> > > > > >
>> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test Hive
>> 2.x
>> > > on
>> > > > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit tests.
>> > > > > > The pros are:
>> > > > > > - Simple test matrix.
>> > > > > > - No need to slow down test run or maintain version specific
>> golden
>> > > > files
>> > > > > > (**).
>> > > > > > - 2.x looks like the logical place to move to JDK8.
>> > > > > > - Users transitioning to JDK8 for all other services do not
>> have to
>> > > > > > maintain
>> > > > > > multiple java versions on the cluster.
>> > > > > >
>> > > > > > Option (3) looks most attractive to me.
>> > > > > >
>> > > > > > Moving to JDK8 also lines us up better for Java 9, which is on
>> the
>> > > > > > horizon (Oracle will end public updates for Java 8 in Sep 2017)
>> > > > > >
>> > > > > > Around 100 of the latest crop of failures are due to one cause
>> > > > > > (HIVE-13409).
>> > > > > > I can take a pass at triaging the rest if there is consensus
>> around
>> > > > what
>> > > > > > to do overall.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Mohit
>> > > > > >
>> > > > > > (*) Most test failures are due to JDK8 using a different hash
>> > > function
>> > > > > for
>> > > > > > HashMap  compared to JDK7. This results in (mostly benign, but
>> hard
>> > > > > > to debug) ordering differences in q-file output related to
>> > > > serialization
>> > > > > > order of map entries, numbering of stages in query plan, etc.
>> > > > > >
>> > > > > > (**) In some cases, hash function related ordering differences
>> > > > originate
>> > > > > > in external libraries like Avro, antlr, json ,Thrift's map, etc.
>> > for
>> > > > > which
>> > > > > > code
>> > > > > > changes are either more involved or led to more test failures.
>> To
>> > > > > address
>> > > > > > this,
>> > > > > > we added support for version specific golden files (HIVE-9109).
>> > > > > Currently,
>> > > > > > there
>> > > > > > are ~40 golden files with ".java1.8.out" extension.
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: [discuss] jdk8 support

[Mohit Sabharwal <mo...@cloudera.com>]

Created HIVE-13547 to track switching 2x tests to JDK8.

On Wed, Apr 13, 2016 at 10:02 AM, Sergio Pena <se...@cloudera.com>
wrote:

> I agree with such change as JDK7 is not longer supported.
>
> Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
> replace the path from java7 to java8. But I think we should fix all JDK8
> issues or most of them before doing the change or we will end up having a
> lot of failures on all JIRAs running pre-commit tests.
>
> +1 with the change.
>
> On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <ss...@apache.org> wrote:
>
> > Option 3 sounds good. I'd ideally like to make JDK8 the minimum
> requirement
> > soon as well.
> >
> > On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <sz...@cloudera.com> wrote:
> >
> > > Sounds like a good plan, +1
> > >
> > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <mo...@cloudera.com>
> > > wrote:
> > >
> > > > Thanks, Ashutosh. Makes sense to keep the source and target as 1.7
> > since
> > > > we're not using any JDK8 specific features yet. So, option (3)
> > > essentially
> > > > just means using JDK8 exclusively to build & test Hive2.
> > > >
> > > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> > hashutosh@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Mohit,
> > > > >
> > > > > I also think option 3 makes sense. We should strive to keep test
> > matrix
> > > > > small so that we can do fast QA runs for dev patches.
> > > > > We can just use jdk7 to build & test Hive1 and jdk8 to build & test
> > > > Hive2.
> > > > > However, I am not sure of explicitly dropping support altogether
> for
> > > jdk7
> > > > > on Hive2. We should make sure that in pom.xml java source & target
> > > > > compatibility is still 1.7 (which already is the case currently) so
> > > that
> > > > > Hive2 is still compatible with jdk7. Unit tests as I said we can
> run
> > on
> > > > > jdk8.
> > > > >
> > > > > Thanks,
> > > > > Ashutosh
> > > > >
> > > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <
> mohit@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
> > > > > > In order to support Java 8, we fixed quite a bunch of tests in
> > > > > > HIVE-8607 (*) early last year. However, since our pre-commit
> tests
> > > run
> > > > > > on JDK7 only, any JDK8 test failures are getting ignored. As a
> > > result,
> > > > > > the count has crept up
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > > > > > >
> > > > > > back
> > > > > > from zero to 125.
> > > > > >
> > > > > > Some options to address this:
> > > > > >
> > > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
> > > > > > and 2.x: This will further slow down the pre-commit run.
> > > > > >
> > > > > > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
> > > > > > both 1.x and 2.x): It's a cheap hack. More messy, though failures
> > > > > > won't get ignored.
> > > > > >
> > > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test Hive
> 2.x
> > > on
> > > > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit tests.
> > > > > > The pros are:
> > > > > > - Simple test matrix.
> > > > > > - No need to slow down test run or maintain version specific
> golden
> > > > files
> > > > > > (**).
> > > > > > - 2.x looks like the logical place to move to JDK8.
> > > > > > - Users transitioning to JDK8 for all other services do not have
> to
> > > > > > maintain
> > > > > > multiple java versions on the cluster.
> > > > > >
> > > > > > Option (3) looks most attractive to me.
> > > > > >
> > > > > > Moving to JDK8 also lines us up better for Java 9, which is on
> the
> > > > > > horizon (Oracle will end public updates for Java 8 in Sep 2017)
> > > > > >
> > > > > > Around 100 of the latest crop of failures are due to one cause
> > > > > > (HIVE-13409).
> > > > > > I can take a pass at triaging the rest if there is consensus
> around
> > > > what
> > > > > > to do overall.
> > > > > >
> > > > > > Thanks,
> > > > > > Mohit
> > > > > >
> > > > > > (*) Most test failures are due to JDK8 using a different hash
> > > function
> > > > > for
> > > > > > HashMap  compared to JDK7. This results in (mostly benign, but
> hard
> > > > > > to debug) ordering differences in q-file output related to
> > > > serialization
> > > > > > order of map entries, numbering of stages in query plan, etc.
> > > > > >
> > > > > > (**) In some cases, hash function related ordering differences
> > > > originate
> > > > > > in external libraries like Avro, antlr, json ,Thrift's map, etc.
> > for
> > > > > which
> > > > > > code
> > > > > > changes are either more involved or led to more test failures.
> To
> > > > > address
> > > > > > this,
> > > > > > we added support for version specific golden files (HIVE-9109).
> > > > > Currently,
> > > > > > there
> > > > > > are ~40 golden files with ".java1.8.out" extension.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [discuss] jdk8 support

[Sergio Pena <se...@cloudera.com>]

I agree with such change as JDK7 is not longer supported.

Changes on Jenkins and Hive PTest shouldn't be hard. We just need to
replace the path from java7 to java8. But I think we should fix all JDK8
issues or most of them before doing the change or we will end up having a
lot of failures on all JIRAs running pre-commit tests.

+1 with the change.

On Mon, Apr 11, 2016 at 9:34 PM, Siddharth Seth <ss...@apache.org> wrote:

> Option 3 sounds good. I'd ideally like to make JDK8 the minimum requirement
> soon as well.
>
> On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <sz...@cloudera.com> wrote:
>
> > Sounds like a good plan, +1
> >
> > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <mo...@cloudera.com>
> > wrote:
> >
> > > Thanks, Ashutosh. Makes sense to keep the source and target as 1.7
> since
> > > we're not using any JDK8 specific features yet. So, option (3)
> > essentially
> > > just means using JDK8 exclusively to build & test Hive2.
> > >
> > > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <
> hashutosh@apache.org>
> > > wrote:
> > >
> > > > Hi Mohit,
> > > >
> > > > I also think option 3 makes sense. We should strive to keep test
> matrix
> > > > small so that we can do fast QA runs for dev patches.
> > > > We can just use jdk7 to build & test Hive1 and jdk8 to build & test
> > > Hive2.
> > > > However, I am not sure of explicitly dropping support altogether for
> > jdk7
> > > > on Hive2. We should make sure that in pom.xml java source & target
> > > > compatibility is still 1.7 (which already is the case currently) so
> > that
> > > > Hive2 is still compatible with jdk7. Unit tests as I said we can run
> on
> > > > jdk8.
> > > >
> > > > Thanks,
> > > > Ashutosh
> > > >
> > > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <mo...@cloudera.com>
> > > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
> > > > > In order to support Java 8, we fixed quite a bunch of tests in
> > > > > HIVE-8607 (*) early last year. However, since our pre-commit tests
> > run
> > > > > on JDK7 only, any JDK8 test failures are getting ignored. As a
> > result,
> > > > > the count has crept up
> > > > > <
> > > > >
> > > >
> > >
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > > > > >
> > > > > back
> > > > > from zero to 125.
> > > > >
> > > > > Some options to address this:
> > > > >
> > > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
> > > > > and 2.x: This will further slow down the pre-commit run.
> > > > >
> > > > > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
> > > > > both 1.x and 2.x): It's a cheap hack. More messy, though failures
> > > > > won't get ignored.
> > > > >
> > > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test Hive 2.x
> > on
> > > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit tests.
> > > > > The pros are:
> > > > > - Simple test matrix.
> > > > > - No need to slow down test run or maintain version specific golden
> > > files
> > > > > (**).
> > > > > - 2.x looks like the logical place to move to JDK8.
> > > > > - Users transitioning to JDK8 for all other services do not have to
> > > > > maintain
> > > > > multiple java versions on the cluster.
> > > > >
> > > > > Option (3) looks most attractive to me.
> > > > >
> > > > > Moving to JDK8 also lines us up better for Java 9, which is on the
> > > > > horizon (Oracle will end public updates for Java 8 in Sep 2017)
> > > > >
> > > > > Around 100 of the latest crop of failures are due to one cause
> > > > > (HIVE-13409).
> > > > > I can take a pass at triaging the rest if there is consensus around
> > > what
> > > > > to do overall.
> > > > >
> > > > > Thanks,
> > > > > Mohit
> > > > >
> > > > > (*) Most test failures are due to JDK8 using a different hash
> > function
> > > > for
> > > > > HashMap  compared to JDK7. This results in (mostly benign, but hard
> > > > > to debug) ordering differences in q-file output related to
> > > serialization
> > > > > order of map entries, numbering of stages in query plan, etc.
> > > > >
> > > > > (**) In some cases, hash function related ordering differences
> > > originate
> > > > > in external libraries like Avro, antlr, json ,Thrift's map, etc.
> for
> > > > which
> > > > > code
> > > > > changes are either more involved or led to more test failures.  To
> > > > address
> > > > > this,
> > > > > we added support for version specific golden files (HIVE-9109).
> > > > Currently,
> > > > > there
> > > > > are ~40 golden files with ".java1.8.out" extension.
> > > > >
> > > >
> > >
> >
>

Re: [discuss] jdk8 support

[Siddharth Seth <ss...@apache.org>]

Option 3 sounds good. I'd ideally like to make JDK8 the minimum requirement
soon as well.

On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho <sz...@cloudera.com> wrote:

> Sounds like a good plan, +1
>
> On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <mo...@cloudera.com>
> wrote:
>
> > Thanks, Ashutosh. Makes sense to keep the source and target as 1.7 since
> > we're not using any JDK8 specific features yet. So, option (3)
> essentially
> > just means using JDK8 exclusively to build & test Hive2.
> >
> > On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <ha...@apache.org>
> > wrote:
> >
> > > Hi Mohit,
> > >
> > > I also think option 3 makes sense. We should strive to keep test matrix
> > > small so that we can do fast QA runs for dev patches.
> > > We can just use jdk7 to build & test Hive1 and jdk8 to build & test
> > Hive2.
> > > However, I am not sure of explicitly dropping support altogether for
> jdk7
> > > on Hive2. We should make sure that in pom.xml java source & target
> > > compatibility is still 1.7 (which already is the case currently) so
> that
> > > Hive2 is still compatible with jdk7. Unit tests as I said we can run on
> > > jdk8.
> > >
> > > Thanks,
> > > Ashutosh
> > >
> > > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <mo...@cloudera.com>
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
> > > > In order to support Java 8, we fixed quite a bunch of tests in
> > > > HIVE-8607 (*) early last year. However, since our pre-commit tests
> run
> > > > on JDK7 only, any JDK8 test failures are getting ignored. As a
> result,
> > > > the count has crept up
> > > > <
> > > >
> > >
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > > > >
> > > > back
> > > > from zero to 125.
> > > >
> > > > Some options to address this:
> > > >
> > > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
> > > > and 2.x: This will further slow down the pre-commit run.
> > > >
> > > > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
> > > > both 1.x and 2.x): It's a cheap hack. More messy, though failures
> > > > won't get ignored.
> > > >
> > > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test Hive 2.x
> on
> > > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit tests.
> > > > The pros are:
> > > > - Simple test matrix.
> > > > - No need to slow down test run or maintain version specific golden
> > files
> > > > (**).
> > > > - 2.x looks like the logical place to move to JDK8.
> > > > - Users transitioning to JDK8 for all other services do not have to
> > > > maintain
> > > > multiple java versions on the cluster.
> > > >
> > > > Option (3) looks most attractive to me.
> > > >
> > > > Moving to JDK8 also lines us up better for Java 9, which is on the
> > > > horizon (Oracle will end public updates for Java 8 in Sep 2017)
> > > >
> > > > Around 100 of the latest crop of failures are due to one cause
> > > > (HIVE-13409).
> > > > I can take a pass at triaging the rest if there is consensus around
> > what
> > > > to do overall.
> > > >
> > > > Thanks,
> > > > Mohit
> > > >
> > > > (*) Most test failures are due to JDK8 using a different hash
> function
> > > for
> > > > HashMap  compared to JDK7. This results in (mostly benign, but hard
> > > > to debug) ordering differences in q-file output related to
> > serialization
> > > > order of map entries, numbering of stages in query plan, etc.
> > > >
> > > > (**) In some cases, hash function related ordering differences
> > originate
> > > > in external libraries like Avro, antlr, json ,Thrift's map, etc. for
> > > which
> > > > code
> > > > changes are either more involved or led to more test failures.  To
> > > address
> > > > this,
> > > > we added support for version specific golden files (HIVE-9109).
> > > Currently,
> > > > there
> > > > are ~40 golden files with ".java1.8.out" extension.
> > > >
> > >
> >
>

Re: [discuss] jdk8 support

[Szehon Ho <sz...@cloudera.com>]

Sounds like a good plan, +1

On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal <mo...@cloudera.com> wrote:

> Thanks, Ashutosh. Makes sense to keep the source and target as 1.7 since
> we're not using any JDK8 specific features yet. So, option (3) essentially
> just means using JDK8 exclusively to build & test Hive2.
>
> On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <ha...@apache.org>
> wrote:
>
> > Hi Mohit,
> >
> > I also think option 3 makes sense. We should strive to keep test matrix
> > small so that we can do fast QA runs for dev patches.
> > We can just use jdk7 to build & test Hive1 and jdk8 to build & test
> Hive2.
> > However, I am not sure of explicitly dropping support altogether for jdk7
> > on Hive2. We should make sure that in pom.xml java source & target
> > compatibility is still 1.7 (which already is the case currently) so that
> > Hive2 is still compatible with jdk7. Unit tests as I said we can run on
> > jdk8.
> >
> > Thanks,
> > Ashutosh
> >
> > On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <mo...@cloudera.com>
> > wrote:
> >
> > > Hi folks,
> > >
> > > Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
> > > In order to support Java 8, we fixed quite a bunch of tests in
> > > HIVE-8607 (*) early last year. However, since our pre-commit tests run
> > > on JDK7 only, any JDK8 test failures are getting ignored. As a result,
> > > the count has crept up
> > > <
> > >
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > > >
> > > back
> > > from zero to 125.
> > >
> > > Some options to address this:
> > >
> > > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
> > > and 2.x: This will further slow down the pre-commit run.
> > >
> > > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
> > > both 1.x and 2.x): It's a cheap hack. More messy, though failures
> > > won't get ignored.
> > >
> > > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test Hive 2.x on
> > > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit tests.
> > > The pros are:
> > > - Simple test matrix.
> > > - No need to slow down test run or maintain version specific golden
> files
> > > (**).
> > > - 2.x looks like the logical place to move to JDK8.
> > > - Users transitioning to JDK8 for all other services do not have to
> > > maintain
> > > multiple java versions on the cluster.
> > >
> > > Option (3) looks most attractive to me.
> > >
> > > Moving to JDK8 also lines us up better for Java 9, which is on the
> > > horizon (Oracle will end public updates for Java 8 in Sep 2017)
> > >
> > > Around 100 of the latest crop of failures are due to one cause
> > > (HIVE-13409).
> > > I can take a pass at triaging the rest if there is consensus around
> what
> > > to do overall.
> > >
> > > Thanks,
> > > Mohit
> > >
> > > (*) Most test failures are due to JDK8 using a different hash function
> > for
> > > HashMap  compared to JDK7. This results in (mostly benign, but hard
> > > to debug) ordering differences in q-file output related to
> serialization
> > > order of map entries, numbering of stages in query plan, etc.
> > >
> > > (**) In some cases, hash function related ordering differences
> originate
> > > in external libraries like Avro, antlr, json ,Thrift's map, etc. for
> > which
> > > code
> > > changes are either more involved or led to more test failures.  To
> > address
> > > this,
> > > we added support for version specific golden files (HIVE-9109).
> > Currently,
> > > there
> > > are ~40 golden files with ".java1.8.out" extension.
> > >
> >
>

Re: [discuss] jdk8 support

[Mohit Sabharwal <mo...@cloudera.com>]

Thanks, Ashutosh. Makes sense to keep the source and target as 1.7 since
we're not using any JDK8 specific features yet. So, option (3) essentially
just means using JDK8 exclusively to build & test Hive2.

On Sat, Apr 9, 2016 at 12:23 PM, Ashutosh Chauhan <ha...@apache.org>
wrote:

> Hi Mohit,
>
> I also think option 3 makes sense. We should strive to keep test matrix
> small so that we can do fast QA runs for dev patches.
> We can just use jdk7 to build & test Hive1 and jdk8 to build & test Hive2.
> However, I am not sure of explicitly dropping support altogether for jdk7
> on Hive2. We should make sure that in pom.xml java source & target
> compatibility is still 1.7 (which already is the case currently) so that
> Hive2 is still compatible with jdk7. Unit tests as I said we can run on
> jdk8.
>
> Thanks,
> Ashutosh
>
> On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <mo...@cloudera.com>
> wrote:
>
> > Hi folks,
> >
> > Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
> > In order to support Java 8, we fixed quite a bunch of tests in
> > HIVE-8607 (*) early last year. However, since our pre-commit tests run
> > on JDK7 only, any JDK8 test failures are getting ignored. As a result,
> > the count has crept up
> > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> > >
> > back
> > from zero to 125.
> >
> > Some options to address this:
> >
> > (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
> > and 2.x: This will further slow down the pre-commit run.
> >
> > (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
> > both 1.x and 2.x): It's a cheap hack. More messy, though failures
> > won't get ignored.
> >
> > (3) Drop support for JDK7 in Hive 2.x, i.e. build and test Hive 2.x on
> > JDK8 only. For Hive 1.x, continue running JDK7 pre-commit tests.
> > The pros are:
> > - Simple test matrix.
> > - No need to slow down test run or maintain version specific golden files
> > (**).
> > - 2.x looks like the logical place to move to JDK8.
> > - Users transitioning to JDK8 for all other services do not have to
> > maintain
> > multiple java versions on the cluster.
> >
> > Option (3) looks most attractive to me.
> >
> > Moving to JDK8 also lines us up better for Java 9, which is on the
> > horizon (Oracle will end public updates for Java 8 in Sep 2017)
> >
> > Around 100 of the latest crop of failures are due to one cause
> > (HIVE-13409).
> > I can take a pass at triaging the rest if there is consensus around what
> > to do overall.
> >
> > Thanks,
> > Mohit
> >
> > (*) Most test failures are due to JDK8 using a different hash function
> for
> > HashMap  compared to JDK7. This results in (mostly benign, but hard
> > to debug) ordering differences in q-file output related to serialization
> > order of map entries, numbering of stages in query plan, etc.
> >
> > (**) In some cases, hash function related ordering differences originate
> > in external libraries like Avro, antlr, json ,Thrift's map, etc. for
> which
> > code
> > changes are either more involved or led to more test failures.  To
> address
> > this,
> > we added support for version specific golden files (HIVE-9109).
> Currently,
> > there
> > are ~40 golden files with ".java1.8.out" extension.
> >
>

Re: [discuss] jdk8 support

[Ashutosh Chauhan <ha...@apache.org>]

Hi Mohit,

I also think option 3 makes sense. We should strive to keep test matrix
small so that we can do fast QA runs for dev patches.
We can just use jdk7 to build & test Hive1 and jdk8 to build & test Hive2.
However, I am not sure of explicitly dropping support altogether for jdk7
on Hive2. We should make sure that in pom.xml java source & target
compatibility is still 1.7 (which already is the case currently) so that
Hive2 is still compatible with jdk7. Unit tests as I said we can run on
jdk8.

Thanks,
Ashutosh

On Fri, Apr 8, 2016 at 4:34 PM, Mohit Sabharwal <mo...@cloudera.com> wrote:

> Hi folks,
>
> Oracle EOL'ed (ended public updates) for Java 7 in April 2015.
> In order to support Java 8, we fixed quite a bunch of tests in
> HIVE-8607 (*) early last year. However, since our pre-commit tests run
> on JDK7 only, any JDK8 test failures are getting ignored. As a result,
> the count has crept up
> <
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/
> >
> back
> from zero to 125.
>
> Some options to address this:
>
> (1) Run pre-commit tests on both JDK7 and JDK8, in both 1.x
> and 2.x: This will further slow down the pre-commit run.
>
> (2) Alternate pre-commit test runs between JDK7 and JDK8 (in
> both 1.x and 2.x): It's a cheap hack. More messy, though failures
> won't get ignored.
>
> (3) Drop support for JDK7 in Hive 2.x, i.e. build and test Hive 2.x on
> JDK8 only. For Hive 1.x, continue running JDK7 pre-commit tests.
> The pros are:
> - Simple test matrix.
> - No need to slow down test run or maintain version specific golden files
> (**).
> - 2.x looks like the logical place to move to JDK8.
> - Users transitioning to JDK8 for all other services do not have to
> maintain
> multiple java versions on the cluster.
>
> Option (3) looks most attractive to me.
>
> Moving to JDK8 also lines us up better for Java 9, which is on the
> horizon (Oracle will end public updates for Java 8 in Sep 2017)
>
> Around 100 of the latest crop of failures are due to one cause
> (HIVE-13409).
> I can take a pass at triaging the rest if there is consensus around what
> to do overall.
>
> Thanks,
> Mohit
>
> (*) Most test failures are due to JDK8 using a different hash function for
> HashMap  compared to JDK7. This results in (mostly benign, but hard
> to debug) ordering differences in q-file output related to serialization
> order of map entries, numbering of stages in query plan, etc.
>
> (**) In some cases, hash function related ordering differences originate
> in external libraries like Avro, antlr, json ,Thrift's map, etc. for which
> code
> changes are either more involved or led to more test failures.  To address
> this,
> we added support for version specific golden files (HIVE-9109). Currently,
> there
> are ~40 golden files with ".java1.8.out" extension.
>