You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Sailesh Mukil <sa...@cloudera.com> on 2018/04/02 19:07:43 UTC

[DISCUSS] Apache Impala 2.12.0 and 3.0 releases

Folks,

It's been about 3 months since our last release and there have been a good
number of bug fixes, features added and performance improvements since then.

I propose that we have a 2.12.0 release soon and a 3.0 release in parallel.
I, as a PMC of Apache Impala volunteer to mange both these releases.

Why have 2 releases in parallel?

We've been maintaining 2 branches in parallel for a while now. A 2.x branch
which works with Hadoop 2.0 and friends, and a master branch, which now
works with Hadoop 3.0 and friends.
We've also made sure to not push any incompatible changes into our 2.x
branch, but only to the master branch, and we have a good number of these
incompatible changes on master now that justifies having a 3.0 release.

There have been 351 commits on master since the last release, Apache Impala
2.11.0. I've gone through them and outlined the commits that are either
considered breaking changes or support a breaking change:

IMPALA-4277: Switch to using Hadoop 3 and friends by default.

IMPALA-6510: [DOCS] Remove refresh_after_connect

IMPALA-6647: Add CREATE fine-grained privilege

IMPALA-6230, IMPALA-6468: Fix the output type of round() and related fns

IMPALA-4277: Support multiple versions of Hadoop ecosystem

IMPALA-6415: [DOCS] fix invalid <ph conref>

IMPALA-6643: Add REFRESH fine-grained privilege

IMPALA-5191, IMPALA-6415: [DOCS] Document breaking change of alias and
ordinal substitution

IMPALA-3916: [DOCS] Reserved keywords updated for Impala 3.0

IMPALA-6240: [DOCS] Document PARQUET_ARRAY_RESOLUTION query option

[DOCS] Removed the obsolete Llama options files

IMPALA-6077: remove Parquet BIT_PACKED def level support

IMPALA-3271: organise and warn on removed startup flags

IMPALA-5037: Default PARQUET_ARRAY_RESOLUTION=THREE_LEVEL

IMPALA-5293: Turn insert clustering on by default

IMPALA-4924 addendum: Change result type to decimal in a TPCH query

IMPALA-3916: Reserve SQL:2016 reserved words   (Disabled on 2.x with
IMPALA-3916: Change reserved_words_version to 2.11.0)

IMPALA-4319: remove some deprecated query options

IMPALA-4924: Enable Decimal V2 by default

IMPALA-5191: Standardize column alias behavior

IMPALA-3998: Remove refresh_after_connect option from shell

The 3.0 release would come off the master branch and the 2.12.0 release
would come off the 2.x branch.

Please speak up and let the community know if anyone has any objections to
this.

P.S: This is not a VOTE thread but only a DISCUSS thread.

Thanks,
Sailesh

Re: Re: Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

Posted by Sailesh Mukil <sa...@cloudera.com>.
An update:

Impala 2.12.0 has been released.

All remaining tasks for Impala 3.0.0 have been completed. The release is
now in preparation and a vote will commence soon.

On Fri, Apr 13, 2018 at 11:51 AM, Sailesh Mukil <sa...@cloudera.com>
wrote:

> Thank you all.
>
> I've triaged all the remaining open JIRAs for Impala 2.12.0 and Impala 3.0
> and we have 2 more breaking changes that need to go in for Impala 3.0:
> https://issues.apache.org/jira/browse/IMPALA-6340
> https://issues.apache.org/jira/browse/IMPALA-5893
>
> There are also a couple of patches that are very close to completion that
> will make it in:
> https://issues.apache.org/jira/browse/IMPALA-6790
> https://issues.apache.org/jira/browse/IMPALA-6793
>
> And lastly, there are a few docs JIRAs that are being worked on for Impala
> 3.0 and Impala 2.12.0.
>
> IMPALA-6340 is probably the most important change at this point, since
> DECIMAL_V2 is enabled by default in Impala 3.0 and this is a blocker bug
> that needs to be fixed to allow the feature to be used reliably.
>
> All the above JIRAs are being worked on as top priorities and we should
> have them completed soon. I will start the remaining formalities as soon as
> those changes are in.
>
> Full list of open 2.12.0 JIRAs:
> https://issues.apache.org/jira/browse/IMPALA-6793?jql=
> project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%22%20%3D%20%
> 22Impala%202.12.0%22
>
> Full list of open 3.0 JIRAs:
> https://issues.apache.org/jira/browse/IMPALA-6793?jql=
> project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%22%20%3D%20%
> 22Impala%203.0%22
>
> Thanks to all the authors and reviewers of the above JIRAs for working on
> them in a timely manner.
>
> - Sailesh
>
> On Tue, Apr 10, 2018 at 11:28 AM, Tim Armstrong <ta...@cloudera.com>
> wrote:
>
>> We also just merged a small breaking change to pave the way for Jinchul
>> Kim's patch: https://gerrit.cloudera.org/#/c/9957/ (otherwise we might
>> have
>> to wait for the next major release to merge that).
>>
>> On Mon, Apr 2, 2018 at 8:33 PM, Quanlong Huang <hu...@126.com>
>> wrote:
>>
>> > Thanks, Tim and Jim! Then I have no other suggestions.+1
>> >
>> > At 2018-04-03 10:57:58, "Jim Apple" <jb...@cloudera.com> wrote:
>> > >Additionally, because this patch doesn't break any backwards
>> > compatibility,
>> > >it's fine to merge it for 2.13 and 3.1 when those releases are cut. And
>> > >since there are no minimum waiting period between releases, we could
>> cut
>> > >those right after that patch lands if we like.
>> > >
>> > >On Mon, Apr 2, 2018 at 4:05 PM, Tim Armstrong <tarmstrong@cloudera.com
>> >
>> > >wrote:
>> > >
>> > >> I don't think we should hold the release for the ORC scanner patch -
>> > it's a
>> > >> great feature, but there are many other equally important features
>> and
>> > >> improvements that have been waiting for months to be released.
>> > >>
>> > >> Generally, there are always going to be big changes that are "nearly
>> > there"
>> > >> and I think we're best off releasing frequently rather than trying to
>> > hold
>> > >> releases for specific features. This is especially true for large
>> > features
>> > >> where it sometimes take a bit of bake time to flush out all of the
>> > issues
>> > >> with the initial patch.
>> > >>
>> > >> - Tim
>> > >>
>> > >> On Mon, Apr 2, 2018 at 3:49 PM, Quanlong Huang <
>> huang_quanlong@126.com>
>> > >> wrote:
>> > >>
>> > >> > Hi friends,
>> > >> >
>> > >> >
>> > >> > Is it possible to wait until IMPALA-5717 (
>> > https://gerrit.cloudera.org/#
>> > >> > /c/9134/) is merged? We're eager to upgrade our Impala cluster with
>> > the
>> > >> > latest ORC-support. I've heard from the community that many other
>> > >> companies
>> > >> > are waiting for it too. As it's an experimental feature so far, we
>> can
>> > >> get
>> > >> > more feedback on this release.
>> > >> >
>> > >> >
>> > >> > As Tim said, this patch is closed to be merged. Could you wait
>> until
>> > it's
>> > >> > merged?
>> > >> >
>> > >> >
>> > >> > Thanks,
>> > >> > Quanlong
>> > >> >
>> > >> > At 2018-04-03 05:56:11, "Jim Apple" <jb...@cloudera.com> wrote:
>> > >> > >agreed.
>> > >> > >
>> > >> > >On Mon, Apr 2, 2018 at 2:50 PM, Philip Zeyliger <
>> philip@cloudera.com
>> > >
>> > >> > wrote:
>> > >> > >
>> > >> > >> Sounds like a good plan to me!
>> > >> > >>
>> > >> > >> -- Philip
>> > >> > >>
>> > >> >
>> > >>
>> >
>>
>
>

Re: Re: Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

Posted by Sailesh Mukil <sa...@cloudera.com>.
Thank you all.

I've triaged all the remaining open JIRAs for Impala 2.12.0 and Impala 3.0
and we have 2 more breaking changes that need to go in for Impala 3.0:
https://issues.apache.org/jira/browse/IMPALA-6340
https://issues.apache.org/jira/browse/IMPALA-5893

There are also a couple of patches that are very close to completion that
will make it in:
https://issues.apache.org/jira/browse/IMPALA-6790
https://issues.apache.org/jira/browse/IMPALA-6793

And lastly, there are a few docs JIRAs that are being worked on for Impala
3.0 and Impala 2.12.0.

IMPALA-6340 is probably the most important change at this point, since
DECIMAL_V2 is enabled by default in Impala 3.0 and this is a blocker bug
that needs to be fixed to allow the feature to be used reliably.

All the above JIRAs are being worked on as top priorities and we should
have them completed soon. I will start the remaining formalities as soon as
those changes are in.

Full list of open 2.12.0 JIRAs:
https://issues.apache.org/jira/browse/IMPALA-6793?jql=project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%22%20%3D%20%22Impala%202.12.0%22

Full list of open 3.0 JIRAs:
https://issues.apache.org/jira/browse/IMPALA-6793?jql=project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%22%20%3D%20%22Impala%203.0%22

Thanks to all the authors and reviewers of the above JIRAs for working on
them in a timely manner.

- Sailesh

On Tue, Apr 10, 2018 at 11:28 AM, Tim Armstrong <ta...@cloudera.com>
wrote:

> We also just merged a small breaking change to pave the way for Jinchul
> Kim's patch: https://gerrit.cloudera.org/#/c/9957/ (otherwise we might
> have
> to wait for the next major release to merge that).
>
> On Mon, Apr 2, 2018 at 8:33 PM, Quanlong Huang <hu...@126.com>
> wrote:
>
> > Thanks, Tim and Jim! Then I have no other suggestions.+1
> >
> > At 2018-04-03 10:57:58, "Jim Apple" <jb...@cloudera.com> wrote:
> > >Additionally, because this patch doesn't break any backwards
> > compatibility,
> > >it's fine to merge it for 2.13 and 3.1 when those releases are cut. And
> > >since there are no minimum waiting period between releases, we could cut
> > >those right after that patch lands if we like.
> > >
> > >On Mon, Apr 2, 2018 at 4:05 PM, Tim Armstrong <ta...@cloudera.com>
> > >wrote:
> > >
> > >> I don't think we should hold the release for the ORC scanner patch -
> > it's a
> > >> great feature, but there are many other equally important features and
> > >> improvements that have been waiting for months to be released.
> > >>
> > >> Generally, there are always going to be big changes that are "nearly
> > there"
> > >> and I think we're best off releasing frequently rather than trying to
> > hold
> > >> releases for specific features. This is especially true for large
> > features
> > >> where it sometimes take a bit of bake time to flush out all of the
> > issues
> > >> with the initial patch.
> > >>
> > >> - Tim
> > >>
> > >> On Mon, Apr 2, 2018 at 3:49 PM, Quanlong Huang <
> huang_quanlong@126.com>
> > >> wrote:
> > >>
> > >> > Hi friends,
> > >> >
> > >> >
> > >> > Is it possible to wait until IMPALA-5717 (
> > https://gerrit.cloudera.org/#
> > >> > /c/9134/) is merged? We're eager to upgrade our Impala cluster with
> > the
> > >> > latest ORC-support. I've heard from the community that many other
> > >> companies
> > >> > are waiting for it too. As it's an experimental feature so far, we
> can
> > >> get
> > >> > more feedback on this release.
> > >> >
> > >> >
> > >> > As Tim said, this patch is closed to be merged. Could you wait until
> > it's
> > >> > merged?
> > >> >
> > >> >
> > >> > Thanks,
> > >> > Quanlong
> > >> >
> > >> > At 2018-04-03 05:56:11, "Jim Apple" <jb...@cloudera.com> wrote:
> > >> > >agreed.
> > >> > >
> > >> > >On Mon, Apr 2, 2018 at 2:50 PM, Philip Zeyliger <
> philip@cloudera.com
> > >
> > >> > wrote:
> > >> > >
> > >> > >> Sounds like a good plan to me!
> > >> > >>
> > >> > >> -- Philip
> > >> > >>
> > >> >
> > >>
> >
>

Re: Re: Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

Posted by Tim Armstrong <ta...@cloudera.com>.
We also just merged a small breaking change to pave the way for Jinchul
Kim's patch: https://gerrit.cloudera.org/#/c/9957/ (otherwise we might have
to wait for the next major release to merge that).

On Mon, Apr 2, 2018 at 8:33 PM, Quanlong Huang <hu...@126.com>
wrote:

> Thanks, Tim and Jim! Then I have no other suggestions.+1
>
> At 2018-04-03 10:57:58, "Jim Apple" <jb...@cloudera.com> wrote:
> >Additionally, because this patch doesn't break any backwards
> compatibility,
> >it's fine to merge it for 2.13 and 3.1 when those releases are cut. And
> >since there are no minimum waiting period between releases, we could cut
> >those right after that patch lands if we like.
> >
> >On Mon, Apr 2, 2018 at 4:05 PM, Tim Armstrong <ta...@cloudera.com>
> >wrote:
> >
> >> I don't think we should hold the release for the ORC scanner patch -
> it's a
> >> great feature, but there are many other equally important features and
> >> improvements that have been waiting for months to be released.
> >>
> >> Generally, there are always going to be big changes that are "nearly
> there"
> >> and I think we're best off releasing frequently rather than trying to
> hold
> >> releases for specific features. This is especially true for large
> features
> >> where it sometimes take a bit of bake time to flush out all of the
> issues
> >> with the initial patch.
> >>
> >> - Tim
> >>
> >> On Mon, Apr 2, 2018 at 3:49 PM, Quanlong Huang <hu...@126.com>
> >> wrote:
> >>
> >> > Hi friends,
> >> >
> >> >
> >> > Is it possible to wait until IMPALA-5717 (
> https://gerrit.cloudera.org/#
> >> > /c/9134/) is merged? We're eager to upgrade our Impala cluster with
> the
> >> > latest ORC-support. I've heard from the community that many other
> >> companies
> >> > are waiting for it too. As it's an experimental feature so far, we can
> >> get
> >> > more feedback on this release.
> >> >
> >> >
> >> > As Tim said, this patch is closed to be merged. Could you wait until
> it's
> >> > merged?
> >> >
> >> >
> >> > Thanks,
> >> > Quanlong
> >> >
> >> > At 2018-04-03 05:56:11, "Jim Apple" <jb...@cloudera.com> wrote:
> >> > >agreed.
> >> > >
> >> > >On Mon, Apr 2, 2018 at 2:50 PM, Philip Zeyliger <philip@cloudera.com
> >
> >> > wrote:
> >> > >
> >> > >> Sounds like a good plan to me!
> >> > >>
> >> > >> -- Philip
> >> > >>
> >> >
> >>
>

Re:Re: Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

Posted by Quanlong Huang <hu...@126.com>.
Thanks, Tim and Jim! Then I have no other suggestions.+1

At 2018-04-03 10:57:58, "Jim Apple" <jb...@cloudera.com> wrote:
>Additionally, because this patch doesn't break any backwards compatibility,
>it's fine to merge it for 2.13 and 3.1 when those releases are cut. And
>since there are no minimum waiting period between releases, we could cut
>those right after that patch lands if we like.
>
>On Mon, Apr 2, 2018 at 4:05 PM, Tim Armstrong <ta...@cloudera.com>
>wrote:
>
>> I don't think we should hold the release for the ORC scanner patch - it's a
>> great feature, but there are many other equally important features and
>> improvements that have been waiting for months to be released.
>>
>> Generally, there are always going to be big changes that are "nearly there"
>> and I think we're best off releasing frequently rather than trying to hold
>> releases for specific features. This is especially true for large features
>> where it sometimes take a bit of bake time to flush out all of the issues
>> with the initial patch.
>>
>> - Tim
>>
>> On Mon, Apr 2, 2018 at 3:49 PM, Quanlong Huang <hu...@126.com>
>> wrote:
>>
>> > Hi friends,
>> >
>> >
>> > Is it possible to wait until IMPALA-5717 (https://gerrit.cloudera.org/#
>> > /c/9134/) is merged? We're eager to upgrade our Impala cluster with the
>> > latest ORC-support. I've heard from the community that many other
>> companies
>> > are waiting for it too. As it's an experimental feature so far, we can
>> get
>> > more feedback on this release.
>> >
>> >
>> > As Tim said, this patch is closed to be merged. Could you wait until it's
>> > merged?
>> >
>> >
>> > Thanks,
>> > Quanlong
>> >
>> > At 2018-04-03 05:56:11, "Jim Apple" <jb...@cloudera.com> wrote:
>> > >agreed.
>> > >
>> > >On Mon, Apr 2, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
>> > wrote:
>> > >
>> > >> Sounds like a good plan to me!
>> > >>
>> > >> -- Philip
>> > >>
>> >
>>

Re: Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

Posted by Jim Apple <jb...@cloudera.com>.
Additionally, because this patch doesn't break any backwards compatibility,
it's fine to merge it for 2.13 and 3.1 when those releases are cut. And
since there are no minimum waiting period between releases, we could cut
those right after that patch lands if we like.

On Mon, Apr 2, 2018 at 4:05 PM, Tim Armstrong <ta...@cloudera.com>
wrote:

> I don't think we should hold the release for the ORC scanner patch - it's a
> great feature, but there are many other equally important features and
> improvements that have been waiting for months to be released.
>
> Generally, there are always going to be big changes that are "nearly there"
> and I think we're best off releasing frequently rather than trying to hold
> releases for specific features. This is especially true for large features
> where it sometimes take a bit of bake time to flush out all of the issues
> with the initial patch.
>
> - Tim
>
> On Mon, Apr 2, 2018 at 3:49 PM, Quanlong Huang <hu...@126.com>
> wrote:
>
> > Hi friends,
> >
> >
> > Is it possible to wait until IMPALA-5717 (https://gerrit.cloudera.org/#
> > /c/9134/) is merged? We're eager to upgrade our Impala cluster with the
> > latest ORC-support. I've heard from the community that many other
> companies
> > are waiting for it too. As it's an experimental feature so far, we can
> get
> > more feedback on this release.
> >
> >
> > As Tim said, this patch is closed to be merged. Could you wait until it's
> > merged?
> >
> >
> > Thanks,
> > Quanlong
> >
> > At 2018-04-03 05:56:11, "Jim Apple" <jb...@cloudera.com> wrote:
> > >agreed.
> > >
> > >On Mon, Apr 2, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
> > wrote:
> > >
> > >> Sounds like a good plan to me!
> > >>
> > >> -- Philip
> > >>
> >
>

Re: Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

Posted by Tim Armstrong <ta...@cloudera.com>.
I don't think we should hold the release for the ORC scanner patch - it's a
great feature, but there are many other equally important features and
improvements that have been waiting for months to be released.

Generally, there are always going to be big changes that are "nearly there"
and I think we're best off releasing frequently rather than trying to hold
releases for specific features. This is especially true for large features
where it sometimes take a bit of bake time to flush out all of the issues
with the initial patch.

- Tim

On Mon, Apr 2, 2018 at 3:49 PM, Quanlong Huang <hu...@126.com>
wrote:

> Hi friends,
>
>
> Is it possible to wait until IMPALA-5717 (https://gerrit.cloudera.org/#
> /c/9134/) is merged? We're eager to upgrade our Impala cluster with the
> latest ORC-support. I've heard from the community that many other companies
> are waiting for it too. As it's an experimental feature so far, we can get
> more feedback on this release.
>
>
> As Tim said, this patch is closed to be merged. Could you wait until it's
> merged?
>
>
> Thanks,
> Quanlong
>
> At 2018-04-03 05:56:11, "Jim Apple" <jb...@cloudera.com> wrote:
> >agreed.
> >
> >On Mon, Apr 2, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com>
> wrote:
> >
> >> Sounds like a good plan to me!
> >>
> >> -- Philip
> >>
>

Re:Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

Posted by Quanlong Huang <hu...@126.com>.
Hi friends,


Is it possible to wait until IMPALA-5717 (https://gerrit.cloudera.org/#/c/9134/) is merged? We're eager to upgrade our Impala cluster with the latest ORC-support. I've heard from the community that many other companies are waiting for it too. As it's an experimental feature so far, we can get more feedback on this release.


As Tim said, this patch is closed to be merged. Could you wait until it's merged?


Thanks,
Quanlong

At 2018-04-03 05:56:11, "Jim Apple" <jb...@cloudera.com> wrote:
>agreed.
>
>On Mon, Apr 2, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com> wrote:
>
>> Sounds like a good plan to me!
>>
>> -- Philip
>>

Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

Posted by Jim Apple <jb...@cloudera.com>.
agreed.

On Mon, Apr 2, 2018 at 2:50 PM, Philip Zeyliger <ph...@cloudera.com> wrote:

> Sounds like a good plan to me!
>
> -- Philip
>

Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

Posted by Philip Zeyliger <ph...@cloudera.com>.
Sounds like a good plan to me!

-- Philip