You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Tim Armstrong <ta...@cloudera.com> on 2020/06/17 21:47:16 UTC
Re: Impala 4 Breaking Changes
Another thing that we're looking into is changing the encoded runtime
profile representation to be more efficient - see
https://issues.apache.org/jira/browse/IMPALA-9378. So that might be a
default we will try to change in Impala 4.0.
On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <ta...@cloudera.com>
wrote:
> An addendum to this - we're also considering whether to increase the
> minimum CPU version so that we can simplify development and focus more on
> optimising for the most common CPUs - see
> https://issues.apache.org/jira/browse/IMPALA-9690
>
> Running on commodity hardware is important and we want people to be able
> to develop on whatever machine they have access to. At most we're likely
> going to require AVX2 support, which has been supported by the vast
> majority of CPUs for the better part of a decade.
>
> On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <jo...@cloudera.com>
> wrote:
>
>> Impala is starting development on Impala 4.0. There are many exciting
>> projects underway including full support for Hive ACID tables, improved
>> multithreading support, and fault tolerance improvements, but Impala 4.0
>> also serves as a time to alter behavior and platform support. Here is the
>> current list of notable breaking changes that are planned in Impala 4.0:
>>
>> Remove support for older operating systems:
>>
>> Maintaining support for older operating systems requires a considerable
>> amount of effort, especially with changing security requirements. Each
>> extra operating system consumes valuable effort and can conflict with
>> support for newer operating systems, so we have decided to remove support
>> for some older operating systems. Specifically, we plan to drop support for:
>>
>> -
>>
>> Centos 6
>> -
>>
>> Ubuntu 14
>> -
>>
>> Debian 8
>>
>> Each of these is approaching or past its end of life. Since Centos6 was
>> the only operating system still needing Python 2.6 and Python 2.6 is long
>> past its end of life, Python 2.6 will no longer be supported. In addition,
>> Centos 7 support will be focused on newer versions such as 7.5 and above.
>>
>> Remove support for Sentry:
>>
>> Over the past year and a half, Impala's Ranger authorization
>> functionality has achieved parity and surpassed the existing Sentry
>> authorization functionality. Impala's Sentry support requires ongoing
>> effort to test and maintain, including maintenance on Sentry itself to
>> address security updates. Given the diminished activity in the Sentry
>> community, there is no timeline for addressing these security updates.
>> Unless there is a maintainer for Sentry support, Impala plans to focus its
>> efforts on Ranger for its authorization system and drop Sentry.
>>
>> Remove support for Impala-lzo:
>>
>> Impala-lzo provides code to allow Impala to read the LZO compressed
>> tables. LZO is GPL licensed, which is why this support is not included
>> directly. The Impala-lzo code interacts with internal Impala code at a
>> level that is error prone and intricate. Given the low adoption of LZO and
>> the other compression options available, Impala plans to remove Impala-lzo
>> support along with the low level interface it used.
>>
>> Deprecations:
>>
>> In addition, we also plan to deprecate several existing features so that
>> they can be removed in a future release. Here is a summary of the most
>> notable ones:
>>
>> -
>>
>> Impala will default to using on demand metadata management (i.e. the
>> local catalog implementation) as described in this documentation:
>> http://impala.apache.org/docs/build/html/topics/impala_metadata.html
>> The old metadata system is now deprecated and may be removed in a
>> future release.
>> -
>>
>> Impala is deprecating the Beeswax client protocol (i.e. clients that
>> connect via beeswax_port) in favor of the HiveServer2 client protocol.
>> -
>>
>> Impala is deprecating the old decimal_v2=false behavior from Impala
>> 2.x. The decimal_v2 query option has defaulted to true since Impala 3.
>>
>>
>> Impala is always open to new developers, and we welcome feedback on these
>> plans. Further discussion of other changes and deprecations is ongoing at
>> dev@impala.apache.org.
>>
>>
>> Thanks,
>>
>> Joe McDonnell
>>
>>
Re: Impala 4 Breaking Changes
Posted by Jim Apple <ap...@jbapple.com>.
Oh, now I see this email against cutting a fresh branch. I'll digest this
and respond later.
On Thu, Jun 3, 2021 at 2:10 AM Quanlong Huang <hu...@gmail.com>
wrote:
> Sorry to be late here! I'm -1 for merging HEAD into 4.0. Because it will
> bump the CDP_BUILD_NUMBER again which increases the difficulty for 3.x
> users (who may still be using Hive 2.x or 3.x) to adopt Impala-4.0. There
> are two new commits in HEAD that bump the CDP_BUILD_NUMBER.
>
> We may have already depended on some Hive 4.0 APIs. But those features may
> not be needed by 3.x users. They can revert some changes to make Impala-4.0
> compatible with older versions of Hive.
> The lower GBN we have, the easier it is for them to upgrade.
>
> Does those good commits of HEAD depend on bumping the GBN? If not, we can
> just pick them to the 4.0 branch.
>
> Thanks,
> Quanlong
>
> On Wed, May 26, 2021 at 8:52 PM Zoltán Borók-Nagy <bo...@apache.org>
> wrote:
>
> > There's been a handful of good commits since May 1, so +1 for merging
> HEAD
> > into 4.0.
> >
> > Cheers,
> > Zoltan
> >
> > On Wed, May 26, 2021 at 2:11 PM Jim Apple <jb...@apache.org> wrote:
> >
> > > We're only waiting on one patch for the docs, but since I cut the 4.0
> > > branch on May 1, a number of things have landed. Maybe it would be
> best
> > to
> > > merge HEAD into 4.0.0 once the Sentry doc change lands, thereby
> including
> > > everything we've talked about cherrypicking -- including 9690.
> > >
> > > Thoughts?
> > >
> > > On Wed, May 26, 2021 at 12:40 AM Quanlong Huang <
> huangquanlong@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I see IMPALA-9690 is done but still missing in branch-4.0.0 and I
> think
> > > we
> > > > should pick it
> > > >
> > > > - c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2 for
> > x86_64
> > > >
> > > > I'd be good to also carry on these fixes for query retry:
> > > >
> > > > - 42684b4 IMPALA-10413: fix impalad crashes when canceling the
> > > retrying
> > > > query
> > > > - d111443 IMPALA-10704: Fix retried query id not being
> unregistered
> > > when
> > > > retry fails
> > > >
> > > > If you are ok, I can push them to branch-4.0.0.
> > > >
> > > > Thanks,
> > > > Quanlong
> > > >
> > > > On Sun, May 16, 2021 at 6:02 AM Jim Apple <jb...@apache.org>
> wrote:
> > > >
> > > > > Sounds good!
> > > > >
> > > > > On Sat, May 15, 2021 at 1:29 PM Joe McDonnell <
> > > joemcdonnell@cloudera.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jim,
> > > > > >
> > > > > > I talked with the developer for the Sentry docs bug this past
> week,
> > > and
> > > > > we
> > > > > > went through the list of changes needed for that. The developer
> is
> > > > hoping
> > > > > > to work on it next week.
> > > > > >
> > > > > > If there is any further delay, I'm open to options of working
> > around
> > > > it.
> > > > > >
> > > > > > Thanks,
> > > > > > Joe
> > > > > >
> > > > > > On Sat, May 15, 2021 at 12:43 PM Jim Apple <jb...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > This Sentry one I'm concerned about. The ticket has been open
> > for a
> > > > > year.
> > > > > > > We could do a 4.0 release without the docs, or we could maybe
> > put a
> > > > > note
> > > > > > on
> > > > > > > the docs that Sentry support is no longer available.
> > > > > > >
> > > > > > > Any other ideas on how to tackle this while the ticket waits
> for
> > a
> > > > > patch?
> > > > > > >
> > > > > > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <
> > > > > joemcdonnell@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I agree that it is a good idea for us to get going on the 4.0
> > > > > release.
> > > > > > > >
> > > > > > > > I'm working on a change to fix IMPALA-9690. I will post an
> > > updated
> > > > > > review
> > > > > > > > once a test run completes. If everything is looking good, I
> > would
> > > > > like
> > > > > > to
> > > > > > > > get this in the 4.0 release.
> > > > > > > >
> > > > > > > > There is also one documentation fix that I think belongs in
> the
> > > > > Impala
> > > > > > > 4.0
> > > > > > > > release:
> > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove
> > > Sentry
> > > > > > > > references
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Joe
> > > > > > > >
> > > > > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <apache@jbapple.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Sure thing! Let us know if you find owners for anything
> that
> > > > > someone
> > > > > > > > wants
> > > > > > > > > to champion as a 4.0 blocker.
> > > > > > > > >
> > > > > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> > > > > > huangquanlong@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank Jim! I planned to go through all the JIRAs target
> at
> > > 4.0:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > > > > > > >
> > > > > > > > > > But due to some other urgent works and the recent
> holiday,
> > my
> > > > > > > progress
> > > > > > > > is
> > > > > > > > > > slow. Anyway, I think it's time to branch out. We've been
> > > > waiting
> > > > > > too
> > > > > > > > > long.
> > > > > > > > > > Thanks for creating the branch.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Quanlong
> > > > > > > > > >
> > > > > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <
> > apache@jbapple.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I've created a 4.0 branch (with all changes through the
> > end
> > > > of
> > > > > > > April)
> > > > > > > > > and
> > > > > > > > > > > started testing a release candidate. Everyone who wants
> > to
> > > > > vote,
> > > > > > > > > > >
> > > > > >
> https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > > > > > > is a
> > > > > > > > > > > guide to both releasing and voting (at the bottom).
> I'll
> > > let
> > > > > you
> > > > > > > all
> > > > > > > > > know
> > > > > > > > > > > when I have something to vote on.
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > > Jim
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > > > > > > csringhofer@cloudera.com
> > > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > > > > > > My preferred solution would be to deprecate support
> for
> > > x64
> > > > > > > without
> > > > > > > > > > AVX2
> > > > > > > > > > > in
> > > > > > > > > > > > 4.0, but not start removing the related logic yet.
> > > > > > > > > > > > - We could even add a DCHECK + flag to crash by
> default
> > > if
> > > > no
> > > > > > > AVX2
> > > > > > > > is
> > > > > > > > > > > > detected, and a message that points them to Impala
> > > mailing
> > > > > > > > > > lists/related
> > > > > > > > > > > > Jira.
> > > > > > > > > > > > - After 4.0 is released people would be free to
> > implement
> > > > > > things
> > > > > > > > that
> > > > > > > > > > > > assume AVX2 support on x64, for example compiling
> > > > Impala/the
> > > > > > > whole
> > > > > > > > > > > > toolchain with -mavx2. I think that currently people
> > are
> > > > held
> > > > > > > back
> > > > > > > > > from
> > > > > > > > > > > > doing this - why put effort in doing+benchmarking a
> > > change
> > > > if
> > > > > > you
> > > > > > > > > can't
> > > > > > > > > > > > push it to master.
> > > > > > > > > > > > - If a user appears with a valid use case that needs
> > x64
> > > > > > without
> > > > > > > > AVX2
> > > > > > > > > > > then
> > > > > > > > > > > > we can revise our plans, but we shouldn't ignore low
> > > > hanging
> > > > > > > fruits
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > sake of imaginary users :)
> > > > > > > > > > > >
> > > > > > > > > > > > -Csaba
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > > > > > > shant@superdupershant.com
> > > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > +1
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding IMPALA-9690 if someone has the time to
> > clean
> > > up
> > > > > the
> > > > > > > old
> > > > > > > > > > > > > conditional cpu logic it would be great, but if the
> > ARM
> > > > > folks
> > > > > > > > still
> > > > > > > > > > > > > need some of that logic it's not critical enough to
> > > > block,
> > > > > > > just a
> > > > > > > > > > very
> > > > > > > > > > > > > nice to have?
> > > > > > > > > > > > >
> > > > > > > > > > > > > -Shant
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I agree that it would be good to get this done
> even
> > > if
> > > > > it's
> > > > > > > not
> > > > > > > > > > > > perfect,
> > > > > > > > > > > > > > there's so much good stuff.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I agree that we should wrap up Impala 4. A
> large
> > > > amount
> > > > > > of
> > > > > > > > good
> > > > > > > > > > > work
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We need to decide which breaking changes are
> > truly
> > > > > > blockers
> > > > > > > > for
> > > > > > > > > > an
> > > > > > > > > > > > > Impala 4
> > > > > > > > > > > > > > > release. My feeling is that we can't hold the
> > > release
> > > > > for
> > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > breaking changes unless someone signs up to do
> > > those
> > > > > > > changes.
> > > > > > > > > > > > Breaking
> > > > > > > > > > > > > > > compatibility is useful sometimes, but it can't
> > be
> > > an
> > > > > > > > > indefinite
> > > > > > > > > > > hold
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > releases. I think a release discussion thread
> is
> > a
> > > > good
> > > > > > way
> > > > > > > > to
> > > > > > > > > > > > > stimulate
> > > > > > > > > > > > > > > this discussion.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Apart from the two JIRAs you listed, one other
> > > change
> > > > > > > > mentioned
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > original email is switching to use the new
> > > on-demand
> > > > > > > metadata
> > > > > > > > > by
> > > > > > > > > > > > > default.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Joe
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > > > > > > > huangquanlong@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Sure. If there are no objections, I'll raise
> a
> > > > > > discussion
> > > > > > > > > > thread
> > > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > 4.0 release. There are still some unresolved
> > > > breaking
> > > > > > > > > changes,
> > > > > > > > > > > e.g.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the default
> file
> > > > > format
> > > > > > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU
> > > > > requirements
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > > > > > > > jbapple@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for bringing this up again,
> Quanlong!
> > I
> > > > > would
> > > > > > > love
> > > > > > > > > to
> > > > > > > > > > > see
> > > > > > > > > > > > a
> > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > > release soon so that 4.1 releases and what
> > not
> > > > can
> > > > > be
> > > > > > > > > > prepared.
> > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > changlist from the 3.x line is going to be
> > > > enormous
> > > > > > and
> > > > > > > > > could
> > > > > > > > > > > be
> > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > users to digest.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong
> > Huang
> > > <
> > > > > > > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Reviving this thread. I think it's time
> to
> > > > create
> > > > > > the
> > > > > > > > > 4.0.0
> > > > > > > > > > > > > branch
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > prepare for the release now. Any breaking
> > > > change
> > > > > is
> > > > > > > > > landing
> > > > > > > > > > > and
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > wait?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim
> > > Armstrong <
> > > > > > > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Another thing that we're looking into
> is
> > > > > changing
> > > > > > > the
> > > > > > > > > > > encoded
> > > > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > > profile representation to be more
> > > efficient -
> > > > > see
> > > > > > > > > > > > > > > > > > >
> > > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> > > > > > > > So
> > > > > > > > > > that
> > > > > > > > > > > > > might
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > default we will try to change in Impala
> > > 4.0.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim
> > > > Armstrong
> > > > > <
> > > > > > > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > An addendum to this - we're also
> > > > considering
> > > > > > > > whether
> > > > > > > > > to
> > > > > > > > > > > > > increase
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > minimum CPU version so that we can
> > > simplify
> > > > > > > > > development
> > > > > > > > > > > and
> > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > optimising for the most common CPUs -
> > see
> > > > > > > > > > > > > > > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Running on commodity hardware is
> > > important
> > > > > and
> > > > > > we
> > > > > > > > > want
> > > > > > > > > > > > > people to
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > > to develop on whatever machine they
> > have
> > > > > access
> > > > > > > to.
> > > > > > > > > At
> > > > > > > > > > > most
> > > > > > > > > > > > > we're
> > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > going to require AVX2 support, which
> > has
> > > > been
> > > > > > > > > supported
> > > > > > > > > > > by
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > vast
> > > > > > > > > > > > > > > > > > > > majority of CPUs for the better part
> > of a
> > > > > > decade.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe
> > > > > McDonnell <
> > > > > > > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >> Impala is starting development on
> > Impala
> > > > > 4.0.
> > > > > > > > There
> > > > > > > > > > are
> > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > exciting
> > > > > > > > > > > > > > > > > > > >> projects underway including full
> > support
> > > > for
> > > > > > > Hive
> > > > > > > > > ACID
> > > > > > > > > > > > > tables,
> > > > > > > > > > > > > > > > > > improved
> > > > > > > > > > > > > > > > > > > >> multithreading support, and fault
> > > > tolerance
> > > > > > > > > > > improvements,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > > > > >> also serves as a time to alter
> > behavior
> > > > and
> > > > > > > > platform
> > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > Here
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >> current list of notable breaking
> > changes
> > > > > that
> > > > > > > are
> > > > > > > > > > > planned
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Remove support for older operating
> > > > systems:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Maintaining support for older
> > operating
> > > > > > systems
> > > > > > > > > > > requires a
> > > > > > > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > > > > > > >> amount of effort, especially with
> > > changing
> > > > > > > > security
> > > > > > > > > > > > > > > requirements.
> > > > > > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > > > > >> extra operating system consumes
> > valuable
> > > > > > effort
> > > > > > > > and
> > > > > > > > > > can
> > > > > > > > > > > > > conflict
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > >> support for newer operating systems,
> > so
> > > we
> > > > > > have
> > > > > > > > > > decided
> > > > > > > > > > > to
> > > > > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > >> for some older operating systems.
> > > > > > Specifically,
> > > > > > > we
> > > > > > > > > > plan
> > > > > > > > > > > to
> > > > > > > > > > > > > drop
> > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Each of these is approaching or past
> > its
> > > > end
> > > > > > of
> > > > > > > > > life.
> > > > > > > > > > > > Since
> > > > > > > > > > > > > > > > Centos6
> > > > > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > > >> the only operating system still
> > needing
> > > > > Python
> > > > > > > 2.6
> > > > > > > > > and
> > > > > > > > > > > > > Python
> > > > > > > > > > > > > > > 2.6
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > > > >> past its end of life, Python 2.6
> will
> > no
> > > > > > longer
> > > > > > > be
> > > > > > > > > > > > > supported. In
> > > > > > > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > > > > > > >> Centos 7 support will be focused on
> > > newer
> > > > > > > versions
> > > > > > > > > > such
> > > > > > > > > > > as
> > > > > > > > > > > > > 7.5
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Over the past year and a half,
> > Impala's
> > > > > Ranger
> > > > > > > > > > > > authorization
> > > > > > > > > > > > > > > > > > > >> functionality has achieved parity
> and
> > > > > > surpassed
> > > > > > > > the
> > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > > >> authorization functionality.
> Impala's
> > > > Sentry
> > > > > > > > support
> > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > > > >> effort to test and maintain,
> including
> > > > > > > maintenance
> > > > > > > > > on
> > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > >> address security updates. Given the
> > > > > diminished
> > > > > > > > > > activity
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > > >> community, there is no timeline for
> > > > > addressing
> > > > > > > > these
> > > > > > > > > > > > > security
> > > > > > > > > > > > > > > > > updates.
> > > > > > > > > > > > > > > > > > > >> Unless there is a maintainer for
> > Sentry
> > > > > > support,
> > > > > > > > > > Impala
> > > > > > > > > > > > > plans to
> > > > > > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > >> efforts on Ranger for its
> > authorization
> > > > > system
> > > > > > > and
> > > > > > > > > > drop
> > > > > > > > > > > > > Sentry.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Impala-lzo provides code to allow
> > Impala
> > > > to
> > > > > > read
> > > > > > > > the
> > > > > > > > > > LZO
> > > > > > > > > > > > > > > > compressed
> > > > > > > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which
> is
> > > why
> > > > > this
> > > > > > > > > support
> > > > > > > > > > > is
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > > > > >> directly. The Impala-lzo code
> > interacts
> > > > with
> > > > > > > > > internal
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > code
> > > > > > > > > > > > > > > > > at a
> > > > > > > > > > > > > > > > > > > >> level that is error prone and
> > intricate.
> > > > > Given
> > > > > > > the
> > > > > > > > > low
> > > > > > > > > > > > > adoption
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > LZO
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > >> the other compression options
> > available,
> > > > > > Impala
> > > > > > > > > plans
> > > > > > > > > > to
> > > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > > > > > > >> support along with the low level
> > > interface
> > > > > it
> > > > > > > > used.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> In addition, we also plan to
> deprecate
> > > > > several
> > > > > > > > > > existing
> > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > >> they can be removed in a future
> > release.
> > > > > Here
> > > > > > > is a
> > > > > > > > > > > summary
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Impala will default to using on
> > > demand
> > > > > > > metadata
> > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >> local catalog implementation) as
> > > > > described
> > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > > > > > documentation:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > > > > > > >> The old metadata system is now
> > > > deprecated
> > > > > > and
> > > > > > > > may
> > > > > > > > > > be
> > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Impala is deprecating the Beeswax
> > > > client
> > > > > > > > protocol
> > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > clients
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > >> connect via beeswax_port) in
> favor
> > of
> > > > the
> > > > > > > > > > HiveServer2
> > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Impala is deprecating the old
> > > > > > > decimal_v2=false
> > > > > > > > > > > behavior
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query option
> > has
> > > > > > > defaulted
> > > > > > > > to
> > > > > > > > > > > true
> > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Impala is always open to new
> > developers,
> > > > and
> > > > > > we
> > > > > > > > > > welcome
> > > > > > > > > > > > > feedback
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > >> plans. Further discussion of other
> > > changes
> > > > > and
> > > > > > > > > > > > deprecations
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Quanlong Huang <hu...@gmail.com>.
There are some minor conflicts in cherry-picking IMPALA-7501 due to
missing IMPALA-10613 (Standup HMS thrift server in Catalog). Other commits
can be picked cleanly.
Now I'm testing my private 4.0.0 branch with these commits:
- IMPALA-10683: Skip test 'test_double_precision' for non-HDFS test env
- IMPALA-9690: Set minimum CPU requirement to AVX2 for x86_64
- IMPALA-10413: fix impalad crashes when canceling the retrying query
- IMPALA-5121: Fix AVG() on timestamp col with
use_local_tz_for_unix_timestamp_conversions
- IMPALA-10704: Fix retried query id not being unregistered when retry
fails
- IMPALA-9770: [DOCS] Remove Sentry references in documentation
- IMPALA-7501: Slim down partition metadata in LocalCatalog mode
Jobs are:
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/14039/
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/4296/
Once they pass, I'll push them to branch-4.0.0 and continue the release
process. Let me know if you have any concerns.
Thanks,
Quanlong
On Sat, Jun 5, 2021 at 9:41 AM Quanlong Huang <hu...@gmail.com>
wrote:
> Thank Zoltan for listing these commits!
>
> IMPALA-10413 can be resolved. I marked the fix version as 4.0 since we are
> going to pick it. Also did the same for IMPALA-10704.
>
> Besides that, I think we should also pick the doc fix about removing
> Sentry related stuffs:
>
> - 1a84a14 IMPALA-9770: [DOCS] Remove Sentry references in documentation
>
> IMPALA-7501 was recently resolved. I'd be good if we pick it as well,
> because it fixes the scalability issue of local-catalog mode on large
> tables (avoiding OOM and reducing memory footprints).
>
> - bb30621 IMPALA-7501: Slim down partition metadata in LocalCatalog
> mode
>
> Thanks,
> Quanlong
>
> On Fri, Jun 4, 2021 at 5:47 PM Zoltán Borók-Nagy <bo...@apache.org>
> wrote:
>
>> I didn't have any specifics in mind when I wrote there are good commits on
>> master since May 1.
>> My principle was only that it would be nice to release what's latest,
>> since
>> some time has passed since 4.0 was branched.
>>
>> Now that I looked at the commit list, maybe we could consider adding the
>> followings at least:
>>
>> - IMPALA-10683: Skip test 'test_double_precision' for non-HDFS test env
>> - IMPALA-9690: Set minimum CPU requirement to AVX2 for x86_64
>> - IMPALA-10413: fix impalad crashes when canceling the retrying query
>> - IMPALA-5121: Fix AVG() on timestamp col with
>> use_local_tz_for_unix_timestamp_conversions
>> - IMPALA-10704: Fix retried query id not being unregistered when retry
>> fails
>>
>> IMPALA-9690 <https://issues.apache.org/jira/browse/IMPALA-9690> and
>> IMPALA-5121 <https://issues.apache.org/jira/browse/IMPALA-5121> are even
>> targeted for 4.0 in Jira.
>> IMPALA-10413 <https://issues.apache.org/jira/browse/IMPALA-10413> is
>> targeted for 4.0, but the Jira is not resolved yet. A patch has been
>> already merged, so I asked at the Jira ticket whether we can resolve it.
>> IMPALA-10704 <https://issues.apache.org/jira/browse/IMPALA-10704> is
>> targeted for 4.1, but fixes an issue introduced by IMPALA-10413.
>>
>> Thanks,
>> Zoltan
>>
>>
>> On Fri, Jun 4, 2021 at 5:06 AM Jim Apple <jb...@apache.org> wrote:
>>
>> > Once we know the answer to that last question, I'd be interested in
>> reading
>> > more about your vision for what should go in 4.0, Quanlong --
>> > i.e."everything committed on June 2 except patches X and Y".
>> >
>> > On Thu, Jun 3, 2021 at 2:10 AM Quanlong Huang <hu...@gmail.com>
>> > wrote:
>> >
>> > > Sorry to be late here! I'm -1 for merging HEAD into 4.0. Because it
>> will
>> > > bump the CDP_BUILD_NUMBER again which increases the difficulty for 3.x
>> > > users (who may still be using Hive 2.x or 3.x) to adopt Impala-4.0.
>> There
>> > > are two new commits in HEAD that bump the CDP_BUILD_NUMBER.
>> > >
>> > > We may have already depended on some Hive 4.0 APIs. But those features
>> > may
>> > > not be needed by 3.x users. They can revert some changes to make
>> > Impala-4.0
>> > > compatible with older versions of Hive.
>> > > The lower GBN we have, the easier it is for them to upgrade.
>> > >
>> > > Does those good commits of HEAD depend on bumping the GBN? If not, we
>> can
>> > > just pick them to the 4.0 branch.
>> > >
>> > > Thanks,
>> > > Quanlong
>> > >
>> > > On Wed, May 26, 2021 at 8:52 PM Zoltán Borók-Nagy <
>> boroknagyz@apache.org
>> > >
>> > > wrote:
>> > >
>> > > > There's been a handful of good commits since May 1, so +1 for
>> merging
>> > > HEAD
>> > > > into 4.0.
>> > > >
>> > > > Cheers,
>> > > > Zoltan
>> > > >
>> > > > On Wed, May 26, 2021 at 2:11 PM Jim Apple <jb...@apache.org>
>> wrote:
>> > > >
>> > > > > We're only waiting on one patch for the docs, but since I cut the
>> 4.0
>> > > > > branch on May 1, a number of things have landed. Maybe it would
>> be
>> > > best
>> > > > to
>> > > > > merge HEAD into 4.0.0 once the Sentry doc change lands, thereby
>> > > including
>> > > > > everything we've talked about cherrypicking -- including 9690.
>> > > > >
>> > > > > Thoughts?
>> > > > >
>> > > > > On Wed, May 26, 2021 at 12:40 AM Quanlong Huang <
>> > > huangquanlong@gmail.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > I see IMPALA-9690 is done but still missing in branch-4.0.0 and
>> I
>> > > think
>> > > > > we
>> > > > > > should pick it
>> > > > > >
>> > > > > > - c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2
>> for
>> > > > x86_64
>> > > > > >
>> > > > > > I'd be good to also carry on these fixes for query retry:
>> > > > > >
>> > > > > > - 42684b4 IMPALA-10413: fix impalad crashes when canceling
>> the
>> > > > > retrying
>> > > > > > query
>> > > > > > - d111443 IMPALA-10704: Fix retried query id not being
>> > > unregistered
>> > > > > when
>> > > > > > retry fails
>> > > > > >
>> > > > > > If you are ok, I can push them to branch-4.0.0.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Quanlong
>> > > > > >
>> > > > > > On Sun, May 16, 2021 at 6:02 AM Jim Apple <jb...@apache.org>
>> > > wrote:
>> > > > > >
>> > > > > > > Sounds good!
>> > > > > > >
>> > > > > > > On Sat, May 15, 2021 at 1:29 PM Joe McDonnell <
>> > > > > joemcdonnell@cloudera.com
>> > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Jim,
>> > > > > > > >
>> > > > > > > > I talked with the developer for the Sentry docs bug this
>> past
>> > > week,
>> > > > > and
>> > > > > > > we
>> > > > > > > > went through the list of changes needed for that. The
>> developer
>> > > is
>> > > > > > hoping
>> > > > > > > > to work on it next week.
>> > > > > > > >
>> > > > > > > > If there is any further delay, I'm open to options of
>> working
>> > > > around
>> > > > > > it.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Joe
>> > > > > > > >
>> > > > > > > > On Sat, May 15, 2021 at 12:43 PM Jim Apple <
>> jbapple@apache.org
>> > >
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > This Sentry one I'm concerned about. The ticket has been
>> open
>> > > > for a
>> > > > > > > year.
>> > > > > > > > > We could do a 4.0 release without the docs, or we could
>> maybe
>> > > > put a
>> > > > > > > note
>> > > > > > > > on
>> > > > > > > > > the docs that Sentry support is no longer available.
>> > > > > > > > >
>> > > > > > > > > Any other ideas on how to tackle this while the ticket
>> waits
>> > > for
>> > > > a
>> > > > > > > patch?
>> > > > > > > > >
>> > > > > > > > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <
>> > > > > > > joemcdonnell@cloudera.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > I agree that it is a good idea for us to get going on
>> the
>> > 4.0
>> > > > > > > release.
>> > > > > > > > > >
>> > > > > > > > > > I'm working on a change to fix IMPALA-9690. I will post
>> an
>> > > > > updated
>> > > > > > > > review
>> > > > > > > > > > once a test run completes. If everything is looking
>> good, I
>> > > > would
>> > > > > > > like
>> > > > > > > > to
>> > > > > > > > > > get this in the 4.0 release.
>> > > > > > > > > >
>> > > > > > > > > > There is also one documentation fix that I think
>> belongs in
>> > > the
>> > > > > > > Impala
>> > > > > > > > > 4.0
>> > > > > > > > > > release:
>> > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9770 -
>> Remove
>> > > > > Sentry
>> > > > > > > > > > references
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Joe
>> > > > > > > > > >
>> > > > > > > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <
>> > apache@jbapple.com
>> > > >
>> > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Sure thing! Let us know if you find owners for
>> anything
>> > > that
>> > > > > > > someone
>> > > > > > > > > > wants
>> > > > > > > > > > > to champion as a 4.0 blocker.
>> > > > > > > > > > >
>> > > > > > > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
>> > > > > > > > huangquanlong@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Thank Jim! I planned to go through all the JIRAs
>> target
>> > > at
>> > > > > 4.0:
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
>> > > > > > > > > > > >
>> > > > > > > > > > > > But due to some other urgent works and the recent
>> > > holiday,
>> > > > my
>> > > > > > > > > progress
>> > > > > > > > > > is
>> > > > > > > > > > > > slow. Anyway, I think it's time to branch out. We've
>> > been
>> > > > > > waiting
>> > > > > > > > too
>> > > > > > > > > > > long.
>> > > > > > > > > > > > Thanks for creating the branch.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Regards,
>> > > > > > > > > > > > Quanlong
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <
>> > > > apache@jbapple.com
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > I've created a 4.0 branch (with all changes
>> through
>> > the
>> > > > end
>> > > > > > of
>> > > > > > > > > April)
>> > > > > > > > > > > and
>> > > > > > > > > > > > > started testing a release candidate. Everyone who
>> > wants
>> > > > to
>> > > > > > > vote,
>> > > > > > > > > > > > >
>> > > > > > > >
>> > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
>> > > > > > > > > > is a
>> > > > > > > > > > > > > guide to both releasing and voting (at the
>> bottom).
>> > > I'll
>> > > > > let
>> > > > > > > you
>> > > > > > > > > all
>> > > > > > > > > > > know
>> > > > > > > > > > > > > when I have something to vote on.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks!
>> > > > > > > > > > > > > Jim
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
>> > > > > > > > > > > > csringhofer@cloudera.com
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > About IMPALA-9690 (AVX support):
>> > > > > > > > > > > > > > My preferred solution would be to deprecate
>> support
>> > > for
>> > > > > x64
>> > > > > > > > > without
>> > > > > > > > > > > > AVX2
>> > > > > > > > > > > > > in
>> > > > > > > > > > > > > > 4.0, but not start removing the related logic
>> yet.
>> > > > > > > > > > > > > > - We could even add a DCHECK + flag to crash by
>> > > default
>> > > > > if
>> > > > > > no
>> > > > > > > > > AVX2
>> > > > > > > > > > is
>> > > > > > > > > > > > > > detected, and a message that points them to
>> Impala
>> > > > > mailing
>> > > > > > > > > > > > lists/related
>> > > > > > > > > > > > > > Jira.
>> > > > > > > > > > > > > > - After 4.0 is released people would be free to
>> > > > implement
>> > > > > > > > things
>> > > > > > > > > > that
>> > > > > > > > > > > > > > assume AVX2 support on x64, for example
>> compiling
>> > > > > > Impala/the
>> > > > > > > > > whole
>> > > > > > > > > > > > > > toolchain with -mavx2. I think that currently
>> > people
>> > > > are
>> > > > > > held
>> > > > > > > > > back
>> > > > > > > > > > > from
>> > > > > > > > > > > > > > doing this - why put effort in
>> doing+benchmarking a
>> > > > > change
>> > > > > > if
>> > > > > > > > you
>> > > > > > > > > > > can't
>> > > > > > > > > > > > > > push it to master.
>> > > > > > > > > > > > > > - If a user appears with a valid use case that
>> > needs
>> > > > x64
>> > > > > > > > without
>> > > > > > > > > > AVX2
>> > > > > > > > > > > > > then
>> > > > > > > > > > > > > > we can revise our plans, but we shouldn't ignore
>> > low
>> > > > > > hanging
>> > > > > > > > > fruits
>> > > > > > > > > > > for
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > sake of imaginary users :)
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > -Csaba
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian
>> <
>> > > > > > > > > > > > > shant@superdupershant.com
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > +1
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Regarding IMPALA-9690 if someone has the time
>> to
>> > > > clean
>> > > > > up
>> > > > > > > the
>> > > > > > > > > old
>> > > > > > > > > > > > > > > conditional cpu logic it would be great, but
>> if
>> > the
>> > > > ARM
>> > > > > > > folks
>> > > > > > > > > > still
>> > > > > > > > > > > > > > > need some of that logic it's not critical
>> enough
>> > to
>> > > > > > block,
>> > > > > > > > > just a
>> > > > > > > > > > > > very
>> > > > > > > > > > > > > > > nice to have?
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > -Shant
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
>> > > > > > > > > > > > > tim.g.armstrong@gmail.com>
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > I agree that it would be good to get this
>> done
>> > > even
>> > > > > if
>> > > > > > > it's
>> > > > > > > > > not
>> > > > > > > > > > > > > > perfect,
>> > > > > > > > > > > > > > > > there's so much good stuff.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
>> > > > > > > > > > > > > joemcdonnell@cloudera.com>
>> > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > I agree that we should wrap up Impala 4. A
>> > > large
>> > > > > > amount
>> > > > > > > > of
>> > > > > > > > > > good
>> > > > > > > > > > > > > work
>> > > > > > > > > > > > > > > has
>> > > > > > > > > > > > > > > > > gone in and it belongs in a release.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > We need to decide which breaking changes
>> are
>> > > > truly
>> > > > > > > > blockers
>> > > > > > > > > > for
>> > > > > > > > > > > > an
>> > > > > > > > > > > > > > > Impala 4
>> > > > > > > > > > > > > > > > > release. My feeling is that we can't hold
>> the
>> > > > > release
>> > > > > > > for
>> > > > > > > > > > > > > > compatibility
>> > > > > > > > > > > > > > > > > breaking changes unless someone signs up
>> to
>> > do
>> > > > > those
>> > > > > > > > > changes.
>> > > > > > > > > > > > > > Breaking
>> > > > > > > > > > > > > > > > > compatibility is useful sometimes, but it
>> > can't
>> > > > be
>> > > > > an
>> > > > > > > > > > > indefinite
>> > > > > > > > > > > > > hold
>> > > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > releases. I think a release discussion
>> thread
>> > > is
>> > > > a
>> > > > > > good
>> > > > > > > > way
>> > > > > > > > > > to
>> > > > > > > > > > > > > > > stimulate
>> > > > > > > > > > > > > > > > > this discussion.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Apart from the two JIRAs you listed, one
>> > other
>> > > > > change
>> > > > > > > > > > mentioned
>> > > > > > > > > > > > in
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > original email is switching to use the new
>> > > > > on-demand
>> > > > > > > > > metadata
>> > > > > > > > > > > by
>> > > > > > > > > > > > > > > default.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > > > > Joe
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong
>> > Huang <
>> > > > > > > > > > > > > > huangquanlong@gmail.com
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Sure. If there are no objections, I'll
>> > raise
>> > > a
>> > > > > > > > discussion
>> > > > > > > > > > > > thread
>> > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > 4.0 release. There are still some
>> > unresolved
>> > > > > > breaking
>> > > > > > > > > > > changes,
>> > > > > > > > > > > > > e.g.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the
>> default
>> > > file
>> > > > > > > format
>> > > > > > > > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64
>> CPU
>> > > > > > > requirements
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Thanks
>> > > > > > > > > > > > > > > > > > Quanlong
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim
>> Apple
>> > <
>> > > > > > > > > > > jbapple@apache.org
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Thanks for bringing this up again,
>> > > Quanlong!
>> > > > I
>> > > > > > > would
>> > > > > > > > > love
>> > > > > > > > > > > to
>> > > > > > > > > > > > > see
>> > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > 4.0
>> > > > > > > > > > > > > > > > > > > release soon so that 4.1 releases and
>> > what
>> > > > not
>> > > > > > can
>> > > > > > > be
>> > > > > > > > > > > > prepared.
>> > > > > > > > > > > > > > > Already
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > changlist from the 3.x line is going
>> to
>> > be
>> > > > > > enormous
>> > > > > > > > and
>> > > > > > > > > > > could
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > hard
>> > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > users to digest.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM
>> Quanlong
>> > > > Huang
>> > > > > <
>> > > > > > > > > > > > > > > > > huangquanlong@gmail.com>
>> > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Reviving this thread. I think it's
>> time
>> > > to
>> > > > > > create
>> > > > > > > > the
>> > > > > > > > > > > 4.0.0
>> > > > > > > > > > > > > > > branch
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > prepare for the release now. Any
>> > breaking
>> > > > > > change
>> > > > > > > is
>> > > > > > > > > > > landing
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > should
>> > > > > > > > > > > > > > > > > > > > wait?
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > > > > > > > Quanlong
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim
>> > > > > Armstrong <
>> > > > > > > > > > > > > > > > > tarmstrong@cloudera.com
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Another thing that we're looking
>> into
>> > > is
>> > > > > > > changing
>> > > > > > > > > the
>> > > > > > > > > > > > > encoded
>> > > > > > > > > > > > > > > > > runtime
>> > > > > > > > > > > > > > > > > > > > > profile representation to be more
>> > > > > efficient -
>> > > > > > > see
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
>> > > > > > > > > > So
>> > > > > > > > > > > > that
>> > > > > > > > > > > > > > > might
>> > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > > > > default we will try to change in
>> > Impala
>> > > > > 4.0.
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM
>> Tim
>> > > > > > Armstrong
>> > > > > > > <
>> > > > > > > > > > > > > > > > > > > tarmstrong@cloudera.com>
>> > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > An addendum to this - we're also
>> > > > > > considering
>> > > > > > > > > > whether
>> > > > > > > > > > > to
>> > > > > > > > > > > > > > > increase
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > minimum CPU version so that we
>> can
>> > > > > simplify
>> > > > > > > > > > > development
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > focus
>> > > > > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > > > > > > optimising for the most common
>> > CPUs -
>> > > > see
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Running on commodity hardware is
>> > > > > important
>> > > > > > > and
>> > > > > > > > we
>> > > > > > > > > > > want
>> > > > > > > > > > > > > > > people to
>> > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > able
>> > > > > > > > > > > > > > > > > > > > > > to develop on whatever machine
>> they
>> > > > have
>> > > > > > > access
>> > > > > > > > > to.
>> > > > > > > > > > > At
>> > > > > > > > > > > > > most
>> > > > > > > > > > > > > > > we're
>> > > > > > > > > > > > > > > > > > > > likely
>> > > > > > > > > > > > > > > > > > > > > > going to require AVX2 support,
>> > which
>> > > > has
>> > > > > > been
>> > > > > > > > > > > supported
>> > > > > > > > > > > > > by
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > vast
>> > > > > > > > > > > > > > > > > > > > > > majority of CPUs for the better
>> > part
>> > > > of a
>> > > > > > > > decade.
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM
>> Joe
>> > > > > > > McDonnell <
>> > > > > > > > > > > > > > > > > > > > joemcdonnell@cloudera.com
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >> Impala is starting development
>> on
>> > > > Impala
>> > > > > > > 4.0.
>> > > > > > > > > > There
>> > > > > > > > > > > > are
>> > > > > > > > > > > > > > many
>> > > > > > > > > > > > > > > > > > > exciting
>> > > > > > > > > > > > > > > > > > > > > >> projects underway including
>> full
>> > > > support
>> > > > > > for
>> > > > > > > > > Hive
>> > > > > > > > > > > ACID
>> > > > > > > > > > > > > > > tables,
>> > > > > > > > > > > > > > > > > > > > improved
>> > > > > > > > > > > > > > > > > > > > > >> multithreading support, and
>> fault
>> > > > > > tolerance
>> > > > > > > > > > > > > improvements,
>> > > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > > > > Impala
>> > > > > > > > > > > > > > > > > > > > 4.0
>> > > > > > > > > > > > > > > > > > > > > >> also serves as a time to alter
>> > > > behavior
>> > > > > > and
>> > > > > > > > > > platform
>> > > > > > > > > > > > > > > support.
>> > > > > > > > > > > > > > > > > Here
>> > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > >> current list of notable
>> breaking
>> > > > changes
>> > > > > > > that
>> > > > > > > > > are
>> > > > > > > > > > > > > planned
>> > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > Impala
>> > > > > > > > > > > > > > > > > > > > 4.0:
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Remove support for older
>> operating
>> > > > > > systems:
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Maintaining support for older
>> > > > operating
>> > > > > > > > systems
>> > > > > > > > > > > > > requires a
>> > > > > > > > > > > > > > > > > > > > considerable
>> > > > > > > > > > > > > > > > > > > > > >> amount of effort, especially
>> with
>> > > > > changing
>> > > > > > > > > > security
>> > > > > > > > > > > > > > > > > requirements.
>> > > > > > > > > > > > > > > > > > > Each
>> > > > > > > > > > > > > > > > > > > > > >> extra operating system consumes
>> > > > valuable
>> > > > > > > > effort
>> > > > > > > > > > and
>> > > > > > > > > > > > can
>> > > > > > > > > > > > > > > conflict
>> > > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > >> support for newer operating
>> > systems,
>> > > > so
>> > > > > we
>> > > > > > > > have
>> > > > > > > > > > > > decided
>> > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > remove
>> > > > > > > > > > > > > > > > > > > > > support
>> > > > > > > > > > > > > > > > > > > > > >> for some older operating
>> systems.
>> > > > > > > > Specifically,
>> > > > > > > > > we
>> > > > > > > > > > > > plan
>> > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > drop
>> > > > > > > > > > > > > > > > > > > > support
>> > > > > > > > > > > > > > > > > > > > > for:
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> -
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Centos 6
>> > > > > > > > > > > > > > > > > > > > > >> -
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Ubuntu 14
>> > > > > > > > > > > > > > > > > > > > > >> -
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Debian 8
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Each of these is approaching or
>> > past
>> > > > its
>> > > > > > end
>> > > > > > > > of
>> > > > > > > > > > > life.
>> > > > > > > > > > > > > > Since
>> > > > > > > > > > > > > > > > > > Centos6
>> > > > > > > > > > > > > > > > > > > > was
>> > > > > > > > > > > > > > > > > > > > > >> the only operating system still
>> > > > needing
>> > > > > > > Python
>> > > > > > > > > 2.6
>> > > > > > > > > > > and
>> > > > > > > > > > > > > > > Python
>> > > > > > > > > > > > > > > > > 2.6
>> > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > long
>> > > > > > > > > > > > > > > > > > > > > >> past its end of life, Python
>> 2.6
>> > > will
>> > > > no
>> > > > > > > > longer
>> > > > > > > > > be
>> > > > > > > > > > > > > > > supported. In
>> > > > > > > > > > > > > > > > > > > > > addition,
>> > > > > > > > > > > > > > > > > > > > > >> Centos 7 support will be
>> focused
>> > on
>> > > > > newer
>> > > > > > > > > versions
>> > > > > > > > > > > > such
>> > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > 7.5
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > above.
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Remove support for Sentry:
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Over the past year and a half,
>> > > > Impala's
>> > > > > > > Ranger
>> > > > > > > > > > > > > > authorization
>> > > > > > > > > > > > > > > > > > > > > >> functionality has achieved
>> parity
>> > > and
>> > > > > > > > surpassed
>> > > > > > > > > > the
>> > > > > > > > > > > > > > existing
>> > > > > > > > > > > > > > > > > > Sentry
>> > > > > > > > > > > > > > > > > > > > > >> authorization functionality.
>> > > Impala's
>> > > > > > Sentry
>> > > > > > > > > > support
>> > > > > > > > > > > > > > > requires
>> > > > > > > > > > > > > > > > > > > ongoing
>> > > > > > > > > > > > > > > > > > > > > >> effort to test and maintain,
>> > > including
>> > > > > > > > > maintenance
>> > > > > > > > > > > on
>> > > > > > > > > > > > > > Sentry
>> > > > > > > > > > > > > > > > > > itself
>> > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > >> address security updates. Given
>> > the
>> > > > > > > diminished
>> > > > > > > > > > > > activity
>> > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > Sentry
>> > > > > > > > > > > > > > > > > > > > > >> community, there is no timeline
>> > for
>> > > > > > > addressing
>> > > > > > > > > > these
>> > > > > > > > > > > > > > > security
>> > > > > > > > > > > > > > > > > > > updates.
>> > > > > > > > > > > > > > > > > > > > > >> Unless there is a maintainer
>> for
>> > > > Sentry
>> > > > > > > > support,
>> > > > > > > > > > > > Impala
>> > > > > > > > > > > > > > > plans to
>> > > > > > > > > > > > > > > > > > > focus
>> > > > > > > > > > > > > > > > > > > > > its
>> > > > > > > > > > > > > > > > > > > > > >> efforts on Ranger for its
>> > > > authorization
>> > > > > > > system
>> > > > > > > > > and
>> > > > > > > > > > > > drop
>> > > > > > > > > > > > > > > Sentry.
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Remove support for Impala-lzo:
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Impala-lzo provides code to
>> allow
>> > > > Impala
>> > > > > > to
>> > > > > > > > read
>> > > > > > > > > > the
>> > > > > > > > > > > > LZO
>> > > > > > > > > > > > > > > > > > compressed
>> > > > > > > > > > > > > > > > > > > > > >> tables. LZO is GPL licensed,
>> which
>> > > is
>> > > > > why
>> > > > > > > this
>> > > > > > > > > > > support
>> > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > included
>> > > > > > > > > > > > > > > > > > > > > >> directly. The Impala-lzo code
>> > > > interacts
>> > > > > > with
>> > > > > > > > > > > internal
>> > > > > > > > > > > > > > Impala
>> > > > > > > > > > > > > > > > > code
>> > > > > > > > > > > > > > > > > > > at a
>> > > > > > > > > > > > > > > > > > > > > >> level that is error prone and
>> > > > intricate.
>> > > > > > > Given
>> > > > > > > > > the
>> > > > > > > > > > > low
>> > > > > > > > > > > > > > > adoption
>> > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > LZO
>> > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > >> the other compression options
>> > > > available,
>> > > > > > > > Impala
>> > > > > > > > > > > plans
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > remove
>> > > > > > > > > > > > > > > > > > > > > Impala-lzo
>> > > > > > > > > > > > > > > > > > > > > >> support along with the low
>> level
>> > > > > interface
>> > > > > > > it
>> > > > > > > > > > used.
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Deprecations:
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> In addition, we also plan to
>> > > deprecate
>> > > > > > > several
>> > > > > > > > > > > > existing
>> > > > > > > > > > > > > > > features
>> > > > > > > > > > > > > > > > > > so
>> > > > > > > > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > > > > >> they can be removed in a future
>> > > > release.
>> > > > > > > Here
>> > > > > > > > > is a
>> > > > > > > > > > > > > summary
>> > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > most
>> > > > > > > > > > > > > > > > > > > > > >> notable ones:
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> -
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Impala will default to
>> using on
>> > > > > demand
>> > > > > > > > > metadata
>> > > > > > > > > > > > > > > management
>> > > > > > > > > > > > > > > > > > (i.e.
>> > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > >> local catalog
>> implementation)
>> > as
>> > > > > > > described
>> > > > > > > > in
>> > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > documentation:
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > >
>> > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
>> > > > > > > > > > > > > > > > > > > > > >> The old metadata system is
>> now
>> > > > > > deprecated
>> > > > > > > > and
>> > > > > > > > > > may
>> > > > > > > > > > > > be
>> > > > > > > > > > > > > > > removed
>> > > > > > > > > > > > > > > > > > in a
>> > > > > > > > > > > > > > > > > > > > > >> future release.
>> > > > > > > > > > > > > > > > > > > > > >> -
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Impala is deprecating the
>> > Beeswax
>> > > > > > client
>> > > > > > > > > > protocol
>> > > > > > > > > > > > > (i.e.
>> > > > > > > > > > > > > > > > > clients
>> > > > > > > > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > > > > >> connect via beeswax_port) in
>> > > favor
>> > > > of
>> > > > > > the
>> > > > > > > > > > > > HiveServer2
>> > > > > > > > > > > > > > > client
>> > > > > > > > > > > > > > > > > > > > > protocol.
>> > > > > > > > > > > > > > > > > > > > > >> -
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Impala is deprecating the
>> old
>> > > > > > > > > decimal_v2=false
>> > > > > > > > > > > > > behavior
>> > > > > > > > > > > > > > > from
>> > > > > > > > > > > > > > > > > > > Impala
>> > > > > > > > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query
>> > option
>> > > > has
>> > > > > > > > > defaulted
>> > > > > > > > > > to
>> > > > > > > > > > > > > true
>> > > > > > > > > > > > > > > since
>> > > > > > > > > > > > > > > > > > > Impala
>> > > > > > > > > > > > > > > > > > > > > 3.
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Impala is always open to new
>> > > > developers,
>> > > > > > and
>> > > > > > > > we
>> > > > > > > > > > > > welcome
>> > > > > > > > > > > > > > > feedback
>> > > > > > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > > > > > these
>> > > > > > > > > > > > > > > > > > > > > >> plans. Further discussion of
>> other
>> > > > > changes
>> > > > > > > and
>> > > > > > > > > > > > > > deprecations
>> > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > ongoing
>> > > > > > > > > > > > > > > > > > > > > at
>> > > > > > > > > > > > > > > > > > > > > >> dev@impala.apache.org.
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Thanks,
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >> Joe McDonnell
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Impala 4 Breaking Changes
Posted by Quanlong Huang <hu...@gmail.com>.
Thank Zoltan for listing these commits!
IMPALA-10413 can be resolved. I marked the fix version as 4.0 since we are
going to pick it. Also did the same for IMPALA-10704.
Besides that, I think we should also pick the doc fix about removing Sentry
related stuffs:
- 1a84a14 IMPALA-9770: [DOCS] Remove Sentry references in documentation
IMPALA-7501 was recently resolved. I'd be good if we pick it as well,
because it fixes the scalability issue of local-catalog mode on large
tables (avoiding OOM and reducing memory footprints).
- bb30621 IMPALA-7501: Slim down partition metadata in LocalCatalog mode
Thanks,
Quanlong
On Fri, Jun 4, 2021 at 5:47 PM Zoltán Borók-Nagy <bo...@apache.org>
wrote:
> I didn't have any specifics in mind when I wrote there are good commits on
> master since May 1.
> My principle was only that it would be nice to release what's latest, since
> some time has passed since 4.0 was branched.
>
> Now that I looked at the commit list, maybe we could consider adding the
> followings at least:
>
> - IMPALA-10683: Skip test 'test_double_precision' for non-HDFS test env
> - IMPALA-9690: Set minimum CPU requirement to AVX2 for x86_64
> - IMPALA-10413: fix impalad crashes when canceling the retrying query
> - IMPALA-5121: Fix AVG() on timestamp col with
> use_local_tz_for_unix_timestamp_conversions
> - IMPALA-10704: Fix retried query id not being unregistered when retry
> fails
>
> IMPALA-9690 <https://issues.apache.org/jira/browse/IMPALA-9690> and
> IMPALA-5121 <https://issues.apache.org/jira/browse/IMPALA-5121> are even
> targeted for 4.0 in Jira.
> IMPALA-10413 <https://issues.apache.org/jira/browse/IMPALA-10413> is
> targeted for 4.0, but the Jira is not resolved yet. A patch has been
> already merged, so I asked at the Jira ticket whether we can resolve it.
> IMPALA-10704 <https://issues.apache.org/jira/browse/IMPALA-10704> is
> targeted for 4.1, but fixes an issue introduced by IMPALA-10413.
>
> Thanks,
> Zoltan
>
>
> On Fri, Jun 4, 2021 at 5:06 AM Jim Apple <jb...@apache.org> wrote:
>
> > Once we know the answer to that last question, I'd be interested in
> reading
> > more about your vision for what should go in 4.0, Quanlong --
> > i.e."everything committed on June 2 except patches X and Y".
> >
> > On Thu, Jun 3, 2021 at 2:10 AM Quanlong Huang <hu...@gmail.com>
> > wrote:
> >
> > > Sorry to be late here! I'm -1 for merging HEAD into 4.0. Because it
> will
> > > bump the CDP_BUILD_NUMBER again which increases the difficulty for 3.x
> > > users (who may still be using Hive 2.x or 3.x) to adopt Impala-4.0.
> There
> > > are two new commits in HEAD that bump the CDP_BUILD_NUMBER.
> > >
> > > We may have already depended on some Hive 4.0 APIs. But those features
> > may
> > > not be needed by 3.x users. They can revert some changes to make
> > Impala-4.0
> > > compatible with older versions of Hive.
> > > The lower GBN we have, the easier it is for them to upgrade.
> > >
> > > Does those good commits of HEAD depend on bumping the GBN? If not, we
> can
> > > just pick them to the 4.0 branch.
> > >
> > > Thanks,
> > > Quanlong
> > >
> > > On Wed, May 26, 2021 at 8:52 PM Zoltán Borók-Nagy <
> boroknagyz@apache.org
> > >
> > > wrote:
> > >
> > > > There's been a handful of good commits since May 1, so +1 for merging
> > > HEAD
> > > > into 4.0.
> > > >
> > > > Cheers,
> > > > Zoltan
> > > >
> > > > On Wed, May 26, 2021 at 2:11 PM Jim Apple <jb...@apache.org>
> wrote:
> > > >
> > > > > We're only waiting on one patch for the docs, but since I cut the
> 4.0
> > > > > branch on May 1, a number of things have landed. Maybe it would be
> > > best
> > > > to
> > > > > merge HEAD into 4.0.0 once the Sentry doc change lands, thereby
> > > including
> > > > > everything we've talked about cherrypicking -- including 9690.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > On Wed, May 26, 2021 at 12:40 AM Quanlong Huang <
> > > huangquanlong@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I see IMPALA-9690 is done but still missing in branch-4.0.0 and I
> > > think
> > > > > we
> > > > > > should pick it
> > > > > >
> > > > > > - c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2 for
> > > > x86_64
> > > > > >
> > > > > > I'd be good to also carry on these fixes for query retry:
> > > > > >
> > > > > > - 42684b4 IMPALA-10413: fix impalad crashes when canceling the
> > > > > retrying
> > > > > > query
> > > > > > - d111443 IMPALA-10704: Fix retried query id not being
> > > unregistered
> > > > > when
> > > > > > retry fails
> > > > > >
> > > > > > If you are ok, I can push them to branch-4.0.0.
> > > > > >
> > > > > > Thanks,
> > > > > > Quanlong
> > > > > >
> > > > > > On Sun, May 16, 2021 at 6:02 AM Jim Apple <jb...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Sounds good!
> > > > > > >
> > > > > > > On Sat, May 15, 2021 at 1:29 PM Joe McDonnell <
> > > > > joemcdonnell@cloudera.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jim,
> > > > > > > >
> > > > > > > > I talked with the developer for the Sentry docs bug this past
> > > week,
> > > > > and
> > > > > > > we
> > > > > > > > went through the list of changes needed for that. The
> developer
> > > is
> > > > > > hoping
> > > > > > > > to work on it next week.
> > > > > > > >
> > > > > > > > If there is any further delay, I'm open to options of working
> > > > around
> > > > > > it.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Joe
> > > > > > > >
> > > > > > > > On Sat, May 15, 2021 at 12:43 PM Jim Apple <
> jbapple@apache.org
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > This Sentry one I'm concerned about. The ticket has been
> open
> > > > for a
> > > > > > > year.
> > > > > > > > > We could do a 4.0 release without the docs, or we could
> maybe
> > > > put a
> > > > > > > note
> > > > > > > > on
> > > > > > > > > the docs that Sentry support is no longer available.
> > > > > > > > >
> > > > > > > > > Any other ideas on how to tackle this while the ticket
> waits
> > > for
> > > > a
> > > > > > > patch?
> > > > > > > > >
> > > > > > > > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <
> > > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I agree that it is a good idea for us to get going on the
> > 4.0
> > > > > > > release.
> > > > > > > > > >
> > > > > > > > > > I'm working on a change to fix IMPALA-9690. I will post
> an
> > > > > updated
> > > > > > > > review
> > > > > > > > > > once a test run completes. If everything is looking
> good, I
> > > > would
> > > > > > > like
> > > > > > > > to
> > > > > > > > > > get this in the 4.0 release.
> > > > > > > > > >
> > > > > > > > > > There is also one documentation fix that I think belongs
> in
> > > the
> > > > > > > Impala
> > > > > > > > > 4.0
> > > > > > > > > > release:
> > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9770 -
> Remove
> > > > > Sentry
> > > > > > > > > > references
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Joe
> > > > > > > > > >
> > > > > > > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <
> > apache@jbapple.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Sure thing! Let us know if you find owners for anything
> > > that
> > > > > > > someone
> > > > > > > > > > wants
> > > > > > > > > > > to champion as a 4.0 blocker.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> > > > > > > > huangquanlong@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thank Jim! I planned to go through all the JIRAs
> target
> > > at
> > > > > 4.0:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > > > > > > > > >
> > > > > > > > > > > > But due to some other urgent works and the recent
> > > holiday,
> > > > my
> > > > > > > > > progress
> > > > > > > > > > is
> > > > > > > > > > > > slow. Anyway, I think it's time to branch out. We've
> > been
> > > > > > waiting
> > > > > > > > too
> > > > > > > > > > > long.
> > > > > > > > > > > > Thanks for creating the branch.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Quanlong
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <
> > > > apache@jbapple.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I've created a 4.0 branch (with all changes through
> > the
> > > > end
> > > > > > of
> > > > > > > > > April)
> > > > > > > > > > > and
> > > > > > > > > > > > > started testing a release candidate. Everyone who
> > wants
> > > > to
> > > > > > > vote,
> > > > > > > > > > > > >
> > > > > > > >
> > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > > > > > > > > is a
> > > > > > > > > > > > > guide to both releasing and voting (at the bottom).
> > > I'll
> > > > > let
> > > > > > > you
> > > > > > > > > all
> > > > > > > > > > > know
> > > > > > > > > > > > > when I have something to vote on.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > Jim
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > > > > > > > > csringhofer@cloudera.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > > > > > > > > My preferred solution would be to deprecate
> support
> > > for
> > > > > x64
> > > > > > > > > without
> > > > > > > > > > > > AVX2
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > 4.0, but not start removing the related logic
> yet.
> > > > > > > > > > > > > > - We could even add a DCHECK + flag to crash by
> > > default
> > > > > if
> > > > > > no
> > > > > > > > > AVX2
> > > > > > > > > > is
> > > > > > > > > > > > > > detected, and a message that points them to
> Impala
> > > > > mailing
> > > > > > > > > > > > lists/related
> > > > > > > > > > > > > > Jira.
> > > > > > > > > > > > > > - After 4.0 is released people would be free to
> > > > implement
> > > > > > > > things
> > > > > > > > > > that
> > > > > > > > > > > > > > assume AVX2 support on x64, for example compiling
> > > > > > Impala/the
> > > > > > > > > whole
> > > > > > > > > > > > > > toolchain with -mavx2. I think that currently
> > people
> > > > are
> > > > > > held
> > > > > > > > > back
> > > > > > > > > > > from
> > > > > > > > > > > > > > doing this - why put effort in
> doing+benchmarking a
> > > > > change
> > > > > > if
> > > > > > > > you
> > > > > > > > > > > can't
> > > > > > > > > > > > > > push it to master.
> > > > > > > > > > > > > > - If a user appears with a valid use case that
> > needs
> > > > x64
> > > > > > > > without
> > > > > > > > > > AVX2
> > > > > > > > > > > > > then
> > > > > > > > > > > > > > we can revise our plans, but we shouldn't ignore
> > low
> > > > > > hanging
> > > > > > > > > fruits
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > sake of imaginary users :)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -Csaba
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > > > > > > > > shant@superdupershant.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regarding IMPALA-9690 if someone has the time
> to
> > > > clean
> > > > > up
> > > > > > > the
> > > > > > > > > old
> > > > > > > > > > > > > > > conditional cpu logic it would be great, but if
> > the
> > > > ARM
> > > > > > > folks
> > > > > > > > > > still
> > > > > > > > > > > > > > > need some of that logic it's not critical
> enough
> > to
> > > > > > block,
> > > > > > > > > just a
> > > > > > > > > > > > very
> > > > > > > > > > > > > > > nice to have?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > -Shant
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > > > > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I agree that it would be good to get this
> done
> > > even
> > > > > if
> > > > > > > it's
> > > > > > > > > not
> > > > > > > > > > > > > > perfect,
> > > > > > > > > > > > > > > > there's so much good stuff.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > > > > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I agree that we should wrap up Impala 4. A
> > > large
> > > > > > amount
> > > > > > > > of
> > > > > > > > > > good
> > > > > > > > > > > > > work
> > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We need to decide which breaking changes
> are
> > > > truly
> > > > > > > > blockers
> > > > > > > > > > for
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > Impala 4
> > > > > > > > > > > > > > > > > release. My feeling is that we can't hold
> the
> > > > > release
> > > > > > > for
> > > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > breaking changes unless someone signs up to
> > do
> > > > > those
> > > > > > > > > changes.
> > > > > > > > > > > > > > Breaking
> > > > > > > > > > > > > > > > > compatibility is useful sometimes, but it
> > can't
> > > > be
> > > > > an
> > > > > > > > > > > indefinite
> > > > > > > > > > > > > hold
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > releases. I think a release discussion
> thread
> > > is
> > > > a
> > > > > > good
> > > > > > > > way
> > > > > > > > > > to
> > > > > > > > > > > > > > > stimulate
> > > > > > > > > > > > > > > > > this discussion.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Apart from the two JIRAs you listed, one
> > other
> > > > > change
> > > > > > > > > > mentioned
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > original email is switching to use the new
> > > > > on-demand
> > > > > > > > > metadata
> > > > > > > > > > > by
> > > > > > > > > > > > > > > default.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Joe
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong
> > Huang <
> > > > > > > > > > > > > > huangquanlong@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Sure. If there are no objections, I'll
> > raise
> > > a
> > > > > > > > discussion
> > > > > > > > > > > > thread
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > 4.0 release. There are still some
> > unresolved
> > > > > > breaking
> > > > > > > > > > > changes,
> > > > > > > > > > > > > e.g.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the
> default
> > > file
> > > > > > > format
> > > > > > > > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU
> > > > > > > requirements
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim
> Apple
> > <
> > > > > > > > > > > jbapple@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for bringing this up again,
> > > Quanlong!
> > > > I
> > > > > > > would
> > > > > > > > > love
> > > > > > > > > > > to
> > > > > > > > > > > > > see
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > > > > release soon so that 4.1 releases and
> > what
> > > > not
> > > > > > can
> > > > > > > be
> > > > > > > > > > > > prepared.
> > > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > changlist from the 3.x line is going to
> > be
> > > > > > enormous
> > > > > > > > and
> > > > > > > > > > > could
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > users to digest.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM
> Quanlong
> > > > Huang
> > > > > <
> > > > > > > > > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Reviving this thread. I think it's
> time
> > > to
> > > > > > create
> > > > > > > > the
> > > > > > > > > > > 4.0.0
> > > > > > > > > > > > > > > branch
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > prepare for the release now. Any
> > breaking
> > > > > > change
> > > > > > > is
> > > > > > > > > > > landing
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > wait?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim
> > > > > Armstrong <
> > > > > > > > > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Another thing that we're looking
> into
> > > is
> > > > > > > changing
> > > > > > > > > the
> > > > > > > > > > > > > encoded
> > > > > > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > > > > profile representation to be more
> > > > > efficient -
> > > > > > > see
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> > > > > > > > > > So
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > default we will try to change in
> > Impala
> > > > > 4.0.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM
> Tim
> > > > > > Armstrong
> > > > > > > <
> > > > > > > > > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > An addendum to this - we're also
> > > > > > considering
> > > > > > > > > > whether
> > > > > > > > > > > to
> > > > > > > > > > > > > > > increase
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > minimum CPU version so that we
> can
> > > > > simplify
> > > > > > > > > > > development
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > > optimising for the most common
> > CPUs -
> > > > see
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Running on commodity hardware is
> > > > > important
> > > > > > > and
> > > > > > > > we
> > > > > > > > > > > want
> > > > > > > > > > > > > > > people to
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > > > > to develop on whatever machine
> they
> > > > have
> > > > > > > access
> > > > > > > > > to.
> > > > > > > > > > > At
> > > > > > > > > > > > > most
> > > > > > > > > > > > > > > we're
> > > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > > going to require AVX2 support,
> > which
> > > > has
> > > > > > been
> > > > > > > > > > > supported
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > vast
> > > > > > > > > > > > > > > > > > > > > > majority of CPUs for the better
> > part
> > > > of a
> > > > > > > > decade.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM
> Joe
> > > > > > > McDonnell <
> > > > > > > > > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >> Impala is starting development
> on
> > > > Impala
> > > > > > > 4.0.
> > > > > > > > > > There
> > > > > > > > > > > > are
> > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > exciting
> > > > > > > > > > > > > > > > > > > > > >> projects underway including full
> > > > support
> > > > > > for
> > > > > > > > > Hive
> > > > > > > > > > > ACID
> > > > > > > > > > > > > > > tables,
> > > > > > > > > > > > > > > > > > > > improved
> > > > > > > > > > > > > > > > > > > > > >> multithreading support, and
> fault
> > > > > > tolerance
> > > > > > > > > > > > > improvements,
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > > > > > > >> also serves as a time to alter
> > > > behavior
> > > > > > and
> > > > > > > > > > platform
> > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > > Here
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > >> current list of notable breaking
> > > > changes
> > > > > > > that
> > > > > > > > > are
> > > > > > > > > > > > > planned
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Remove support for older
> operating
> > > > > > systems:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Maintaining support for older
> > > > operating
> > > > > > > > systems
> > > > > > > > > > > > > requires a
> > > > > > > > > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > > > > > > > > >> amount of effort, especially
> with
> > > > > changing
> > > > > > > > > > security
> > > > > > > > > > > > > > > > > requirements.
> > > > > > > > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > > > > > > >> extra operating system consumes
> > > > valuable
> > > > > > > > effort
> > > > > > > > > > and
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > conflict
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > >> support for newer operating
> > systems,
> > > > so
> > > > > we
> > > > > > > > have
> > > > > > > > > > > > decided
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > > > >> for some older operating
> systems.
> > > > > > > > Specifically,
> > > > > > > > > we
> > > > > > > > > > > > plan
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > drop
> > > > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Each of these is approaching or
> > past
> > > > its
> > > > > > end
> > > > > > > > of
> > > > > > > > > > > life.
> > > > > > > > > > > > > > Since
> > > > > > > > > > > > > > > > > > Centos6
> > > > > > > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > > > > >> the only operating system still
> > > > needing
> > > > > > > Python
> > > > > > > > > 2.6
> > > > > > > > > > > and
> > > > > > > > > > > > > > > Python
> > > > > > > > > > > > > > > > > 2.6
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > > > > > >> past its end of life, Python 2.6
> > > will
> > > > no
> > > > > > > > longer
> > > > > > > > > be
> > > > > > > > > > > > > > > supported. In
> > > > > > > > > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > > > > > > > > >> Centos 7 support will be focused
> > on
> > > > > newer
> > > > > > > > > versions
> > > > > > > > > > > > such
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > 7.5
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Over the past year and a half,
> > > > Impala's
> > > > > > > Ranger
> > > > > > > > > > > > > > authorization
> > > > > > > > > > > > > > > > > > > > > >> functionality has achieved
> parity
> > > and
> > > > > > > > surpassed
> > > > > > > > > > the
> > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > > > > >> authorization functionality.
> > > Impala's
> > > > > > Sentry
> > > > > > > > > > support
> > > > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > > > > > >> effort to test and maintain,
> > > including
> > > > > > > > > maintenance
> > > > > > > > > > > on
> > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > >> address security updates. Given
> > the
> > > > > > > diminished
> > > > > > > > > > > > activity
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > > > > >> community, there is no timeline
> > for
> > > > > > > addressing
> > > > > > > > > > these
> > > > > > > > > > > > > > > security
> > > > > > > > > > > > > > > > > > > updates.
> > > > > > > > > > > > > > > > > > > > > >> Unless there is a maintainer for
> > > > Sentry
> > > > > > > > support,
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > plans to
> > > > > > > > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > > > >> efforts on Ranger for its
> > > > authorization
> > > > > > > system
> > > > > > > > > and
> > > > > > > > > > > > drop
> > > > > > > > > > > > > > > Sentry.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Impala-lzo provides code to
> allow
> > > > Impala
> > > > > > to
> > > > > > > > read
> > > > > > > > > > the
> > > > > > > > > > > > LZO
> > > > > > > > > > > > > > > > > > compressed
> > > > > > > > > > > > > > > > > > > > > >> tables. LZO is GPL licensed,
> which
> > > is
> > > > > why
> > > > > > > this
> > > > > > > > > > > support
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > > > > > > >> directly. The Impala-lzo code
> > > > interacts
> > > > > > with
> > > > > > > > > > > internal
> > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > code
> > > > > > > > > > > > > > > > > > > at a
> > > > > > > > > > > > > > > > > > > > > >> level that is error prone and
> > > > intricate.
> > > > > > > Given
> > > > > > > > > the
> > > > > > > > > > > low
> > > > > > > > > > > > > > > adoption
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > LZO
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > >> the other compression options
> > > > available,
> > > > > > > > Impala
> > > > > > > > > > > plans
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > > > > > > > > >> support along with the low level
> > > > > interface
> > > > > > > it
> > > > > > > > > > used.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> In addition, we also plan to
> > > deprecate
> > > > > > > several
> > > > > > > > > > > > existing
> > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > >> they can be removed in a future
> > > > release.
> > > > > > > Here
> > > > > > > > > is a
> > > > > > > > > > > > > summary
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Impala will default to using
> on
> > > > > demand
> > > > > > > > > metadata
> > > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > >> local catalog implementation)
> > as
> > > > > > > described
> > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > documentation:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > > > > > > > > >> The old metadata system is
> now
> > > > > > deprecated
> > > > > > > > and
> > > > > > > > > > may
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Impala is deprecating the
> > Beeswax
> > > > > > client
> > > > > > > > > > protocol
> > > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > > > clients
> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > >> connect via beeswax_port) in
> > > favor
> > > > of
> > > > > > the
> > > > > > > > > > > > HiveServer2
> > > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Impala is deprecating the old
> > > > > > > > > decimal_v2=false
> > > > > > > > > > > > > behavior
> > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query
> > option
> > > > has
> > > > > > > > > defaulted
> > > > > > > > > > to
> > > > > > > > > > > > > true
> > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Impala is always open to new
> > > > developers,
> > > > > > and
> > > > > > > > we
> > > > > > > > > > > > welcome
> > > > > > > > > > > > > > > feedback
> > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > >> plans. Further discussion of
> other
> > > > > changes
> > > > > > > and
> > > > > > > > > > > > > > deprecations
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Zoltán Borók-Nagy <bo...@apache.org>.
I didn't have any specifics in mind when I wrote there are good commits on
master since May 1.
My principle was only that it would be nice to release what's latest, since
some time has passed since 4.0 was branched.
Now that I looked at the commit list, maybe we could consider adding the
followings at least:
- IMPALA-10683: Skip test 'test_double_precision' for non-HDFS test env
- IMPALA-9690: Set minimum CPU requirement to AVX2 for x86_64
- IMPALA-10413: fix impalad crashes when canceling the retrying query
- IMPALA-5121: Fix AVG() on timestamp col with
use_local_tz_for_unix_timestamp_conversions
- IMPALA-10704: Fix retried query id not being unregistered when retry
fails
IMPALA-9690 <https://issues.apache.org/jira/browse/IMPALA-9690> and
IMPALA-5121 <https://issues.apache.org/jira/browse/IMPALA-5121> are even
targeted for 4.0 in Jira.
IMPALA-10413 <https://issues.apache.org/jira/browse/IMPALA-10413> is
targeted for 4.0, but the Jira is not resolved yet. A patch has been
already merged, so I asked at the Jira ticket whether we can resolve it.
IMPALA-10704 <https://issues.apache.org/jira/browse/IMPALA-10704> is
targeted for 4.1, but fixes an issue introduced by IMPALA-10413.
Thanks,
Zoltan
On Fri, Jun 4, 2021 at 5:06 AM Jim Apple <jb...@apache.org> wrote:
> Once we know the answer to that last question, I'd be interested in reading
> more about your vision for what should go in 4.0, Quanlong --
> i.e."everything committed on June 2 except patches X and Y".
>
> On Thu, Jun 3, 2021 at 2:10 AM Quanlong Huang <hu...@gmail.com>
> wrote:
>
> > Sorry to be late here! I'm -1 for merging HEAD into 4.0. Because it will
> > bump the CDP_BUILD_NUMBER again which increases the difficulty for 3.x
> > users (who may still be using Hive 2.x or 3.x) to adopt Impala-4.0. There
> > are two new commits in HEAD that bump the CDP_BUILD_NUMBER.
> >
> > We may have already depended on some Hive 4.0 APIs. But those features
> may
> > not be needed by 3.x users. They can revert some changes to make
> Impala-4.0
> > compatible with older versions of Hive.
> > The lower GBN we have, the easier it is for them to upgrade.
> >
> > Does those good commits of HEAD depend on bumping the GBN? If not, we can
> > just pick them to the 4.0 branch.
> >
> > Thanks,
> > Quanlong
> >
> > On Wed, May 26, 2021 at 8:52 PM Zoltán Borók-Nagy <boroknagyz@apache.org
> >
> > wrote:
> >
> > > There's been a handful of good commits since May 1, so +1 for merging
> > HEAD
> > > into 4.0.
> > >
> > > Cheers,
> > > Zoltan
> > >
> > > On Wed, May 26, 2021 at 2:11 PM Jim Apple <jb...@apache.org> wrote:
> > >
> > > > We're only waiting on one patch for the docs, but since I cut the 4.0
> > > > branch on May 1, a number of things have landed. Maybe it would be
> > best
> > > to
> > > > merge HEAD into 4.0.0 once the Sentry doc change lands, thereby
> > including
> > > > everything we've talked about cherrypicking -- including 9690.
> > > >
> > > > Thoughts?
> > > >
> > > > On Wed, May 26, 2021 at 12:40 AM Quanlong Huang <
> > huangquanlong@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I see IMPALA-9690 is done but still missing in branch-4.0.0 and I
> > think
> > > > we
> > > > > should pick it
> > > > >
> > > > > - c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2 for
> > > x86_64
> > > > >
> > > > > I'd be good to also carry on these fixes for query retry:
> > > > >
> > > > > - 42684b4 IMPALA-10413: fix impalad crashes when canceling the
> > > > retrying
> > > > > query
> > > > > - d111443 IMPALA-10704: Fix retried query id not being
> > unregistered
> > > > when
> > > > > retry fails
> > > > >
> > > > > If you are ok, I can push them to branch-4.0.0.
> > > > >
> > > > > Thanks,
> > > > > Quanlong
> > > > >
> > > > > On Sun, May 16, 2021 at 6:02 AM Jim Apple <jb...@apache.org>
> > wrote:
> > > > >
> > > > > > Sounds good!
> > > > > >
> > > > > > On Sat, May 15, 2021 at 1:29 PM Joe McDonnell <
> > > > joemcdonnell@cloudera.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jim,
> > > > > > >
> > > > > > > I talked with the developer for the Sentry docs bug this past
> > week,
> > > > and
> > > > > > we
> > > > > > > went through the list of changes needed for that. The developer
> > is
> > > > > hoping
> > > > > > > to work on it next week.
> > > > > > >
> > > > > > > If there is any further delay, I'm open to options of working
> > > around
> > > > > it.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Joe
> > > > > > >
> > > > > > > On Sat, May 15, 2021 at 12:43 PM Jim Apple <jbapple@apache.org
> >
> > > > wrote:
> > > > > > >
> > > > > > > > This Sentry one I'm concerned about. The ticket has been open
> > > for a
> > > > > > year.
> > > > > > > > We could do a 4.0 release without the docs, or we could maybe
> > > put a
> > > > > > note
> > > > > > > on
> > > > > > > > the docs that Sentry support is no longer available.
> > > > > > > >
> > > > > > > > Any other ideas on how to tackle this while the ticket waits
> > for
> > > a
> > > > > > patch?
> > > > > > > >
> > > > > > > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <
> > > > > > joemcdonnell@cloudera.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I agree that it is a good idea for us to get going on the
> 4.0
> > > > > > release.
> > > > > > > > >
> > > > > > > > > I'm working on a change to fix IMPALA-9690. I will post an
> > > > updated
> > > > > > > review
> > > > > > > > > once a test run completes. If everything is looking good, I
> > > would
> > > > > > like
> > > > > > > to
> > > > > > > > > get this in the 4.0 release.
> > > > > > > > >
> > > > > > > > > There is also one documentation fix that I think belongs in
> > the
> > > > > > Impala
> > > > > > > > 4.0
> > > > > > > > > release:
> > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove
> > > > Sentry
> > > > > > > > > references
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Joe
> > > > > > > > >
> > > > > > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <
> apache@jbapple.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Sure thing! Let us know if you find owners for anything
> > that
> > > > > > someone
> > > > > > > > > wants
> > > > > > > > > > to champion as a 4.0 blocker.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> > > > > > > huangquanlong@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thank Jim! I planned to go through all the JIRAs target
> > at
> > > > 4.0:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > > > > > > > >
> > > > > > > > > > > But due to some other urgent works and the recent
> > holiday,
> > > my
> > > > > > > > progress
> > > > > > > > > is
> > > > > > > > > > > slow. Anyway, I think it's time to branch out. We've
> been
> > > > > waiting
> > > > > > > too
> > > > > > > > > > long.
> > > > > > > > > > > Thanks for creating the branch.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Quanlong
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <
> > > apache@jbapple.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I've created a 4.0 branch (with all changes through
> the
> > > end
> > > > > of
> > > > > > > > April)
> > > > > > > > > > and
> > > > > > > > > > > > started testing a release candidate. Everyone who
> wants
> > > to
> > > > > > vote,
> > > > > > > > > > > >
> > > > > > >
> > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > > > > > > > is a
> > > > > > > > > > > > guide to both releasing and voting (at the bottom).
> > I'll
> > > > let
> > > > > > you
> > > > > > > > all
> > > > > > > > > > know
> > > > > > > > > > > > when I have something to vote on.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > > Jim
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > > > > > > > csringhofer@cloudera.com
> > > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > > > > > > > My preferred solution would be to deprecate support
> > for
> > > > x64
> > > > > > > > without
> > > > > > > > > > > AVX2
> > > > > > > > > > > > in
> > > > > > > > > > > > > 4.0, but not start removing the related logic yet.
> > > > > > > > > > > > > - We could even add a DCHECK + flag to crash by
> > default
> > > > if
> > > > > no
> > > > > > > > AVX2
> > > > > > > > > is
> > > > > > > > > > > > > detected, and a message that points them to Impala
> > > > mailing
> > > > > > > > > > > lists/related
> > > > > > > > > > > > > Jira.
> > > > > > > > > > > > > - After 4.0 is released people would be free to
> > > implement
> > > > > > > things
> > > > > > > > > that
> > > > > > > > > > > > > assume AVX2 support on x64, for example compiling
> > > > > Impala/the
> > > > > > > > whole
> > > > > > > > > > > > > toolchain with -mavx2. I think that currently
> people
> > > are
> > > > > held
> > > > > > > > back
> > > > > > > > > > from
> > > > > > > > > > > > > doing this - why put effort in doing+benchmarking a
> > > > change
> > > > > if
> > > > > > > you
> > > > > > > > > > can't
> > > > > > > > > > > > > push it to master.
> > > > > > > > > > > > > - If a user appears with a valid use case that
> needs
> > > x64
> > > > > > > without
> > > > > > > > > AVX2
> > > > > > > > > > > > then
> > > > > > > > > > > > > we can revise our plans, but we shouldn't ignore
> low
> > > > > hanging
> > > > > > > > fruits
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > sake of imaginary users :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > -Csaba
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > > > > > > > shant@superdupershant.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > +1
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding IMPALA-9690 if someone has the time to
> > > clean
> > > > up
> > > > > > the
> > > > > > > > old
> > > > > > > > > > > > > > conditional cpu logic it would be great, but if
> the
> > > ARM
> > > > > > folks
> > > > > > > > > still
> > > > > > > > > > > > > > need some of that logic it's not critical enough
> to
> > > > > block,
> > > > > > > > just a
> > > > > > > > > > > very
> > > > > > > > > > > > > > nice to have?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -Shant
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > > > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I agree that it would be good to get this done
> > even
> > > > if
> > > > > > it's
> > > > > > > > not
> > > > > > > > > > > > > perfect,
> > > > > > > > > > > > > > > there's so much good stuff.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > > > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I agree that we should wrap up Impala 4. A
> > large
> > > > > amount
> > > > > > > of
> > > > > > > > > good
> > > > > > > > > > > > work
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We need to decide which breaking changes are
> > > truly
> > > > > > > blockers
> > > > > > > > > for
> > > > > > > > > > > an
> > > > > > > > > > > > > > Impala 4
> > > > > > > > > > > > > > > > release. My feeling is that we can't hold the
> > > > release
> > > > > > for
> > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > breaking changes unless someone signs up to
> do
> > > > those
> > > > > > > > changes.
> > > > > > > > > > > > > Breaking
> > > > > > > > > > > > > > > > compatibility is useful sometimes, but it
> can't
> > > be
> > > > an
> > > > > > > > > > indefinite
> > > > > > > > > > > > hold
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > releases. I think a release discussion thread
> > is
> > > a
> > > > > good
> > > > > > > way
> > > > > > > > > to
> > > > > > > > > > > > > > stimulate
> > > > > > > > > > > > > > > > this discussion.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Apart from the two JIRAs you listed, one
> other
> > > > change
> > > > > > > > > mentioned
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > original email is switching to use the new
> > > > on-demand
> > > > > > > > metadata
> > > > > > > > > > by
> > > > > > > > > > > > > > default.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Joe
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong
> Huang <
> > > > > > > > > > > > > huangquanlong@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Sure. If there are no objections, I'll
> raise
> > a
> > > > > > > discussion
> > > > > > > > > > > thread
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > 4.0 release. There are still some
> unresolved
> > > > > breaking
> > > > > > > > > > changes,
> > > > > > > > > > > > e.g.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the default
> > file
> > > > > > format
> > > > > > > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU
> > > > > > requirements
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple
> <
> > > > > > > > > > jbapple@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for bringing this up again,
> > Quanlong!
> > > I
> > > > > > would
> > > > > > > > love
> > > > > > > > > > to
> > > > > > > > > > > > see
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > > > release soon so that 4.1 releases and
> what
> > > not
> > > > > can
> > > > > > be
> > > > > > > > > > > prepared.
> > > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > changlist from the 3.x line is going to
> be
> > > > > enormous
> > > > > > > and
> > > > > > > > > > could
> > > > > > > > > > > > be
> > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > users to digest.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong
> > > Huang
> > > > <
> > > > > > > > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Reviving this thread. I think it's time
> > to
> > > > > create
> > > > > > > the
> > > > > > > > > > 4.0.0
> > > > > > > > > > > > > > branch
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > prepare for the release now. Any
> breaking
> > > > > change
> > > > > > is
> > > > > > > > > > landing
> > > > > > > > > > > > and
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > wait?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim
> > > > Armstrong <
> > > > > > > > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Another thing that we're looking into
> > is
> > > > > > changing
> > > > > > > > the
> > > > > > > > > > > > encoded
> > > > > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > > > profile representation to be more
> > > > efficient -
> > > > > > see
> > > > > > > > > > > > > > > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> > > > > > > > > So
> > > > > > > > > > > that
> > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > default we will try to change in
> Impala
> > > > 4.0.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim
> > > > > Armstrong
> > > > > > <
> > > > > > > > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > An addendum to this - we're also
> > > > > considering
> > > > > > > > > whether
> > > > > > > > > > to
> > > > > > > > > > > > > > increase
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > minimum CPU version so that we can
> > > > simplify
> > > > > > > > > > development
> > > > > > > > > > > > and
> > > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > optimising for the most common
> CPUs -
> > > see
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Running on commodity hardware is
> > > > important
> > > > > > and
> > > > > > > we
> > > > > > > > > > want
> > > > > > > > > > > > > > people to
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > > > to develop on whatever machine they
> > > have
> > > > > > access
> > > > > > > > to.
> > > > > > > > > > At
> > > > > > > > > > > > most
> > > > > > > > > > > > > > we're
> > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > going to require AVX2 support,
> which
> > > has
> > > > > been
> > > > > > > > > > supported
> > > > > > > > > > > > by
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > vast
> > > > > > > > > > > > > > > > > > > > > majority of CPUs for the better
> part
> > > of a
> > > > > > > decade.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe
> > > > > > McDonnell <
> > > > > > > > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >> Impala is starting development on
> > > Impala
> > > > > > 4.0.
> > > > > > > > > There
> > > > > > > > > > > are
> > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > exciting
> > > > > > > > > > > > > > > > > > > > >> projects underway including full
> > > support
> > > > > for
> > > > > > > > Hive
> > > > > > > > > > ACID
> > > > > > > > > > > > > > tables,
> > > > > > > > > > > > > > > > > > > improved
> > > > > > > > > > > > > > > > > > > > >> multithreading support, and fault
> > > > > tolerance
> > > > > > > > > > > > improvements,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > > > > > >> also serves as a time to alter
> > > behavior
> > > > > and
> > > > > > > > > platform
> > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > Here
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > >> current list of notable breaking
> > > changes
> > > > > > that
> > > > > > > > are
> > > > > > > > > > > > planned
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Remove support for older operating
> > > > > systems:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Maintaining support for older
> > > operating
> > > > > > > systems
> > > > > > > > > > > > requires a
> > > > > > > > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > > > > > > > >> amount of effort, especially with
> > > > changing
> > > > > > > > > security
> > > > > > > > > > > > > > > > requirements.
> > > > > > > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > > > > > >> extra operating system consumes
> > > valuable
> > > > > > > effort
> > > > > > > > > and
> > > > > > > > > > > can
> > > > > > > > > > > > > > conflict
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > >> support for newer operating
> systems,
> > > so
> > > > we
> > > > > > > have
> > > > > > > > > > > decided
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > > >> for some older operating systems.
> > > > > > > Specifically,
> > > > > > > > we
> > > > > > > > > > > plan
> > > > > > > > > > > > to
> > > > > > > > > > > > > > drop
> > > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Each of these is approaching or
> past
> > > its
> > > > > end
> > > > > > > of
> > > > > > > > > > life.
> > > > > > > > > > > > > Since
> > > > > > > > > > > > > > > > > Centos6
> > > > > > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > > > >> the only operating system still
> > > needing
> > > > > > Python
> > > > > > > > 2.6
> > > > > > > > > > and
> > > > > > > > > > > > > > Python
> > > > > > > > > > > > > > > > 2.6
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > > > > >> past its end of life, Python 2.6
> > will
> > > no
> > > > > > > longer
> > > > > > > > be
> > > > > > > > > > > > > > supported. In
> > > > > > > > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > > > > > > > >> Centos 7 support will be focused
> on
> > > > newer
> > > > > > > > versions
> > > > > > > > > > > such
> > > > > > > > > > > > as
> > > > > > > > > > > > > > 7.5
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Over the past year and a half,
> > > Impala's
> > > > > > Ranger
> > > > > > > > > > > > > authorization
> > > > > > > > > > > > > > > > > > > > >> functionality has achieved parity
> > and
> > > > > > > surpassed
> > > > > > > > > the
> > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > > > >> authorization functionality.
> > Impala's
> > > > > Sentry
> > > > > > > > > support
> > > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > > > > >> effort to test and maintain,
> > including
> > > > > > > > maintenance
> > > > > > > > > > on
> > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > >> address security updates. Given
> the
> > > > > > diminished
> > > > > > > > > > > activity
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > > > >> community, there is no timeline
> for
> > > > > > addressing
> > > > > > > > > these
> > > > > > > > > > > > > > security
> > > > > > > > > > > > > > > > > > updates.
> > > > > > > > > > > > > > > > > > > > >> Unless there is a maintainer for
> > > Sentry
> > > > > > > support,
> > > > > > > > > > > Impala
> > > > > > > > > > > > > > plans to
> > > > > > > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > > >> efforts on Ranger for its
> > > authorization
> > > > > > system
> > > > > > > > and
> > > > > > > > > > > drop
> > > > > > > > > > > > > > Sentry.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Impala-lzo provides code to allow
> > > Impala
> > > > > to
> > > > > > > read
> > > > > > > > > the
> > > > > > > > > > > LZO
> > > > > > > > > > > > > > > > > compressed
> > > > > > > > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which
> > is
> > > > why
> > > > > > this
> > > > > > > > > > support
> > > > > > > > > > > > is
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > > > > > >> directly. The Impala-lzo code
> > > interacts
> > > > > with
> > > > > > > > > > internal
> > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > code
> > > > > > > > > > > > > > > > > > at a
> > > > > > > > > > > > > > > > > > > > >> level that is error prone and
> > > intricate.
> > > > > > Given
> > > > > > > > the
> > > > > > > > > > low
> > > > > > > > > > > > > > adoption
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > LZO
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > >> the other compression options
> > > available,
> > > > > > > Impala
> > > > > > > > > > plans
> > > > > > > > > > > to
> > > > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > > > > > > > >> support along with the low level
> > > > interface
> > > > > > it
> > > > > > > > > used.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> In addition, we also plan to
> > deprecate
> > > > > > several
> > > > > > > > > > > existing
> > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > >> they can be removed in a future
> > > release.
> > > > > > Here
> > > > > > > > is a
> > > > > > > > > > > > summary
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Impala will default to using on
> > > > demand
> > > > > > > > metadata
> > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > >> local catalog implementation)
> as
> > > > > > described
> > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > > > > > documentation:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > > > > > > > >> The old metadata system is now
> > > > > deprecated
> > > > > > > and
> > > > > > > > > may
> > > > > > > > > > > be
> > > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Impala is deprecating the
> Beeswax
> > > > > client
> > > > > > > > > protocol
> > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > > clients
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > >> connect via beeswax_port) in
> > favor
> > > of
> > > > > the
> > > > > > > > > > > HiveServer2
> > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Impala is deprecating the old
> > > > > > > > decimal_v2=false
> > > > > > > > > > > > behavior
> > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query
> option
> > > has
> > > > > > > > defaulted
> > > > > > > > > to
> > > > > > > > > > > > true
> > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Impala is always open to new
> > > developers,
> > > > > and
> > > > > > > we
> > > > > > > > > > > welcome
> > > > > > > > > > > > > > feedback
> > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > >> plans. Further discussion of other
> > > > changes
> > > > > > and
> > > > > > > > > > > > > deprecations
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Jim Apple <jb...@apache.org>.
Once we know the answer to that last question, I'd be interested in reading
more about your vision for what should go in 4.0, Quanlong --
i.e."everything committed on June 2 except patches X and Y".
On Thu, Jun 3, 2021 at 2:10 AM Quanlong Huang <hu...@gmail.com>
wrote:
> Sorry to be late here! I'm -1 for merging HEAD into 4.0. Because it will
> bump the CDP_BUILD_NUMBER again which increases the difficulty for 3.x
> users (who may still be using Hive 2.x or 3.x) to adopt Impala-4.0. There
> are two new commits in HEAD that bump the CDP_BUILD_NUMBER.
>
> We may have already depended on some Hive 4.0 APIs. But those features may
> not be needed by 3.x users. They can revert some changes to make Impala-4.0
> compatible with older versions of Hive.
> The lower GBN we have, the easier it is for them to upgrade.
>
> Does those good commits of HEAD depend on bumping the GBN? If not, we can
> just pick them to the 4.0 branch.
>
> Thanks,
> Quanlong
>
> On Wed, May 26, 2021 at 8:52 PM Zoltán Borók-Nagy <bo...@apache.org>
> wrote:
>
> > There's been a handful of good commits since May 1, so +1 for merging
> HEAD
> > into 4.0.
> >
> > Cheers,
> > Zoltan
> >
> > On Wed, May 26, 2021 at 2:11 PM Jim Apple <jb...@apache.org> wrote:
> >
> > > We're only waiting on one patch for the docs, but since I cut the 4.0
> > > branch on May 1, a number of things have landed. Maybe it would be
> best
> > to
> > > merge HEAD into 4.0.0 once the Sentry doc change lands, thereby
> including
> > > everything we've talked about cherrypicking -- including 9690.
> > >
> > > Thoughts?
> > >
> > > On Wed, May 26, 2021 at 12:40 AM Quanlong Huang <
> huangquanlong@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I see IMPALA-9690 is done but still missing in branch-4.0.0 and I
> think
> > > we
> > > > should pick it
> > > >
> > > > - c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2 for
> > x86_64
> > > >
> > > > I'd be good to also carry on these fixes for query retry:
> > > >
> > > > - 42684b4 IMPALA-10413: fix impalad crashes when canceling the
> > > retrying
> > > > query
> > > > - d111443 IMPALA-10704: Fix retried query id not being
> unregistered
> > > when
> > > > retry fails
> > > >
> > > > If you are ok, I can push them to branch-4.0.0.
> > > >
> > > > Thanks,
> > > > Quanlong
> > > >
> > > > On Sun, May 16, 2021 at 6:02 AM Jim Apple <jb...@apache.org>
> wrote:
> > > >
> > > > > Sounds good!
> > > > >
> > > > > On Sat, May 15, 2021 at 1:29 PM Joe McDonnell <
> > > joemcdonnell@cloudera.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jim,
> > > > > >
> > > > > > I talked with the developer for the Sentry docs bug this past
> week,
> > > and
> > > > > we
> > > > > > went through the list of changes needed for that. The developer
> is
> > > > hoping
> > > > > > to work on it next week.
> > > > > >
> > > > > > If there is any further delay, I'm open to options of working
> > around
> > > > it.
> > > > > >
> > > > > > Thanks,
> > > > > > Joe
> > > > > >
> > > > > > On Sat, May 15, 2021 at 12:43 PM Jim Apple <jb...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > This Sentry one I'm concerned about. The ticket has been open
> > for a
> > > > > year.
> > > > > > > We could do a 4.0 release without the docs, or we could maybe
> > put a
> > > > > note
> > > > > > on
> > > > > > > the docs that Sentry support is no longer available.
> > > > > > >
> > > > > > > Any other ideas on how to tackle this while the ticket waits
> for
> > a
> > > > > patch?
> > > > > > >
> > > > > > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <
> > > > > joemcdonnell@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I agree that it is a good idea for us to get going on the 4.0
> > > > > release.
> > > > > > > >
> > > > > > > > I'm working on a change to fix IMPALA-9690. I will post an
> > > updated
> > > > > > review
> > > > > > > > once a test run completes. If everything is looking good, I
> > would
> > > > > like
> > > > > > to
> > > > > > > > get this in the 4.0 release.
> > > > > > > >
> > > > > > > > There is also one documentation fix that I think belongs in
> the
> > > > > Impala
> > > > > > > 4.0
> > > > > > > > release:
> > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove
> > > Sentry
> > > > > > > > references
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Joe
> > > > > > > >
> > > > > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <apache@jbapple.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Sure thing! Let us know if you find owners for anything
> that
> > > > > someone
> > > > > > > > wants
> > > > > > > > > to champion as a 4.0 blocker.
> > > > > > > > >
> > > > > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> > > > > > huangquanlong@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank Jim! I planned to go through all the JIRAs target
> at
> > > 4.0:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > > > > > > >
> > > > > > > > > > But due to some other urgent works and the recent
> holiday,
> > my
> > > > > > > progress
> > > > > > > > is
> > > > > > > > > > slow. Anyway, I think it's time to branch out. We've been
> > > > waiting
> > > > > > too
> > > > > > > > > long.
> > > > > > > > > > Thanks for creating the branch.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Quanlong
> > > > > > > > > >
> > > > > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <
> > apache@jbapple.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I've created a 4.0 branch (with all changes through the
> > end
> > > > of
> > > > > > > April)
> > > > > > > > > and
> > > > > > > > > > > started testing a release candidate. Everyone who wants
> > to
> > > > > vote,
> > > > > > > > > > >
> > > > > >
> https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > > > > > > is a
> > > > > > > > > > > guide to both releasing and voting (at the bottom).
> I'll
> > > let
> > > > > you
> > > > > > > all
> > > > > > > > > know
> > > > > > > > > > > when I have something to vote on.
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > > Jim
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > > > > > > csringhofer@cloudera.com
> > > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > > > > > > My preferred solution would be to deprecate support
> for
> > > x64
> > > > > > > without
> > > > > > > > > > AVX2
> > > > > > > > > > > in
> > > > > > > > > > > > 4.0, but not start removing the related logic yet.
> > > > > > > > > > > > - We could even add a DCHECK + flag to crash by
> default
> > > if
> > > > no
> > > > > > > AVX2
> > > > > > > > is
> > > > > > > > > > > > detected, and a message that points them to Impala
> > > mailing
> > > > > > > > > > lists/related
> > > > > > > > > > > > Jira.
> > > > > > > > > > > > - After 4.0 is released people would be free to
> > implement
> > > > > > things
> > > > > > > > that
> > > > > > > > > > > > assume AVX2 support on x64, for example compiling
> > > > Impala/the
> > > > > > > whole
> > > > > > > > > > > > toolchain with -mavx2. I think that currently people
> > are
> > > > held
> > > > > > > back
> > > > > > > > > from
> > > > > > > > > > > > doing this - why put effort in doing+benchmarking a
> > > change
> > > > if
> > > > > > you
> > > > > > > > > can't
> > > > > > > > > > > > push it to master.
> > > > > > > > > > > > - If a user appears with a valid use case that needs
> > x64
> > > > > > without
> > > > > > > > AVX2
> > > > > > > > > > > then
> > > > > > > > > > > > we can revise our plans, but we shouldn't ignore low
> > > > hanging
> > > > > > > fruits
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > sake of imaginary users :)
> > > > > > > > > > > >
> > > > > > > > > > > > -Csaba
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > > > > > > shant@superdupershant.com
> > > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > +1
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding IMPALA-9690 if someone has the time to
> > clean
> > > up
> > > > > the
> > > > > > > old
> > > > > > > > > > > > > conditional cpu logic it would be great, but if the
> > ARM
> > > > > folks
> > > > > > > > still
> > > > > > > > > > > > > need some of that logic it's not critical enough to
> > > > block,
> > > > > > > just a
> > > > > > > > > > very
> > > > > > > > > > > > > nice to have?
> > > > > > > > > > > > >
> > > > > > > > > > > > > -Shant
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I agree that it would be good to get this done
> even
> > > if
> > > > > it's
> > > > > > > not
> > > > > > > > > > > > perfect,
> > > > > > > > > > > > > > there's so much good stuff.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I agree that we should wrap up Impala 4. A
> large
> > > > amount
> > > > > > of
> > > > > > > > good
> > > > > > > > > > > work
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We need to decide which breaking changes are
> > truly
> > > > > > blockers
> > > > > > > > for
> > > > > > > > > > an
> > > > > > > > > > > > > Impala 4
> > > > > > > > > > > > > > > release. My feeling is that we can't hold the
> > > release
> > > > > for
> > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > breaking changes unless someone signs up to do
> > > those
> > > > > > > changes.
> > > > > > > > > > > > Breaking
> > > > > > > > > > > > > > > compatibility is useful sometimes, but it can't
> > be
> > > an
> > > > > > > > > indefinite
> > > > > > > > > > > hold
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > releases. I think a release discussion thread
> is
> > a
> > > > good
> > > > > > way
> > > > > > > > to
> > > > > > > > > > > > > stimulate
> > > > > > > > > > > > > > > this discussion.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Apart from the two JIRAs you listed, one other
> > > change
> > > > > > > > mentioned
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > original email is switching to use the new
> > > on-demand
> > > > > > > metadata
> > > > > > > > > by
> > > > > > > > > > > > > default.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Joe
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > > > > > > > huangquanlong@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Sure. If there are no objections, I'll raise
> a
> > > > > > discussion
> > > > > > > > > > thread
> > > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > 4.0 release. There are still some unresolved
> > > > breaking
> > > > > > > > > changes,
> > > > > > > > > > > e.g.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the default
> file
> > > > > format
> > > > > > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU
> > > > > requirements
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > > > > > > > jbapple@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for bringing this up again,
> Quanlong!
> > I
> > > > > would
> > > > > > > love
> > > > > > > > > to
> > > > > > > > > > > see
> > > > > > > > > > > > a
> > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > > release soon so that 4.1 releases and what
> > not
> > > > can
> > > > > be
> > > > > > > > > > prepared.
> > > > > > > > > > > > > Already
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > changlist from the 3.x line is going to be
> > > > enormous
> > > > > > and
> > > > > > > > > could
> > > > > > > > > > > be
> > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > users to digest.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong
> > Huang
> > > <
> > > > > > > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Reviving this thread. I think it's time
> to
> > > > create
> > > > > > the
> > > > > > > > > 4.0.0
> > > > > > > > > > > > > branch
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > prepare for the release now. Any breaking
> > > > change
> > > > > is
> > > > > > > > > landing
> > > > > > > > > > > and
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > wait?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim
> > > Armstrong <
> > > > > > > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Another thing that we're looking into
> is
> > > > > changing
> > > > > > > the
> > > > > > > > > > > encoded
> > > > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > > profile representation to be more
> > > efficient -
> > > > > see
> > > > > > > > > > > > > > > > > > >
> > > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> > > > > > > > So
> > > > > > > > > > that
> > > > > > > > > > > > > might
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > default we will try to change in Impala
> > > 4.0.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim
> > > > Armstrong
> > > > > <
> > > > > > > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > An addendum to this - we're also
> > > > considering
> > > > > > > > whether
> > > > > > > > > to
> > > > > > > > > > > > > increase
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > minimum CPU version so that we can
> > > simplify
> > > > > > > > > development
> > > > > > > > > > > and
> > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > optimising for the most common CPUs -
> > see
> > > > > > > > > > > > > > > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Running on commodity hardware is
> > > important
> > > > > and
> > > > > > we
> > > > > > > > > want
> > > > > > > > > > > > > people to
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > > to develop on whatever machine they
> > have
> > > > > access
> > > > > > > to.
> > > > > > > > > At
> > > > > > > > > > > most
> > > > > > > > > > > > > we're
> > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > going to require AVX2 support, which
> > has
> > > > been
> > > > > > > > > supported
> > > > > > > > > > > by
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > vast
> > > > > > > > > > > > > > > > > > > > majority of CPUs for the better part
> > of a
> > > > > > decade.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe
> > > > > McDonnell <
> > > > > > > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >> Impala is starting development on
> > Impala
> > > > > 4.0.
> > > > > > > > There
> > > > > > > > > > are
> > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > exciting
> > > > > > > > > > > > > > > > > > > >> projects underway including full
> > support
> > > > for
> > > > > > > Hive
> > > > > > > > > ACID
> > > > > > > > > > > > > tables,
> > > > > > > > > > > > > > > > > > improved
> > > > > > > > > > > > > > > > > > > >> multithreading support, and fault
> > > > tolerance
> > > > > > > > > > > improvements,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > > > > >> also serves as a time to alter
> > behavior
> > > > and
> > > > > > > > platform
> > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > Here
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >> current list of notable breaking
> > changes
> > > > > that
> > > > > > > are
> > > > > > > > > > > planned
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Remove support for older operating
> > > > systems:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Maintaining support for older
> > operating
> > > > > > systems
> > > > > > > > > > > requires a
> > > > > > > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > > > > > > >> amount of effort, especially with
> > > changing
> > > > > > > > security
> > > > > > > > > > > > > > > requirements.
> > > > > > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > > > > >> extra operating system consumes
> > valuable
> > > > > > effort
> > > > > > > > and
> > > > > > > > > > can
> > > > > > > > > > > > > conflict
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > >> support for newer operating systems,
> > so
> > > we
> > > > > > have
> > > > > > > > > > decided
> > > > > > > > > > > to
> > > > > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > >> for some older operating systems.
> > > > > > Specifically,
> > > > > > > we
> > > > > > > > > > plan
> > > > > > > > > > > to
> > > > > > > > > > > > > drop
> > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Each of these is approaching or past
> > its
> > > > end
> > > > > > of
> > > > > > > > > life.
> > > > > > > > > > > > Since
> > > > > > > > > > > > > > > > Centos6
> > > > > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > > >> the only operating system still
> > needing
> > > > > Python
> > > > > > > 2.6
> > > > > > > > > and
> > > > > > > > > > > > > Python
> > > > > > > > > > > > > > > 2.6
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > > > >> past its end of life, Python 2.6
> will
> > no
> > > > > > longer
> > > > > > > be
> > > > > > > > > > > > > supported. In
> > > > > > > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > > > > > > >> Centos 7 support will be focused on
> > > newer
> > > > > > > versions
> > > > > > > > > > such
> > > > > > > > > > > as
> > > > > > > > > > > > > 7.5
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Over the past year and a half,
> > Impala's
> > > > > Ranger
> > > > > > > > > > > > authorization
> > > > > > > > > > > > > > > > > > > >> functionality has achieved parity
> and
> > > > > > surpassed
> > > > > > > > the
> > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > > >> authorization functionality.
> Impala's
> > > > Sentry
> > > > > > > > support
> > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > > > >> effort to test and maintain,
> including
> > > > > > > maintenance
> > > > > > > > > on
> > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > >> address security updates. Given the
> > > > > diminished
> > > > > > > > > > activity
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > > >> community, there is no timeline for
> > > > > addressing
> > > > > > > > these
> > > > > > > > > > > > > security
> > > > > > > > > > > > > > > > > updates.
> > > > > > > > > > > > > > > > > > > >> Unless there is a maintainer for
> > Sentry
> > > > > > support,
> > > > > > > > > > Impala
> > > > > > > > > > > > > plans to
> > > > > > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > > >> efforts on Ranger for its
> > authorization
> > > > > system
> > > > > > > and
> > > > > > > > > > drop
> > > > > > > > > > > > > Sentry.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Impala-lzo provides code to allow
> > Impala
> > > > to
> > > > > > read
> > > > > > > > the
> > > > > > > > > > LZO
> > > > > > > > > > > > > > > > compressed
> > > > > > > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which
> is
> > > why
> > > > > this
> > > > > > > > > support
> > > > > > > > > > > is
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > > > > >> directly. The Impala-lzo code
> > interacts
> > > > with
> > > > > > > > > internal
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > code
> > > > > > > > > > > > > > > > > at a
> > > > > > > > > > > > > > > > > > > >> level that is error prone and
> > intricate.
> > > > > Given
> > > > > > > the
> > > > > > > > > low
> > > > > > > > > > > > > adoption
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > LZO
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > >> the other compression options
> > available,
> > > > > > Impala
> > > > > > > > > plans
> > > > > > > > > > to
> > > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > > > > > > >> support along with the low level
> > > interface
> > > > > it
> > > > > > > > used.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> In addition, we also plan to
> deprecate
> > > > > several
> > > > > > > > > > existing
> > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > >> they can be removed in a future
> > release.
> > > > > Here
> > > > > > > is a
> > > > > > > > > > > summary
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Impala will default to using on
> > > demand
> > > > > > > metadata
> > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >> local catalog implementation) as
> > > > > described
> > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > > > > > documentation:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > > > > > > >> The old metadata system is now
> > > > deprecated
> > > > > > and
> > > > > > > > may
> > > > > > > > > > be
> > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Impala is deprecating the Beeswax
> > > > client
> > > > > > > > protocol
> > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > clients
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > >> connect via beeswax_port) in
> favor
> > of
> > > > the
> > > > > > > > > > HiveServer2
> > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Impala is deprecating the old
> > > > > > > decimal_v2=false
> > > > > > > > > > > behavior
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query option
> > has
> > > > > > > defaulted
> > > > > > > > to
> > > > > > > > > > > true
> > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Impala is always open to new
> > developers,
> > > > and
> > > > > > we
> > > > > > > > > > welcome
> > > > > > > > > > > > > feedback
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > >> plans. Further discussion of other
> > > changes
> > > > > and
> > > > > > > > > > > > deprecations
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Quanlong Huang <hu...@gmail.com>.
Sorry to be late here! I'm -1 for merging HEAD into 4.0. Because it will
bump the CDP_BUILD_NUMBER again which increases the difficulty for 3.x
users (who may still be using Hive 2.x or 3.x) to adopt Impala-4.0. There
are two new commits in HEAD that bump the CDP_BUILD_NUMBER.
We may have already depended on some Hive 4.0 APIs. But those features may
not be needed by 3.x users. They can revert some changes to make Impala-4.0
compatible with older versions of Hive.
The lower GBN we have, the easier it is for them to upgrade.
Does those good commits of HEAD depend on bumping the GBN? If not, we can
just pick them to the 4.0 branch.
Thanks,
Quanlong
On Wed, May 26, 2021 at 8:52 PM Zoltán Borók-Nagy <bo...@apache.org>
wrote:
> There's been a handful of good commits since May 1, so +1 for merging HEAD
> into 4.0.
>
> Cheers,
> Zoltan
>
> On Wed, May 26, 2021 at 2:11 PM Jim Apple <jb...@apache.org> wrote:
>
> > We're only waiting on one patch for the docs, but since I cut the 4.0
> > branch on May 1, a number of things have landed. Maybe it would be best
> to
> > merge HEAD into 4.0.0 once the Sentry doc change lands, thereby including
> > everything we've talked about cherrypicking -- including 9690.
> >
> > Thoughts?
> >
> > On Wed, May 26, 2021 at 12:40 AM Quanlong Huang <huangquanlong@gmail.com
> >
> > wrote:
> >
> > > Hi all,
> > >
> > > I see IMPALA-9690 is done but still missing in branch-4.0.0 and I think
> > we
> > > should pick it
> > >
> > > - c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2 for
> x86_64
> > >
> > > I'd be good to also carry on these fixes for query retry:
> > >
> > > - 42684b4 IMPALA-10413: fix impalad crashes when canceling the
> > retrying
> > > query
> > > - d111443 IMPALA-10704: Fix retried query id not being unregistered
> > when
> > > retry fails
> > >
> > > If you are ok, I can push them to branch-4.0.0.
> > >
> > > Thanks,
> > > Quanlong
> > >
> > > On Sun, May 16, 2021 at 6:02 AM Jim Apple <jb...@apache.org> wrote:
> > >
> > > > Sounds good!
> > > >
> > > > On Sat, May 15, 2021 at 1:29 PM Joe McDonnell <
> > joemcdonnell@cloudera.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Jim,
> > > > >
> > > > > I talked with the developer for the Sentry docs bug this past week,
> > and
> > > > we
> > > > > went through the list of changes needed for that. The developer is
> > > hoping
> > > > > to work on it next week.
> > > > >
> > > > > If there is any further delay, I'm open to options of working
> around
> > > it.
> > > > >
> > > > > Thanks,
> > > > > Joe
> > > > >
> > > > > On Sat, May 15, 2021 at 12:43 PM Jim Apple <jb...@apache.org>
> > wrote:
> > > > >
> > > > > > This Sentry one I'm concerned about. The ticket has been open
> for a
> > > > year.
> > > > > > We could do a 4.0 release without the docs, or we could maybe
> put a
> > > > note
> > > > > on
> > > > > > the docs that Sentry support is no longer available.
> > > > > >
> > > > > > Any other ideas on how to tackle this while the ticket waits for
> a
> > > > patch?
> > > > > >
> > > > > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <
> > > > joemcdonnell@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I agree that it is a good idea for us to get going on the 4.0
> > > > release.
> > > > > > >
> > > > > > > I'm working on a change to fix IMPALA-9690. I will post an
> > updated
> > > > > review
> > > > > > > once a test run completes. If everything is looking good, I
> would
> > > > like
> > > > > to
> > > > > > > get this in the 4.0 release.
> > > > > > >
> > > > > > > There is also one documentation fix that I think belongs in the
> > > > Impala
> > > > > > 4.0
> > > > > > > release:
> > > > > > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove
> > Sentry
> > > > > > > references
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Joe
> > > > > > >
> > > > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com>
> > > wrote:
> > > > > > >
> > > > > > > > Sure thing! Let us know if you find owners for anything that
> > > > someone
> > > > > > > wants
> > > > > > > > to champion as a 4.0 blocker.
> > > > > > > >
> > > > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> > > > > huangquanlong@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thank Jim! I planned to go through all the JIRAs target at
> > 4.0:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > > > > > >
> > > > > > > > > But due to some other urgent works and the recent holiday,
> my
> > > > > > progress
> > > > > > > is
> > > > > > > > > slow. Anyway, I think it's time to branch out. We've been
> > > waiting
> > > > > too
> > > > > > > > long.
> > > > > > > > > Thanks for creating the branch.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Quanlong
> > > > > > > > >
> > > > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <
> apache@jbapple.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I've created a 4.0 branch (with all changes through the
> end
> > > of
> > > > > > April)
> > > > > > > > and
> > > > > > > > > > started testing a release candidate. Everyone who wants
> to
> > > > vote,
> > > > > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > > > > > is a
> > > > > > > > > > guide to both releasing and voting (at the bottom). I'll
> > let
> > > > you
> > > > > > all
> > > > > > > > know
> > > > > > > > > > when I have something to vote on.
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > Jim
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > > > > > csringhofer@cloudera.com
> > > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > > > > > My preferred solution would be to deprecate support for
> > x64
> > > > > > without
> > > > > > > > > AVX2
> > > > > > > > > > in
> > > > > > > > > > > 4.0, but not start removing the related logic yet.
> > > > > > > > > > > - We could even add a DCHECK + flag to crash by default
> > if
> > > no
> > > > > > AVX2
> > > > > > > is
> > > > > > > > > > > detected, and a message that points them to Impala
> > mailing
> > > > > > > > > lists/related
> > > > > > > > > > > Jira.
> > > > > > > > > > > - After 4.0 is released people would be free to
> implement
> > > > > things
> > > > > > > that
> > > > > > > > > > > assume AVX2 support on x64, for example compiling
> > > Impala/the
> > > > > > whole
> > > > > > > > > > > toolchain with -mavx2. I think that currently people
> are
> > > held
> > > > > > back
> > > > > > > > from
> > > > > > > > > > > doing this - why put effort in doing+benchmarking a
> > change
> > > if
> > > > > you
> > > > > > > > can't
> > > > > > > > > > > push it to master.
> > > > > > > > > > > - If a user appears with a valid use case that needs
> x64
> > > > > without
> > > > > > > AVX2
> > > > > > > > > > then
> > > > > > > > > > > we can revise our plans, but we shouldn't ignore low
> > > hanging
> > > > > > fruits
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > sake of imaginary users :)
> > > > > > > > > > >
> > > > > > > > > > > -Csaba
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > > > > > shant@superdupershant.com
> > > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > +1
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding IMPALA-9690 if someone has the time to
> clean
> > up
> > > > the
> > > > > > old
> > > > > > > > > > > > conditional cpu logic it would be great, but if the
> ARM
> > > > folks
> > > > > > > still
> > > > > > > > > > > > need some of that logic it's not critical enough to
> > > block,
> > > > > > just a
> > > > > > > > > very
> > > > > > > > > > > > nice to have?
> > > > > > > > > > > >
> > > > > > > > > > > > -Shant
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree that it would be good to get this done even
> > if
> > > > it's
> > > > > > not
> > > > > > > > > > > perfect,
> > > > > > > > > > > > > there's so much good stuff.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I agree that we should wrap up Impala 4. A large
> > > amount
> > > > > of
> > > > > > > good
> > > > > > > > > > work
> > > > > > > > > > > > has
> > > > > > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We need to decide which breaking changes are
> truly
> > > > > blockers
> > > > > > > for
> > > > > > > > > an
> > > > > > > > > > > > Impala 4
> > > > > > > > > > > > > > release. My feeling is that we can't hold the
> > release
> > > > for
> > > > > > > > > > > compatibility
> > > > > > > > > > > > > > breaking changes unless someone signs up to do
> > those
> > > > > > changes.
> > > > > > > > > > > Breaking
> > > > > > > > > > > > > > compatibility is useful sometimes, but it can't
> be
> > an
> > > > > > > > indefinite
> > > > > > > > > > hold
> > > > > > > > > > > > on
> > > > > > > > > > > > > > releases. I think a release discussion thread is
> a
> > > good
> > > > > way
> > > > > > > to
> > > > > > > > > > > > stimulate
> > > > > > > > > > > > > > this discussion.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Apart from the two JIRAs you listed, one other
> > change
> > > > > > > mentioned
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > original email is switching to use the new
> > on-demand
> > > > > > metadata
> > > > > > > > by
> > > > > > > > > > > > default.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Joe
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > > > > > > huangquanlong@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Sure. If there are no objections, I'll raise a
> > > > > discussion
> > > > > > > > > thread
> > > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > 4.0 release. There are still some unresolved
> > > breaking
> > > > > > > > changes,
> > > > > > > > > > e.g.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the default file
> > > > format
> > > > > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU
> > > > requirements
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > > > > > > jbapple@apache.org
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for bringing this up again, Quanlong!
> I
> > > > would
> > > > > > love
> > > > > > > > to
> > > > > > > > > > see
> > > > > > > > > > > a
> > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > release soon so that 4.1 releases and what
> not
> > > can
> > > > be
> > > > > > > > > prepared.
> > > > > > > > > > > > Already
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > changlist from the 3.x line is going to be
> > > enormous
> > > > > and
> > > > > > > > could
> > > > > > > > > > be
> > > > > > > > > > > > hard
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > users to digest.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong
> Huang
> > <
> > > > > > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Reviving this thread. I think it's time to
> > > create
> > > > > the
> > > > > > > > 4.0.0
> > > > > > > > > > > > branch
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > prepare for the release now. Any breaking
> > > change
> > > > is
> > > > > > > > landing
> > > > > > > > > > and
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > wait?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim
> > Armstrong <
> > > > > > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Another thing that we're looking into is
> > > > changing
> > > > > > the
> > > > > > > > > > encoded
> > > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > profile representation to be more
> > efficient -
> > > > see
> > > > > > > > > > > > > > > > > >
> > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> > > > > > > So
> > > > > > > > > that
> > > > > > > > > > > > might
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > default we will try to change in Impala
> > 4.0.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim
> > > Armstrong
> > > > <
> > > > > > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > An addendum to this - we're also
> > > considering
> > > > > > > whether
> > > > > > > > to
> > > > > > > > > > > > increase
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > minimum CPU version so that we can
> > simplify
> > > > > > > > development
> > > > > > > > > > and
> > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > optimising for the most common CPUs -
> see
> > > > > > > > > > > > > > > > > > >
> > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Running on commodity hardware is
> > important
> > > > and
> > > > > we
> > > > > > > > want
> > > > > > > > > > > > people to
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > to develop on whatever machine they
> have
> > > > access
> > > > > > to.
> > > > > > > > At
> > > > > > > > > > most
> > > > > > > > > > > > we're
> > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > going to require AVX2 support, which
> has
> > > been
> > > > > > > > supported
> > > > > > > > > > by
> > > > > > > > > > > > the
> > > > > > > > > > > > > > vast
> > > > > > > > > > > > > > > > > > > majority of CPUs for the better part
> of a
> > > > > decade.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe
> > > > McDonnell <
> > > > > > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >> Impala is starting development on
> Impala
> > > > 4.0.
> > > > > > > There
> > > > > > > > > are
> > > > > > > > > > > many
> > > > > > > > > > > > > > > > exciting
> > > > > > > > > > > > > > > > > > >> projects underway including full
> support
> > > for
> > > > > > Hive
> > > > > > > > ACID
> > > > > > > > > > > > tables,
> > > > > > > > > > > > > > > > > improved
> > > > > > > > > > > > > > > > > > >> multithreading support, and fault
> > > tolerance
> > > > > > > > > > improvements,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > > > >> also serves as a time to alter
> behavior
> > > and
> > > > > > > platform
> > > > > > > > > > > > support.
> > > > > > > > > > > > > > Here
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >> current list of notable breaking
> changes
> > > > that
> > > > > > are
> > > > > > > > > > planned
> > > > > > > > > > > in
> > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Remove support for older operating
> > > systems:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Maintaining support for older
> operating
> > > > > systems
> > > > > > > > > > requires a
> > > > > > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > > > > > >> amount of effort, especially with
> > changing
> > > > > > > security
> > > > > > > > > > > > > > requirements.
> > > > > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > > > >> extra operating system consumes
> valuable
> > > > > effort
> > > > > > > and
> > > > > > > > > can
> > > > > > > > > > > > conflict
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > >> support for newer operating systems,
> so
> > we
> > > > > have
> > > > > > > > > decided
> > > > > > > > > > to
> > > > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > >> for some older operating systems.
> > > > > Specifically,
> > > > > > we
> > > > > > > > > plan
> > > > > > > > > > to
> > > > > > > > > > > > drop
> > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Each of these is approaching or past
> its
> > > end
> > > > > of
> > > > > > > > life.
> > > > > > > > > > > Since
> > > > > > > > > > > > > > > Centos6
> > > > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > >> the only operating system still
> needing
> > > > Python
> > > > > > 2.6
> > > > > > > > and
> > > > > > > > > > > > Python
> > > > > > > > > > > > > > 2.6
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > > >> past its end of life, Python 2.6 will
> no
> > > > > longer
> > > > > > be
> > > > > > > > > > > > supported. In
> > > > > > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > > > > > >> Centos 7 support will be focused on
> > newer
> > > > > > versions
> > > > > > > > > such
> > > > > > > > > > as
> > > > > > > > > > > > 7.5
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Over the past year and a half,
> Impala's
> > > > Ranger
> > > > > > > > > > > authorization
> > > > > > > > > > > > > > > > > > >> functionality has achieved parity and
> > > > > surpassed
> > > > > > > the
> > > > > > > > > > > existing
> > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > >> authorization functionality. Impala's
> > > Sentry
> > > > > > > support
> > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > > >> effort to test and maintain, including
> > > > > > maintenance
> > > > > > > > on
> > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > >> address security updates. Given the
> > > > diminished
> > > > > > > > > activity
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > > >> community, there is no timeline for
> > > > addressing
> > > > > > > these
> > > > > > > > > > > > security
> > > > > > > > > > > > > > > > updates.
> > > > > > > > > > > > > > > > > > >> Unless there is a maintainer for
> Sentry
> > > > > support,
> > > > > > > > > Impala
> > > > > > > > > > > > plans to
> > > > > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > >> efforts on Ranger for its
> authorization
> > > > system
> > > > > > and
> > > > > > > > > drop
> > > > > > > > > > > > Sentry.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Impala-lzo provides code to allow
> Impala
> > > to
> > > > > read
> > > > > > > the
> > > > > > > > > LZO
> > > > > > > > > > > > > > > compressed
> > > > > > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which is
> > why
> > > > this
> > > > > > > > support
> > > > > > > > > > is
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > > > >> directly. The Impala-lzo code
> interacts
> > > with
> > > > > > > > internal
> > > > > > > > > > > Impala
> > > > > > > > > > > > > > code
> > > > > > > > > > > > > > > > at a
> > > > > > > > > > > > > > > > > > >> level that is error prone and
> intricate.
> > > > Given
> > > > > > the
> > > > > > > > low
> > > > > > > > > > > > adoption
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > LZO
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > >> the other compression options
> available,
> > > > > Impala
> > > > > > > > plans
> > > > > > > > > to
> > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > > > > > >> support along with the low level
> > interface
> > > > it
> > > > > > > used.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> In addition, we also plan to deprecate
> > > > several
> > > > > > > > > existing
> > > > > > > > > > > > features
> > > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > >> they can be removed in a future
> release.
> > > > Here
> > > > > > is a
> > > > > > > > > > summary
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Impala will default to using on
> > demand
> > > > > > metadata
> > > > > > > > > > > > management
> > > > > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >> local catalog implementation) as
> > > > described
> > > > > in
> > > > > > > > this
> > > > > > > > > > > > > > > documentation:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > > > > > >> The old metadata system is now
> > > deprecated
> > > > > and
> > > > > > > may
> > > > > > > > > be
> > > > > > > > > > > > removed
> > > > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Impala is deprecating the Beeswax
> > > client
> > > > > > > protocol
> > > > > > > > > > (i.e.
> > > > > > > > > > > > > > clients
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > >> connect via beeswax_port) in favor
> of
> > > the
> > > > > > > > > HiveServer2
> > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Impala is deprecating the old
> > > > > > decimal_v2=false
> > > > > > > > > > behavior
> > > > > > > > > > > > from
> > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query option
> has
> > > > > > defaulted
> > > > > > > to
> > > > > > > > > > true
> > > > > > > > > > > > since
> > > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Impala is always open to new
> developers,
> > > and
> > > > > we
> > > > > > > > > welcome
> > > > > > > > > > > > feedback
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > >> plans. Further discussion of other
> > changes
> > > > and
> > > > > > > > > > > deprecations
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Zoltán Borók-Nagy <bo...@apache.org>.
There's been a handful of good commits since May 1, so +1 for merging HEAD
into 4.0.
Cheers,
Zoltan
On Wed, May 26, 2021 at 2:11 PM Jim Apple <jb...@apache.org> wrote:
> We're only waiting on one patch for the docs, but since I cut the 4.0
> branch on May 1, a number of things have landed. Maybe it would be best to
> merge HEAD into 4.0.0 once the Sentry doc change lands, thereby including
> everything we've talked about cherrypicking -- including 9690.
>
> Thoughts?
>
> On Wed, May 26, 2021 at 12:40 AM Quanlong Huang <hu...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I see IMPALA-9690 is done but still missing in branch-4.0.0 and I think
> we
> > should pick it
> >
> > - c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2 for x86_64
> >
> > I'd be good to also carry on these fixes for query retry:
> >
> > - 42684b4 IMPALA-10413: fix impalad crashes when canceling the
> retrying
> > query
> > - d111443 IMPALA-10704: Fix retried query id not being unregistered
> when
> > retry fails
> >
> > If you are ok, I can push them to branch-4.0.0.
> >
> > Thanks,
> > Quanlong
> >
> > On Sun, May 16, 2021 at 6:02 AM Jim Apple <jb...@apache.org> wrote:
> >
> > > Sounds good!
> > >
> > > On Sat, May 15, 2021 at 1:29 PM Joe McDonnell <
> joemcdonnell@cloudera.com
> > >
> > > wrote:
> > >
> > > > Hi Jim,
> > > >
> > > > I talked with the developer for the Sentry docs bug this past week,
> and
> > > we
> > > > went through the list of changes needed for that. The developer is
> > hoping
> > > > to work on it next week.
> > > >
> > > > If there is any further delay, I'm open to options of working around
> > it.
> > > >
> > > > Thanks,
> > > > Joe
> > > >
> > > > On Sat, May 15, 2021 at 12:43 PM Jim Apple <jb...@apache.org>
> wrote:
> > > >
> > > > > This Sentry one I'm concerned about. The ticket has been open for a
> > > year.
> > > > > We could do a 4.0 release without the docs, or we could maybe put a
> > > note
> > > > on
> > > > > the docs that Sentry support is no longer available.
> > > > >
> > > > > Any other ideas on how to tackle this while the ticket waits for a
> > > patch?
> > > > >
> > > > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <
> > > joemcdonnell@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > I agree that it is a good idea for us to get going on the 4.0
> > > release.
> > > > > >
> > > > > > I'm working on a change to fix IMPALA-9690. I will post an
> updated
> > > > review
> > > > > > once a test run completes. If everything is looking good, I would
> > > like
> > > > to
> > > > > > get this in the 4.0 release.
> > > > > >
> > > > > > There is also one documentation fix that I think belongs in the
> > > Impala
> > > > > 4.0
> > > > > > release:
> > > > > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove
> Sentry
> > > > > > references
> > > > > >
> > > > > > Thanks,
> > > > > > Joe
> > > > > >
> > > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com>
> > wrote:
> > > > > >
> > > > > > > Sure thing! Let us know if you find owners for anything that
> > > someone
> > > > > > wants
> > > > > > > to champion as a 4.0 blocker.
> > > > > > >
> > > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> > > > huangquanlong@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thank Jim! I planned to go through all the JIRAs target at
> 4.0:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > > > > >
> > > > > > > > But due to some other urgent works and the recent holiday, my
> > > > > progress
> > > > > > is
> > > > > > > > slow. Anyway, I think it's time to branch out. We've been
> > waiting
> > > > too
> > > > > > > long.
> > > > > > > > Thanks for creating the branch.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Quanlong
> > > > > > > >
> > > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <apache@jbapple.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > I've created a 4.0 branch (with all changes through the end
> > of
> > > > > April)
> > > > > > > and
> > > > > > > > > started testing a release candidate. Everyone who wants to
> > > vote,
> > > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > > > > is a
> > > > > > > > > guide to both releasing and voting (at the bottom). I'll
> let
> > > you
> > > > > all
> > > > > > > know
> > > > > > > > > when I have something to vote on.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Jim
> > > > > > > > >
> > > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > > > > csringhofer@cloudera.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > > > > My preferred solution would be to deprecate support for
> x64
> > > > > without
> > > > > > > > AVX2
> > > > > > > > > in
> > > > > > > > > > 4.0, but not start removing the related logic yet.
> > > > > > > > > > - We could even add a DCHECK + flag to crash by default
> if
> > no
> > > > > AVX2
> > > > > > is
> > > > > > > > > > detected, and a message that points them to Impala
> mailing
> > > > > > > > lists/related
> > > > > > > > > > Jira.
> > > > > > > > > > - After 4.0 is released people would be free to implement
> > > > things
> > > > > > that
> > > > > > > > > > assume AVX2 support on x64, for example compiling
> > Impala/the
> > > > > whole
> > > > > > > > > > toolchain with -mavx2. I think that currently people are
> > held
> > > > > back
> > > > > > > from
> > > > > > > > > > doing this - why put effort in doing+benchmarking a
> change
> > if
> > > > you
> > > > > > > can't
> > > > > > > > > > push it to master.
> > > > > > > > > > - If a user appears with a valid use case that needs x64
> > > > without
> > > > > > AVX2
> > > > > > > > > then
> > > > > > > > > > we can revise our plans, but we shouldn't ignore low
> > hanging
> > > > > fruits
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > sake of imaginary users :)
> > > > > > > > > >
> > > > > > > > > > -Csaba
> > > > > > > > > >
> > > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > > > > shant@superdupershant.com
> > > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1
> > > > > > > > > > >
> > > > > > > > > > > Regarding IMPALA-9690 if someone has the time to clean
> up
> > > the
> > > > > old
> > > > > > > > > > > conditional cpu logic it would be great, but if the ARM
> > > folks
> > > > > > still
> > > > > > > > > > > need some of that logic it's not critical enough to
> > block,
> > > > > just a
> > > > > > > > very
> > > > > > > > > > > nice to have?
> > > > > > > > > > >
> > > > > > > > > > > -Shant
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > I agree that it would be good to get this done even
> if
> > > it's
> > > > > not
> > > > > > > > > > perfect,
> > > > > > > > > > > > there's so much good stuff.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I agree that we should wrap up Impala 4. A large
> > amount
> > > > of
> > > > > > good
> > > > > > > > > work
> > > > > > > > > > > has
> > > > > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We need to decide which breaking changes are truly
> > > > blockers
> > > > > > for
> > > > > > > > an
> > > > > > > > > > > Impala 4
> > > > > > > > > > > > > release. My feeling is that we can't hold the
> release
> > > for
> > > > > > > > > > compatibility
> > > > > > > > > > > > > breaking changes unless someone signs up to do
> those
> > > > > changes.
> > > > > > > > > > Breaking
> > > > > > > > > > > > > compatibility is useful sometimes, but it can't be
> an
> > > > > > > indefinite
> > > > > > > > > hold
> > > > > > > > > > > on
> > > > > > > > > > > > > releases. I think a release discussion thread is a
> > good
> > > > way
> > > > > > to
> > > > > > > > > > > stimulate
> > > > > > > > > > > > > this discussion.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Apart from the two JIRAs you listed, one other
> change
> > > > > > mentioned
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > original email is switching to use the new
> on-demand
> > > > > metadata
> > > > > > > by
> > > > > > > > > > > default.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Joe
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > > > > > huangquanlong@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Sure. If there are no objections, I'll raise a
> > > > discussion
> > > > > > > > thread
> > > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > 4.0 release. There are still some unresolved
> > breaking
> > > > > > > changes,
> > > > > > > > > e.g.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the default file
> > > format
> > > > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU
> > > requirements
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > > > > > jbapple@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for bringing this up again, Quanlong! I
> > > would
> > > > > love
> > > > > > > to
> > > > > > > > > see
> > > > > > > > > > a
> > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > release soon so that 4.1 releases and what not
> > can
> > > be
> > > > > > > > prepared.
> > > > > > > > > > > Already
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > changlist from the 3.x line is going to be
> > enormous
> > > > and
> > > > > > > could
> > > > > > > > > be
> > > > > > > > > > > hard
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > users to digest.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang
> <
> > > > > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Reviving this thread. I think it's time to
> > create
> > > > the
> > > > > > > 4.0.0
> > > > > > > > > > > branch
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > prepare for the release now. Any breaking
> > change
> > > is
> > > > > > > landing
> > > > > > > > > and
> > > > > > > > > > > we
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > wait?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim
> Armstrong <
> > > > > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Another thing that we're looking into is
> > > changing
> > > > > the
> > > > > > > > > encoded
> > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > profile representation to be more
> efficient -
> > > see
> > > > > > > > > > > > > > > > >
> > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> > > > > > So
> > > > > > > > that
> > > > > > > > > > > might
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > default we will try to change in Impala
> 4.0.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim
> > Armstrong
> > > <
> > > > > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > An addendum to this - we're also
> > considering
> > > > > > whether
> > > > > > > to
> > > > > > > > > > > increase
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > minimum CPU version so that we can
> simplify
> > > > > > > development
> > > > > > > > > and
> > > > > > > > > > > focus
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > > > > > >
> > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Running on commodity hardware is
> important
> > > and
> > > > we
> > > > > > > want
> > > > > > > > > > > people to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > to develop on whatever machine they have
> > > access
> > > > > to.
> > > > > > > At
> > > > > > > > > most
> > > > > > > > > > > we're
> > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > going to require AVX2 support, which has
> > been
> > > > > > > supported
> > > > > > > > > by
> > > > > > > > > > > the
> > > > > > > > > > > > > vast
> > > > > > > > > > > > > > > > > > majority of CPUs for the better part of a
> > > > decade.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe
> > > McDonnell <
> > > > > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >> Impala is starting development on Impala
> > > 4.0.
> > > > > > There
> > > > > > > > are
> > > > > > > > > > many
> > > > > > > > > > > > > > > exciting
> > > > > > > > > > > > > > > > > >> projects underway including full support
> > for
> > > > > Hive
> > > > > > > ACID
> > > > > > > > > > > tables,
> > > > > > > > > > > > > > > > improved
> > > > > > > > > > > > > > > > > >> multithreading support, and fault
> > tolerance
> > > > > > > > > improvements,
> > > > > > > > > > > but
> > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > > >> also serves as a time to alter behavior
> > and
> > > > > > platform
> > > > > > > > > > > support.
> > > > > > > > > > > > > Here
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >> current list of notable breaking changes
> > > that
> > > > > are
> > > > > > > > > planned
> > > > > > > > > > in
> > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Remove support for older operating
> > systems:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Maintaining support for older operating
> > > > systems
> > > > > > > > > requires a
> > > > > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > > > > >> amount of effort, especially with
> changing
> > > > > > security
> > > > > > > > > > > > > requirements.
> > > > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > > >> extra operating system consumes valuable
> > > > effort
> > > > > > and
> > > > > > > > can
> > > > > > > > > > > conflict
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > >> support for newer operating systems, so
> we
> > > > have
> > > > > > > > decided
> > > > > > > > > to
> > > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > >> for some older operating systems.
> > > > Specifically,
> > > > > we
> > > > > > > > plan
> > > > > > > > > to
> > > > > > > > > > > drop
> > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Each of these is approaching or past its
> > end
> > > > of
> > > > > > > life.
> > > > > > > > > > Since
> > > > > > > > > > > > > > Centos6
> > > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > >> the only operating system still needing
> > > Python
> > > > > 2.6
> > > > > > > and
> > > > > > > > > > > Python
> > > > > > > > > > > > > 2.6
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > >> past its end of life, Python 2.6 will no
> > > > longer
> > > > > be
> > > > > > > > > > > supported. In
> > > > > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > > > > >> Centos 7 support will be focused on
> newer
> > > > > versions
> > > > > > > > such
> > > > > > > > > as
> > > > > > > > > > > 7.5
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Over the past year and a half, Impala's
> > > Ranger
> > > > > > > > > > authorization
> > > > > > > > > > > > > > > > > >> functionality has achieved parity and
> > > > surpassed
> > > > > > the
> > > > > > > > > > existing
> > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > >> authorization functionality. Impala's
> > Sentry
> > > > > > support
> > > > > > > > > > > requires
> > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > >> effort to test and maintain, including
> > > > > maintenance
> > > > > > > on
> > > > > > > > > > Sentry
> > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > >> address security updates. Given the
> > > diminished
> > > > > > > > activity
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > > >> community, there is no timeline for
> > > addressing
> > > > > > these
> > > > > > > > > > > security
> > > > > > > > > > > > > > > updates.
> > > > > > > > > > > > > > > > > >> Unless there is a maintainer for Sentry
> > > > support,
> > > > > > > > Impala
> > > > > > > > > > > plans to
> > > > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > >> efforts on Ranger for its authorization
> > > system
> > > > > and
> > > > > > > > drop
> > > > > > > > > > > Sentry.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Impala-lzo provides code to allow Impala
> > to
> > > > read
> > > > > > the
> > > > > > > > LZO
> > > > > > > > > > > > > > compressed
> > > > > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which is
> why
> > > this
> > > > > > > support
> > > > > > > > > is
> > > > > > > > > > > not
> > > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > > >> directly. The Impala-lzo code interacts
> > with
> > > > > > > internal
> > > > > > > > > > Impala
> > > > > > > > > > > > > code
> > > > > > > > > > > > > > > at a
> > > > > > > > > > > > > > > > > >> level that is error prone and intricate.
> > > Given
> > > > > the
> > > > > > > low
> > > > > > > > > > > adoption
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > LZO
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >> the other compression options available,
> > > > Impala
> > > > > > > plans
> > > > > > > > to
> > > > > > > > > > > remove
> > > > > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > > > > >> support along with the low level
> interface
> > > it
> > > > > > used.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> In addition, we also plan to deprecate
> > > several
> > > > > > > > existing
> > > > > > > > > > > features
> > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > >> they can be removed in a future release.
> > > Here
> > > > > is a
> > > > > > > > > summary
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Impala will default to using on
> demand
> > > > > metadata
> > > > > > > > > > > management
> > > > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >> local catalog implementation) as
> > > described
> > > > in
> > > > > > > this
> > > > > > > > > > > > > > documentation:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > > > > >> The old metadata system is now
> > deprecated
> > > > and
> > > > > > may
> > > > > > > > be
> > > > > > > > > > > removed
> > > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Impala is deprecating the Beeswax
> > client
> > > > > > protocol
> > > > > > > > > (i.e.
> > > > > > > > > > > > > clients
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > >> connect via beeswax_port) in favor of
> > the
> > > > > > > > HiveServer2
> > > > > > > > > > > client
> > > > > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Impala is deprecating the old
> > > > > decimal_v2=false
> > > > > > > > > behavior
> > > > > > > > > > > from
> > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query option has
> > > > > defaulted
> > > > > > to
> > > > > > > > > true
> > > > > > > > > > > since
> > > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Impala is always open to new developers,
> > and
> > > > we
> > > > > > > > welcome
> > > > > > > > > > > feedback
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > >> plans. Further discussion of other
> changes
> > > and
> > > > > > > > > > deprecations
> > > > > > > > > > > is
> > > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Jim Apple <jb...@apache.org>.
We're only waiting on one patch for the docs, but since I cut the 4.0
branch on May 1, a number of things have landed. Maybe it would be best to
merge HEAD into 4.0.0 once the Sentry doc change lands, thereby including
everything we've talked about cherrypicking -- including 9690.
Thoughts?
On Wed, May 26, 2021 at 12:40 AM Quanlong Huang <hu...@gmail.com>
wrote:
> Hi all,
>
> I see IMPALA-9690 is done but still missing in branch-4.0.0 and I think we
> should pick it
>
> - c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2 for x86_64
>
> I'd be good to also carry on these fixes for query retry:
>
> - 42684b4 IMPALA-10413: fix impalad crashes when canceling the retrying
> query
> - d111443 IMPALA-10704: Fix retried query id not being unregistered when
> retry fails
>
> If you are ok, I can push them to branch-4.0.0.
>
> Thanks,
> Quanlong
>
> On Sun, May 16, 2021 at 6:02 AM Jim Apple <jb...@apache.org> wrote:
>
> > Sounds good!
> >
> > On Sat, May 15, 2021 at 1:29 PM Joe McDonnell <joemcdonnell@cloudera.com
> >
> > wrote:
> >
> > > Hi Jim,
> > >
> > > I talked with the developer for the Sentry docs bug this past week, and
> > we
> > > went through the list of changes needed for that. The developer is
> hoping
> > > to work on it next week.
> > >
> > > If there is any further delay, I'm open to options of working around
> it.
> > >
> > > Thanks,
> > > Joe
> > >
> > > On Sat, May 15, 2021 at 12:43 PM Jim Apple <jb...@apache.org> wrote:
> > >
> > > > This Sentry one I'm concerned about. The ticket has been open for a
> > year.
> > > > We could do a 4.0 release without the docs, or we could maybe put a
> > note
> > > on
> > > > the docs that Sentry support is no longer available.
> > > >
> > > > Any other ideas on how to tackle this while the ticket waits for a
> > patch?
> > > >
> > > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <
> > joemcdonnell@cloudera.com>
> > > > wrote:
> > > >
> > > > > I agree that it is a good idea for us to get going on the 4.0
> > release.
> > > > >
> > > > > I'm working on a change to fix IMPALA-9690. I will post an updated
> > > review
> > > > > once a test run completes. If everything is looking good, I would
> > like
> > > to
> > > > > get this in the 4.0 release.
> > > > >
> > > > > There is also one documentation fix that I think belongs in the
> > Impala
> > > > 4.0
> > > > > release:
> > > > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry
> > > > > references
> > > > >
> > > > > Thanks,
> > > > > Joe
> > > > >
> > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com>
> wrote:
> > > > >
> > > > > > Sure thing! Let us know if you find owners for anything that
> > someone
> > > > > wants
> > > > > > to champion as a 4.0 blocker.
> > > > > >
> > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> > > huangquanlong@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > > > >
> > > > > > > But due to some other urgent works and the recent holiday, my
> > > > progress
> > > > > is
> > > > > > > slow. Anyway, I think it's time to branch out. We've been
> waiting
> > > too
> > > > > > long.
> > > > > > > Thanks for creating the branch.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Quanlong
> > > > > > >
> > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com>
> > > wrote:
> > > > > > >
> > > > > > > > I've created a 4.0 branch (with all changes through the end
> of
> > > > April)
> > > > > > and
> > > > > > > > started testing a release candidate. Everyone who wants to
> > vote,
> > > > > > > >
> > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > > > is a
> > > > > > > > guide to both releasing and voting (at the bottom). I'll let
> > you
> > > > all
> > > > > > know
> > > > > > > > when I have something to vote on.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Jim
> > > > > > > >
> > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > > > csringhofer@cloudera.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > > > My preferred solution would be to deprecate support for x64
> > > > without
> > > > > > > AVX2
> > > > > > > > in
> > > > > > > > > 4.0, but not start removing the related logic yet.
> > > > > > > > > - We could even add a DCHECK + flag to crash by default if
> no
> > > > AVX2
> > > > > is
> > > > > > > > > detected, and a message that points them to Impala mailing
> > > > > > > lists/related
> > > > > > > > > Jira.
> > > > > > > > > - After 4.0 is released people would be free to implement
> > > things
> > > > > that
> > > > > > > > > assume AVX2 support on x64, for example compiling
> Impala/the
> > > > whole
> > > > > > > > > toolchain with -mavx2. I think that currently people are
> held
> > > > back
> > > > > > from
> > > > > > > > > doing this - why put effort in doing+benchmarking a change
> if
> > > you
> > > > > > can't
> > > > > > > > > push it to master.
> > > > > > > > > - If a user appears with a valid use case that needs x64
> > > without
> > > > > AVX2
> > > > > > > > then
> > > > > > > > > we can revise our plans, but we shouldn't ignore low
> hanging
> > > > fruits
> > > > > > for
> > > > > > > > the
> > > > > > > > > sake of imaginary users :)
> > > > > > > > >
> > > > > > > > > -Csaba
> > > > > > > > >
> > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > > > shant@superdupershant.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > >
> > > > > > > > > > Regarding IMPALA-9690 if someone has the time to clean up
> > the
> > > > old
> > > > > > > > > > conditional cpu logic it would be great, but if the ARM
> > folks
> > > > > still
> > > > > > > > > > need some of that logic it's not critical enough to
> block,
> > > > just a
> > > > > > > very
> > > > > > > > > > nice to have?
> > > > > > > > > >
> > > > > > > > > > -Shant
> > > > > > > > > >
> > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > I agree that it would be good to get this done even if
> > it's
> > > > not
> > > > > > > > > perfect,
> > > > > > > > > > > there's so much good stuff.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I agree that we should wrap up Impala 4. A large
> amount
> > > of
> > > > > good
> > > > > > > > work
> > > > > > > > > > has
> > > > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > > > >
> > > > > > > > > > > > We need to decide which breaking changes are truly
> > > blockers
> > > > > for
> > > > > > > an
> > > > > > > > > > Impala 4
> > > > > > > > > > > > release. My feeling is that we can't hold the release
> > for
> > > > > > > > > compatibility
> > > > > > > > > > > > breaking changes unless someone signs up to do those
> > > > changes.
> > > > > > > > > Breaking
> > > > > > > > > > > > compatibility is useful sometimes, but it can't be an
> > > > > > indefinite
> > > > > > > > hold
> > > > > > > > > > on
> > > > > > > > > > > > releases. I think a release discussion thread is a
> good
> > > way
> > > > > to
> > > > > > > > > > stimulate
> > > > > > > > > > > > this discussion.
> > > > > > > > > > > >
> > > > > > > > > > > > Apart from the two JIRAs you listed, one other change
> > > > > mentioned
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > original email is switching to use the new on-demand
> > > > metadata
> > > > > > by
> > > > > > > > > > default.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Joe
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > > > > huangquanlong@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Sure. If there are no objections, I'll raise a
> > > discussion
> > > > > > > thread
> > > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > 4.0 release. There are still some unresolved
> breaking
> > > > > > changes,
> > > > > > > > e.g.
> > > > > > > > > > > > >
> > > > > > > > > > > > > - IMPALA-2210: Make Parquet the default file
> > format
> > > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU
> > requirements
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > > > > jbapple@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for bringing this up again, Quanlong! I
> > would
> > > > love
> > > > > > to
> > > > > > > > see
> > > > > > > > > a
> > > > > > > > > > 4.0
> > > > > > > > > > > > > > release soon so that 4.1 releases and what not
> can
> > be
> > > > > > > prepared.
> > > > > > > > > > Already
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > changlist from the 3.x line is going to be
> enormous
> > > and
> > > > > > could
> > > > > > > > be
> > > > > > > > > > hard
> > > > > > > > > > > > for
> > > > > > > > > > > > > > users to digest.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Reviving this thread. I think it's time to
> create
> > > the
> > > > > > 4.0.0
> > > > > > > > > > branch
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > prepare for the release now. Any breaking
> change
> > is
> > > > > > landing
> > > > > > > > and
> > > > > > > > > > we
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > > wait?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Another thing that we're looking into is
> > changing
> > > > the
> > > > > > > > encoded
> > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > profile representation to be more efficient -
> > see
> > > > > > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/IMPALA-9378.
> > > > > So
> > > > > > > that
> > > > > > > > > > might
> > > > > > > > > > > > be
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim
> Armstrong
> > <
> > > > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > An addendum to this - we're also
> considering
> > > > > whether
> > > > > > to
> > > > > > > > > > increase
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > minimum CPU version so that we can simplify
> > > > > > development
> > > > > > > > and
> > > > > > > > > > focus
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > > > > >
> > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Running on commodity hardware is important
> > and
> > > we
> > > > > > want
> > > > > > > > > > people to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > to develop on whatever machine they have
> > access
> > > > to.
> > > > > > At
> > > > > > > > most
> > > > > > > > > > we're
> > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > going to require AVX2 support, which has
> been
> > > > > > supported
> > > > > > > > by
> > > > > > > > > > the
> > > > > > > > > > > > vast
> > > > > > > > > > > > > > > > > majority of CPUs for the better part of a
> > > decade.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe
> > McDonnell <
> > > > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> Impala is starting development on Impala
> > 4.0.
> > > > > There
> > > > > > > are
> > > > > > > > > many
> > > > > > > > > > > > > > exciting
> > > > > > > > > > > > > > > > >> projects underway including full support
> for
> > > > Hive
> > > > > > ACID
> > > > > > > > > > tables,
> > > > > > > > > > > > > > > improved
> > > > > > > > > > > > > > > > >> multithreading support, and fault
> tolerance
> > > > > > > > improvements,
> > > > > > > > > > but
> > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > > >> also serves as a time to alter behavior
> and
> > > > > platform
> > > > > > > > > > support.
> > > > > > > > > > > > Here
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >> current list of notable breaking changes
> > that
> > > > are
> > > > > > > > planned
> > > > > > > > > in
> > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Remove support for older operating
> systems:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Maintaining support for older operating
> > > systems
> > > > > > > > requires a
> > > > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > > > >> amount of effort, especially with changing
> > > > > security
> > > > > > > > > > > > requirements.
> > > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > >> extra operating system consumes valuable
> > > effort
> > > > > and
> > > > > > > can
> > > > > > > > > > conflict
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > >> support for newer operating systems, so we
> > > have
> > > > > > > decided
> > > > > > > > to
> > > > > > > > > > > > remove
> > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > >> for some older operating systems.
> > > Specifically,
> > > > we
> > > > > > > plan
> > > > > > > > to
> > > > > > > > > > drop
> > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Each of these is approaching or past its
> end
> > > of
> > > > > > life.
> > > > > > > > > Since
> > > > > > > > > > > > > Centos6
> > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > >> the only operating system still needing
> > Python
> > > > 2.6
> > > > > > and
> > > > > > > > > > Python
> > > > > > > > > > > > 2.6
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > >> past its end of life, Python 2.6 will no
> > > longer
> > > > be
> > > > > > > > > > supported. In
> > > > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > > > >> Centos 7 support will be focused on newer
> > > > versions
> > > > > > > such
> > > > > > > > as
> > > > > > > > > > 7.5
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Over the past year and a half, Impala's
> > Ranger
> > > > > > > > > authorization
> > > > > > > > > > > > > > > > >> functionality has achieved parity and
> > > surpassed
> > > > > the
> > > > > > > > > existing
> > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > >> authorization functionality. Impala's
> Sentry
> > > > > support
> > > > > > > > > > requires
> > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > >> effort to test and maintain, including
> > > > maintenance
> > > > > > on
> > > > > > > > > Sentry
> > > > > > > > > > > > > itself
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > >> address security updates. Given the
> > diminished
> > > > > > > activity
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > > >> community, there is no timeline for
> > addressing
> > > > > these
> > > > > > > > > > security
> > > > > > > > > > > > > > updates.
> > > > > > > > > > > > > > > > >> Unless there is a maintainer for Sentry
> > > support,
> > > > > > > Impala
> > > > > > > > > > plans to
> > > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > >> efforts on Ranger for its authorization
> > system
> > > > and
> > > > > > > drop
> > > > > > > > > > Sentry.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Impala-lzo provides code to allow Impala
> to
> > > read
> > > > > the
> > > > > > > LZO
> > > > > > > > > > > > > compressed
> > > > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why
> > this
> > > > > > support
> > > > > > > > is
> > > > > > > > > > not
> > > > > > > > > > > > > > included
> > > > > > > > > > > > > > > > >> directly. The Impala-lzo code interacts
> with
> > > > > > internal
> > > > > > > > > Impala
> > > > > > > > > > > > code
> > > > > > > > > > > > > > at a
> > > > > > > > > > > > > > > > >> level that is error prone and intricate.
> > Given
> > > > the
> > > > > > low
> > > > > > > > > > adoption
> > > > > > > > > > > > of
> > > > > > > > > > > > > > LZO
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >> the other compression options available,
> > > Impala
> > > > > > plans
> > > > > > > to
> > > > > > > > > > remove
> > > > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > > > >> support along with the low level interface
> > it
> > > > > used.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> In addition, we also plan to deprecate
> > several
> > > > > > > existing
> > > > > > > > > > features
> > > > > > > > > > > > > so
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > >> they can be removed in a future release.
> > Here
> > > > is a
> > > > > > > > summary
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Impala will default to using on demand
> > > > metadata
> > > > > > > > > > management
> > > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >> local catalog implementation) as
> > described
> > > in
> > > > > > this
> > > > > > > > > > > > > documentation:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > > > >> The old metadata system is now
> deprecated
> > > and
> > > > > may
> > > > > > > be
> > > > > > > > > > removed
> > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Impala is deprecating the Beeswax
> client
> > > > > protocol
> > > > > > > > (i.e.
> > > > > > > > > > > > clients
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > >> connect via beeswax_port) in favor of
> the
> > > > > > > HiveServer2
> > > > > > > > > > client
> > > > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Impala is deprecating the old
> > > > decimal_v2=false
> > > > > > > > behavior
> > > > > > > > > > from
> > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query option has
> > > > defaulted
> > > > > to
> > > > > > > > true
> > > > > > > > > > since
> > > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Impala is always open to new developers,
> and
> > > we
> > > > > > > welcome
> > > > > > > > > > feedback
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > >> plans. Further discussion of other changes
> > and
> > > > > > > > > deprecations
> > > > > > > > > > is
> > > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Quanlong Huang <hu...@gmail.com>.
Hi all,
I see IMPALA-9690 is done but still missing in branch-4.0.0 and I think we
should pick it
- c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2 for x86_64
I'd be good to also carry on these fixes for query retry:
- 42684b4 IMPALA-10413: fix impalad crashes when canceling the retrying
query
- d111443 IMPALA-10704: Fix retried query id not being unregistered when
retry fails
If you are ok, I can push them to branch-4.0.0.
Thanks,
Quanlong
On Sun, May 16, 2021 at 6:02 AM Jim Apple <jb...@apache.org> wrote:
> Sounds good!
>
> On Sat, May 15, 2021 at 1:29 PM Joe McDonnell <jo...@cloudera.com>
> wrote:
>
> > Hi Jim,
> >
> > I talked with the developer for the Sentry docs bug this past week, and
> we
> > went through the list of changes needed for that. The developer is hoping
> > to work on it next week.
> >
> > If there is any further delay, I'm open to options of working around it.
> >
> > Thanks,
> > Joe
> >
> > On Sat, May 15, 2021 at 12:43 PM Jim Apple <jb...@apache.org> wrote:
> >
> > > This Sentry one I'm concerned about. The ticket has been open for a
> year.
> > > We could do a 4.0 release without the docs, or we could maybe put a
> note
> > on
> > > the docs that Sentry support is no longer available.
> > >
> > > Any other ideas on how to tackle this while the ticket waits for a
> patch?
> > >
> > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <
> joemcdonnell@cloudera.com>
> > > wrote:
> > >
> > > > I agree that it is a good idea for us to get going on the 4.0
> release.
> > > >
> > > > I'm working on a change to fix IMPALA-9690. I will post an updated
> > review
> > > > once a test run completes. If everything is looking good, I would
> like
> > to
> > > > get this in the 4.0 release.
> > > >
> > > > There is also one documentation fix that I think belongs in the
> Impala
> > > 4.0
> > > > release:
> > > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry
> > > > references
> > > >
> > > > Thanks,
> > > > Joe
> > > >
> > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com> wrote:
> > > >
> > > > > Sure thing! Let us know if you find owners for anything that
> someone
> > > > wants
> > > > > to champion as a 4.0 blocker.
> > > > >
> > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> > huangquanlong@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > > >
> > > > > > But due to some other urgent works and the recent holiday, my
> > > progress
> > > > is
> > > > > > slow. Anyway, I think it's time to branch out. We've been waiting
> > too
> > > > > long.
> > > > > > Thanks for creating the branch.
> > > > > >
> > > > > > Regards,
> > > > > > Quanlong
> > > > > >
> > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com>
> > wrote:
> > > > > >
> > > > > > > I've created a 4.0 branch (with all changes through the end of
> > > April)
> > > > > and
> > > > > > > started testing a release candidate. Everyone who wants to
> vote,
> > > > > > >
> > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > > is a
> > > > > > > guide to both releasing and voting (at the bottom). I'll let
> you
> > > all
> > > > > know
> > > > > > > when I have something to vote on.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Jim
> > > > > > >
> > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > > csringhofer@cloudera.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > > My preferred solution would be to deprecate support for x64
> > > without
> > > > > > AVX2
> > > > > > > in
> > > > > > > > 4.0, but not start removing the related logic yet.
> > > > > > > > - We could even add a DCHECK + flag to crash by default if no
> > > AVX2
> > > > is
> > > > > > > > detected, and a message that points them to Impala mailing
> > > > > > lists/related
> > > > > > > > Jira.
> > > > > > > > - After 4.0 is released people would be free to implement
> > things
> > > > that
> > > > > > > > assume AVX2 support on x64, for example compiling Impala/the
> > > whole
> > > > > > > > toolchain with -mavx2. I think that currently people are held
> > > back
> > > > > from
> > > > > > > > doing this - why put effort in doing+benchmarking a change if
> > you
> > > > > can't
> > > > > > > > push it to master.
> > > > > > > > - If a user appears with a valid use case that needs x64
> > without
> > > > AVX2
> > > > > > > then
> > > > > > > > we can revise our plans, but we shouldn't ignore low hanging
> > > fruits
> > > > > for
> > > > > > > the
> > > > > > > > sake of imaginary users :)
> > > > > > > >
> > > > > > > > -Csaba
> > > > > > > >
> > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > > shant@superdupershant.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > Regarding IMPALA-9690 if someone has the time to clean up
> the
> > > old
> > > > > > > > > conditional cpu logic it would be great, but if the ARM
> folks
> > > > still
> > > > > > > > > need some of that logic it's not critical enough to block,
> > > just a
> > > > > > very
> > > > > > > > > nice to have?
> > > > > > > > >
> > > > > > > > > -Shant
> > > > > > > > >
> > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > I agree that it would be good to get this done even if
> it's
> > > not
> > > > > > > > perfect,
> > > > > > > > > > there's so much good stuff.
> > > > > > > > > >
> > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I agree that we should wrap up Impala 4. A large amount
> > of
> > > > good
> > > > > > > work
> > > > > > > > > has
> > > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > > >
> > > > > > > > > > > We need to decide which breaking changes are truly
> > blockers
> > > > for
> > > > > > an
> > > > > > > > > Impala 4
> > > > > > > > > > > release. My feeling is that we can't hold the release
> for
> > > > > > > > compatibility
> > > > > > > > > > > breaking changes unless someone signs up to do those
> > > changes.
> > > > > > > > Breaking
> > > > > > > > > > > compatibility is useful sometimes, but it can't be an
> > > > > indefinite
> > > > > > > hold
> > > > > > > > > on
> > > > > > > > > > > releases. I think a release discussion thread is a good
> > way
> > > > to
> > > > > > > > > stimulate
> > > > > > > > > > > this discussion.
> > > > > > > > > > >
> > > > > > > > > > > Apart from the two JIRAs you listed, one other change
> > > > mentioned
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > original email is switching to use the new on-demand
> > > metadata
> > > > > by
> > > > > > > > > default.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Joe
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > > > huangquanlong@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Sure. If there are no objections, I'll raise a
> > discussion
> > > > > > thread
> > > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > 4.0 release. There are still some unresolved breaking
> > > > > changes,
> > > > > > > e.g.
> > > > > > > > > > > >
> > > > > > > > > > > > - IMPALA-2210: Make Parquet the default file
> format
> > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU
> requirements
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > > Quanlong
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > > > jbapple@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for bringing this up again, Quanlong! I
> would
> > > love
> > > > > to
> > > > > > > see
> > > > > > > > a
> > > > > > > > > 4.0
> > > > > > > > > > > > > release soon so that 4.1 releases and what not can
> be
> > > > > > prepared.
> > > > > > > > > Already
> > > > > > > > > > > > the
> > > > > > > > > > > > > changlist from the 3.x line is going to be enormous
> > and
> > > > > could
> > > > > > > be
> > > > > > > > > hard
> > > > > > > > > > > for
> > > > > > > > > > > > > users to digest.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Reviving this thread. I think it's time to create
> > the
> > > > > 4.0.0
> > > > > > > > > branch
> > > > > > > > > > > and
> > > > > > > > > > > > > > prepare for the release now. Any breaking change
> is
> > > > > landing
> > > > > > > and
> > > > > > > > > we
> > > > > > > > > > > > should
> > > > > > > > > > > > > > wait?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Another thing that we're looking into is
> changing
> > > the
> > > > > > > encoded
> > > > > > > > > > > runtime
> > > > > > > > > > > > > > > profile representation to be more efficient -
> see
> > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/IMPALA-9378.
> > > > So
> > > > > > that
> > > > > > > > > might
> > > > > > > > > > > be
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong
> <
> > > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > An addendum to this - we're also considering
> > > > whether
> > > > > to
> > > > > > > > > increase
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > minimum CPU version so that we can simplify
> > > > > development
> > > > > > > and
> > > > > > > > > focus
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Running on commodity hardware is important
> and
> > we
> > > > > want
> > > > > > > > > people to
> > > > > > > > > > > be
> > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > to develop on whatever machine they have
> access
> > > to.
> > > > > At
> > > > > > > most
> > > > > > > > > we're
> > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > going to require AVX2 support, which has been
> > > > > supported
> > > > > > > by
> > > > > > > > > the
> > > > > > > > > > > vast
> > > > > > > > > > > > > > > > majority of CPUs for the better part of a
> > decade.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe
> McDonnell <
> > > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Impala is starting development on Impala
> 4.0.
> > > > There
> > > > > > are
> > > > > > > > many
> > > > > > > > > > > > > exciting
> > > > > > > > > > > > > > > >> projects underway including full support for
> > > Hive
> > > > > ACID
> > > > > > > > > tables,
> > > > > > > > > > > > > > improved
> > > > > > > > > > > > > > > >> multithreading support, and fault tolerance
> > > > > > > improvements,
> > > > > > > > > but
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > >> also serves as a time to alter behavior and
> > > > platform
> > > > > > > > > support.
> > > > > > > > > > > Here
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> current list of notable breaking changes
> that
> > > are
> > > > > > > planned
> > > > > > > > in
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Maintaining support for older operating
> > systems
> > > > > > > requires a
> > > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > > >> amount of effort, especially with changing
> > > > security
> > > > > > > > > > > requirements.
> > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > >> extra operating system consumes valuable
> > effort
> > > > and
> > > > > > can
> > > > > > > > > conflict
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > >> support for newer operating systems, so we
> > have
> > > > > > decided
> > > > > > > to
> > > > > > > > > > > remove
> > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > >> for some older operating systems.
> > Specifically,
> > > we
> > > > > > plan
> > > > > > > to
> > > > > > > > > drop
> > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Each of these is approaching or past its end
> > of
> > > > > life.
> > > > > > > > Since
> > > > > > > > > > > > Centos6
> > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > >> the only operating system still needing
> Python
> > > 2.6
> > > > > and
> > > > > > > > > Python
> > > > > > > > > > > 2.6
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > >> past its end of life, Python 2.6 will no
> > longer
> > > be
> > > > > > > > > supported. In
> > > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > > >> Centos 7 support will be focused on newer
> > > versions
> > > > > > such
> > > > > > > as
> > > > > > > > > 7.5
> > > > > > > > > > > and
> > > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Over the past year and a half, Impala's
> Ranger
> > > > > > > > authorization
> > > > > > > > > > > > > > > >> functionality has achieved parity and
> > surpassed
> > > > the
> > > > > > > > existing
> > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > >> authorization functionality. Impala's Sentry
> > > > support
> > > > > > > > > requires
> > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > >> effort to test and maintain, including
> > > maintenance
> > > > > on
> > > > > > > > Sentry
> > > > > > > > > > > > itself
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > >> address security updates. Given the
> diminished
> > > > > > activity
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > >> community, there is no timeline for
> addressing
> > > > these
> > > > > > > > > security
> > > > > > > > > > > > > updates.
> > > > > > > > > > > > > > > >> Unless there is a maintainer for Sentry
> > support,
> > > > > > Impala
> > > > > > > > > plans to
> > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > >> efforts on Ranger for its authorization
> system
> > > and
> > > > > > drop
> > > > > > > > > Sentry.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to
> > read
> > > > the
> > > > > > LZO
> > > > > > > > > > > > compressed
> > > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why
> this
> > > > > support
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > > > > included
> > > > > > > > > > > > > > > >> directly. The Impala-lzo code interacts with
> > > > > internal
> > > > > > > > Impala
> > > > > > > > > > > code
> > > > > > > > > > > > > at a
> > > > > > > > > > > > > > > >> level that is error prone and intricate.
> Given
> > > the
> > > > > low
> > > > > > > > > adoption
> > > > > > > > > > > of
> > > > > > > > > > > > > LZO
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> the other compression options available,
> > Impala
> > > > > plans
> > > > > > to
> > > > > > > > > remove
> > > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > > >> support along with the low level interface
> it
> > > > used.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> In addition, we also plan to deprecate
> several
> > > > > > existing
> > > > > > > > > features
> > > > > > > > > > > > so
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > >> they can be removed in a future release.
> Here
> > > is a
> > > > > > > summary
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > most
> > > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Impala will default to using on demand
> > > metadata
> > > > > > > > > management
> > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> local catalog implementation) as
> described
> > in
> > > > > this
> > > > > > > > > > > > documentation:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > > >> The old metadata system is now deprecated
> > and
> > > > may
> > > > > > be
> > > > > > > > > removed
> > > > > > > > > > > > in a
> > > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Impala is deprecating the Beeswax client
> > > > protocol
> > > > > > > (i.e.
> > > > > > > > > > > clients
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > >> connect via beeswax_port) in favor of the
> > > > > > HiveServer2
> > > > > > > > > client
> > > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Impala is deprecating the old
> > > decimal_v2=false
> > > > > > > behavior
> > > > > > > > > from
> > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query option has
> > > defaulted
> > > > to
> > > > > > > true
> > > > > > > > > since
> > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Impala is always open to new developers, and
> > we
> > > > > > welcome
> > > > > > > > > feedback
> > > > > > > > > > > > on
> > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > >> plans. Further discussion of other changes
> and
> > > > > > > > deprecations
> > > > > > > > > is
> > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Jim Apple <jb...@apache.org>.
Sounds good!
On Sat, May 15, 2021 at 1:29 PM Joe McDonnell <jo...@cloudera.com>
wrote:
> Hi Jim,
>
> I talked with the developer for the Sentry docs bug this past week, and we
> went through the list of changes needed for that. The developer is hoping
> to work on it next week.
>
> If there is any further delay, I'm open to options of working around it.
>
> Thanks,
> Joe
>
> On Sat, May 15, 2021 at 12:43 PM Jim Apple <jb...@apache.org> wrote:
>
> > This Sentry one I'm concerned about. The ticket has been open for a year.
> > We could do a 4.0 release without the docs, or we could maybe put a note
> on
> > the docs that Sentry support is no longer available.
> >
> > Any other ideas on how to tackle this while the ticket waits for a patch?
> >
> > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <jo...@cloudera.com>
> > wrote:
> >
> > > I agree that it is a good idea for us to get going on the 4.0 release.
> > >
> > > I'm working on a change to fix IMPALA-9690. I will post an updated
> review
> > > once a test run completes. If everything is looking good, I would like
> to
> > > get this in the 4.0 release.
> > >
> > > There is also one documentation fix that I think belongs in the Impala
> > 4.0
> > > release:
> > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry
> > > references
> > >
> > > Thanks,
> > > Joe
> > >
> > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com> wrote:
> > >
> > > > Sure thing! Let us know if you find owners for anything that someone
> > > wants
> > > > to champion as a 4.0 blocker.
> > > >
> > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> huangquanlong@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > >
> > > > > But due to some other urgent works and the recent holiday, my
> > progress
> > > is
> > > > > slow. Anyway, I think it's time to branch out. We've been waiting
> too
> > > > long.
> > > > > Thanks for creating the branch.
> > > > >
> > > > > Regards,
> > > > > Quanlong
> > > > >
> > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com>
> wrote:
> > > > >
> > > > > > I've created a 4.0 branch (with all changes through the end of
> > April)
> > > > and
> > > > > > started testing a release candidate. Everyone who wants to vote,
> > > > > >
> https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > is a
> > > > > > guide to both releasing and voting (at the bottom). I'll let you
> > all
> > > > know
> > > > > > when I have something to vote on.
> > > > > >
> > > > > > Thanks!
> > > > > > Jim
> > > > > >
> > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > csringhofer@cloudera.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > My preferred solution would be to deprecate support for x64
> > without
> > > > > AVX2
> > > > > > in
> > > > > > > 4.0, but not start removing the related logic yet.
> > > > > > > - We could even add a DCHECK + flag to crash by default if no
> > AVX2
> > > is
> > > > > > > detected, and a message that points them to Impala mailing
> > > > > lists/related
> > > > > > > Jira.
> > > > > > > - After 4.0 is released people would be free to implement
> things
> > > that
> > > > > > > assume AVX2 support on x64, for example compiling Impala/the
> > whole
> > > > > > > toolchain with -mavx2. I think that currently people are held
> > back
> > > > from
> > > > > > > doing this - why put effort in doing+benchmarking a change if
> you
> > > > can't
> > > > > > > push it to master.
> > > > > > > - If a user appears with a valid use case that needs x64
> without
> > > AVX2
> > > > > > then
> > > > > > > we can revise our plans, but we shouldn't ignore low hanging
> > fruits
> > > > for
> > > > > > the
> > > > > > > sake of imaginary users :)
> > > > > > >
> > > > > > > -Csaba
> > > > > > >
> > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > shant@superdupershant.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Regarding IMPALA-9690 if someone has the time to clean up the
> > old
> > > > > > > > conditional cpu logic it would be great, but if the ARM folks
> > > still
> > > > > > > > need some of that logic it's not critical enough to block,
> > just a
> > > > > very
> > > > > > > > nice to have?
> > > > > > > >
> > > > > > > > -Shant
> > > > > > > >
> > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I agree that it would be good to get this done even if it's
> > not
> > > > > > > perfect,
> > > > > > > > > there's so much good stuff.
> > > > > > > > >
> > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I agree that we should wrap up Impala 4. A large amount
> of
> > > good
> > > > > > work
> > > > > > > > has
> > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > >
> > > > > > > > > > We need to decide which breaking changes are truly
> blockers
> > > for
> > > > > an
> > > > > > > > Impala 4
> > > > > > > > > > release. My feeling is that we can't hold the release for
> > > > > > > compatibility
> > > > > > > > > > breaking changes unless someone signs up to do those
> > changes.
> > > > > > > Breaking
> > > > > > > > > > compatibility is useful sometimes, but it can't be an
> > > > indefinite
> > > > > > hold
> > > > > > > > on
> > > > > > > > > > releases. I think a release discussion thread is a good
> way
> > > to
> > > > > > > > stimulate
> > > > > > > > > > this discussion.
> > > > > > > > > >
> > > > > > > > > > Apart from the two JIRAs you listed, one other change
> > > mentioned
> > > > > in
> > > > > > > the
> > > > > > > > > > original email is switching to use the new on-demand
> > metadata
> > > > by
> > > > > > > > default.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Joe
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > > huangquanlong@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Sure. If there are no objections, I'll raise a
> discussion
> > > > > thread
> > > > > > > for
> > > > > > > > the
> > > > > > > > > > > 4.0 release. There are still some unresolved breaking
> > > > changes,
> > > > > > e.g.
> > > > > > > > > > >
> > > > > > > > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Quanlong
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > > jbapple@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for bringing this up again, Quanlong! I would
> > love
> > > > to
> > > > > > see
> > > > > > > a
> > > > > > > > 4.0
> > > > > > > > > > > > release soon so that 4.1 releases and what not can be
> > > > > prepared.
> > > > > > > > Already
> > > > > > > > > > > the
> > > > > > > > > > > > changlist from the 3.x line is going to be enormous
> and
> > > > could
> > > > > > be
> > > > > > > > hard
> > > > > > > > > > for
> > > > > > > > > > > > users to digest.
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Reviving this thread. I think it's time to create
> the
> > > > 4.0.0
> > > > > > > > branch
> > > > > > > > > > and
> > > > > > > > > > > > > prepare for the release now. Any breaking change is
> > > > landing
> > > > > > and
> > > > > > > > we
> > > > > > > > > > > should
> > > > > > > > > > > > > wait?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Another thing that we're looking into is changing
> > the
> > > > > > encoded
> > > > > > > > > > runtime
> > > > > > > > > > > > > > profile representation to be more efficient - see
> > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/IMPALA-9378.
> > > So
> > > > > that
> > > > > > > > might
> > > > > > > > > > be
> > > > > > > > > > > a
> > > > > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > An addendum to this - we're also considering
> > > whether
> > > > to
> > > > > > > > increase
> > > > > > > > > > > the
> > > > > > > > > > > > > > > minimum CPU version so that we can simplify
> > > > development
> > > > > > and
> > > > > > > > focus
> > > > > > > > > > > > more
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Running on commodity hardware is important and
> we
> > > > want
> > > > > > > > people to
> > > > > > > > > > be
> > > > > > > > > > > > > able
> > > > > > > > > > > > > > > to develop on whatever machine they have access
> > to.
> > > > At
> > > > > > most
> > > > > > > > we're
> > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > going to require AVX2 support, which has been
> > > > supported
> > > > > > by
> > > > > > > > the
> > > > > > > > > > vast
> > > > > > > > > > > > > > > majority of CPUs for the better part of a
> decade.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Impala is starting development on Impala 4.0.
> > > There
> > > > > are
> > > > > > > many
> > > > > > > > > > > > exciting
> > > > > > > > > > > > > > >> projects underway including full support for
> > Hive
> > > > ACID
> > > > > > > > tables,
> > > > > > > > > > > > > improved
> > > > > > > > > > > > > > >> multithreading support, and fault tolerance
> > > > > > improvements,
> > > > > > > > but
> > > > > > > > > > > Impala
> > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > >> also serves as a time to alter behavior and
> > > platform
> > > > > > > > support.
> > > > > > > > > > Here
> > > > > > > > > > > > is
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> current list of notable breaking changes that
> > are
> > > > > > planned
> > > > > > > in
> > > > > > > > > > > Impala
> > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Maintaining support for older operating
> systems
> > > > > > requires a
> > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > >> amount of effort, especially with changing
> > > security
> > > > > > > > > > requirements.
> > > > > > > > > > > > Each
> > > > > > > > > > > > > > >> extra operating system consumes valuable
> effort
> > > and
> > > > > can
> > > > > > > > conflict
> > > > > > > > > > > > with
> > > > > > > > > > > > > > >> support for newer operating systems, so we
> have
> > > > > decided
> > > > > > to
> > > > > > > > > > remove
> > > > > > > > > > > > > > support
> > > > > > > > > > > > > > >> for some older operating systems.
> Specifically,
> > we
> > > > > plan
> > > > > > to
> > > > > > > > drop
> > > > > > > > > > > > > support
> > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Each of these is approaching or past its end
> of
> > > > life.
> > > > > > > Since
> > > > > > > > > > > Centos6
> > > > > > > > > > > > > was
> > > > > > > > > > > > > > >> the only operating system still needing Python
> > 2.6
> > > > and
> > > > > > > > Python
> > > > > > > > > > 2.6
> > > > > > > > > > > is
> > > > > > > > > > > > > > long
> > > > > > > > > > > > > > >> past its end of life, Python 2.6 will no
> longer
> > be
> > > > > > > > supported. In
> > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > >> Centos 7 support will be focused on newer
> > versions
> > > > > such
> > > > > > as
> > > > > > > > 7.5
> > > > > > > > > > and
> > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Over the past year and a half, Impala's Ranger
> > > > > > > authorization
> > > > > > > > > > > > > > >> functionality has achieved parity and
> surpassed
> > > the
> > > > > > > existing
> > > > > > > > > > > Sentry
> > > > > > > > > > > > > > >> authorization functionality. Impala's Sentry
> > > support
> > > > > > > > requires
> > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > >> effort to test and maintain, including
> > maintenance
> > > > on
> > > > > > > Sentry
> > > > > > > > > > > itself
> > > > > > > > > > > > to
> > > > > > > > > > > > > > >> address security updates. Given the diminished
> > > > > activity
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > >> community, there is no timeline for addressing
> > > these
> > > > > > > > security
> > > > > > > > > > > > updates.
> > > > > > > > > > > > > > >> Unless there is a maintainer for Sentry
> support,
> > > > > Impala
> > > > > > > > plans to
> > > > > > > > > > > > focus
> > > > > > > > > > > > > > its
> > > > > > > > > > > > > > >> efforts on Ranger for its authorization system
> > and
> > > > > drop
> > > > > > > > Sentry.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to
> read
> > > the
> > > > > LZO
> > > > > > > > > > > compressed
> > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why this
> > > > support
> > > > > > is
> > > > > > > > not
> > > > > > > > > > > > included
> > > > > > > > > > > > > > >> directly. The Impala-lzo code interacts with
> > > > internal
> > > > > > > Impala
> > > > > > > > > > code
> > > > > > > > > > > > at a
> > > > > > > > > > > > > > >> level that is error prone and intricate. Given
> > the
> > > > low
> > > > > > > > adoption
> > > > > > > > > > of
> > > > > > > > > > > > LZO
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > >> the other compression options available,
> Impala
> > > > plans
> > > > > to
> > > > > > > > remove
> > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > >> support along with the low level interface it
> > > used.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> In addition, we also plan to deprecate several
> > > > > existing
> > > > > > > > features
> > > > > > > > > > > so
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > >> they can be removed in a future release. Here
> > is a
> > > > > > summary
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > most
> > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Impala will default to using on demand
> > metadata
> > > > > > > > management
> > > > > > > > > > > (i.e.
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> local catalog implementation) as described
> in
> > > > this
> > > > > > > > > > > documentation:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > >
> > > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > >> The old metadata system is now deprecated
> and
> > > may
> > > > > be
> > > > > > > > removed
> > > > > > > > > > > in a
> > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Impala is deprecating the Beeswax client
> > > protocol
> > > > > > (i.e.
> > > > > > > > > > clients
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > >> connect via beeswax_port) in favor of the
> > > > > HiveServer2
> > > > > > > > client
> > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Impala is deprecating the old
> > decimal_v2=false
> > > > > > behavior
> > > > > > > > from
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > >> 2.x. The decimal_v2 query option has
> > defaulted
> > > to
> > > > > > true
> > > > > > > > since
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Impala is always open to new developers, and
> we
> > > > > welcome
> > > > > > > > feedback
> > > > > > > > > > > on
> > > > > > > > > > > > > > these
> > > > > > > > > > > > > > >> plans. Further discussion of other changes and
> > > > > > > deprecations
> > > > > > > > is
> > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Joe McDonnell <jo...@cloudera.com>.
Hi Jim,
I talked with the developer for the Sentry docs bug this past week, and we
went through the list of changes needed for that. The developer is hoping
to work on it next week.
If there is any further delay, I'm open to options of working around it.
Thanks,
Joe
On Sat, May 15, 2021 at 12:43 PM Jim Apple <jb...@apache.org> wrote:
> This Sentry one I'm concerned about. The ticket has been open for a year.
> We could do a 4.0 release without the docs, or we could maybe put a note on
> the docs that Sentry support is no longer available.
>
> Any other ideas on how to tackle this while the ticket waits for a patch?
>
> On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <jo...@cloudera.com>
> wrote:
>
> > I agree that it is a good idea for us to get going on the 4.0 release.
> >
> > I'm working on a change to fix IMPALA-9690. I will post an updated review
> > once a test run completes. If everything is looking good, I would like to
> > get this in the 4.0 release.
> >
> > There is also one documentation fix that I think belongs in the Impala
> 4.0
> > release:
> > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry
> > references
> >
> > Thanks,
> > Joe
> >
> > On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com> wrote:
> >
> > > Sure thing! Let us know if you find owners for anything that someone
> > wants
> > > to champion as a 4.0 blocker.
> > >
> > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <huangquanlong@gmail.com
> >
> > > wrote:
> > >
> > > > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > >
> > > > But due to some other urgent works and the recent holiday, my
> progress
> > is
> > > > slow. Anyway, I think it's time to branch out. We've been waiting too
> > > long.
> > > > Thanks for creating the branch.
> > > >
> > > > Regards,
> > > > Quanlong
> > > >
> > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com> wrote:
> > > >
> > > > > I've created a 4.0 branch (with all changes through the end of
> April)
> > > and
> > > > > started testing a release candidate. Everyone who wants to vote,
> > > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > is a
> > > > > guide to both releasing and voting (at the bottom). I'll let you
> all
> > > know
> > > > > when I have something to vote on.
> > > > >
> > > > > Thanks!
> > > > > Jim
> > > > >
> > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > csringhofer@cloudera.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > About IMPALA-9690 (AVX support):
> > > > > > My preferred solution would be to deprecate support for x64
> without
> > > > AVX2
> > > > > in
> > > > > > 4.0, but not start removing the related logic yet.
> > > > > > - We could even add a DCHECK + flag to crash by default if no
> AVX2
> > is
> > > > > > detected, and a message that points them to Impala mailing
> > > > lists/related
> > > > > > Jira.
> > > > > > - After 4.0 is released people would be free to implement things
> > that
> > > > > > assume AVX2 support on x64, for example compiling Impala/the
> whole
> > > > > > toolchain with -mavx2. I think that currently people are held
> back
> > > from
> > > > > > doing this - why put effort in doing+benchmarking a change if you
> > > can't
> > > > > > push it to master.
> > > > > > - If a user appears with a valid use case that needs x64 without
> > AVX2
> > > > > then
> > > > > > we can revise our plans, but we shouldn't ignore low hanging
> fruits
> > > for
> > > > > the
> > > > > > sake of imaginary users :)
> > > > > >
> > > > > > -Csaba
> > > > > >
> > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > shant@superdupershant.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Regarding IMPALA-9690 if someone has the time to clean up the
> old
> > > > > > > conditional cpu logic it would be great, but if the ARM folks
> > still
> > > > > > > need some of that logic it's not critical enough to block,
> just a
> > > > very
> > > > > > > nice to have?
> > > > > > >
> > > > > > > -Shant
> > > > > > >
> > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > tim.g.armstrong@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > I agree that it would be good to get this done even if it's
> not
> > > > > > perfect,
> > > > > > > > there's so much good stuff.
> > > > > > > >
> > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > joemcdonnell@cloudera.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I agree that we should wrap up Impala 4. A large amount of
> > good
> > > > > work
> > > > > > > has
> > > > > > > > > gone in and it belongs in a release.
> > > > > > > > >
> > > > > > > > > We need to decide which breaking changes are truly blockers
> > for
> > > > an
> > > > > > > Impala 4
> > > > > > > > > release. My feeling is that we can't hold the release for
> > > > > > compatibility
> > > > > > > > > breaking changes unless someone signs up to do those
> changes.
> > > > > > Breaking
> > > > > > > > > compatibility is useful sometimes, but it can't be an
> > > indefinite
> > > > > hold
> > > > > > > on
> > > > > > > > > releases. I think a release discussion thread is a good way
> > to
> > > > > > > stimulate
> > > > > > > > > this discussion.
> > > > > > > > >
> > > > > > > > > Apart from the two JIRAs you listed, one other change
> > mentioned
> > > > in
> > > > > > the
> > > > > > > > > original email is switching to use the new on-demand
> metadata
> > > by
> > > > > > > default.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Joe
> > > > > > > > >
> > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > huangquanlong@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Sure. If there are no objections, I'll raise a discussion
> > > > thread
> > > > > > for
> > > > > > > the
> > > > > > > > > > 4.0 release. There are still some unresolved breaking
> > > changes,
> > > > > e.g.
> > > > > > > > > >
> > > > > > > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Quanlong
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > jbapple@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for bringing this up again, Quanlong! I would
> love
> > > to
> > > > > see
> > > > > > a
> > > > > > > 4.0
> > > > > > > > > > > release soon so that 4.1 releases and what not can be
> > > > prepared.
> > > > > > > Already
> > > > > > > > > > the
> > > > > > > > > > > changlist from the 3.x line is going to be enormous and
> > > could
> > > > > be
> > > > > > > hard
> > > > > > > > > for
> > > > > > > > > > > users to digest.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Reviving this thread. I think it's time to create the
> > > 4.0.0
> > > > > > > branch
> > > > > > > > > and
> > > > > > > > > > > > prepare for the release now. Any breaking change is
> > > landing
> > > > > and
> > > > > > > we
> > > > > > > > > > should
> > > > > > > > > > > > wait?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Quanlong
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Another thing that we're looking into is changing
> the
> > > > > encoded
> > > > > > > > > runtime
> > > > > > > > > > > > > profile representation to be more efficient - see
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> > So
> > > > that
> > > > > > > might
> > > > > > > > > be
> > > > > > > > > > a
> > > > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > An addendum to this - we're also considering
> > whether
> > > to
> > > > > > > increase
> > > > > > > > > > the
> > > > > > > > > > > > > > minimum CPU version so that we can simplify
> > > development
> > > > > and
> > > > > > > focus
> > > > > > > > > > > more
> > > > > > > > > > > > on
> > > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Running on commodity hardware is important and we
> > > want
> > > > > > > people to
> > > > > > > > > be
> > > > > > > > > > > > able
> > > > > > > > > > > > > > to develop on whatever machine they have access
> to.
> > > At
> > > > > most
> > > > > > > we're
> > > > > > > > > > > > likely
> > > > > > > > > > > > > > going to require AVX2 support, which has been
> > > supported
> > > > > by
> > > > > > > the
> > > > > > > > > vast
> > > > > > > > > > > > > > majority of CPUs for the better part of a decade.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Impala is starting development on Impala 4.0.
> > There
> > > > are
> > > > > > many
> > > > > > > > > > > exciting
> > > > > > > > > > > > > >> projects underway including full support for
> Hive
> > > ACID
> > > > > > > tables,
> > > > > > > > > > > > improved
> > > > > > > > > > > > > >> multithreading support, and fault tolerance
> > > > > improvements,
> > > > > > > but
> > > > > > > > > > Impala
> > > > > > > > > > > > 4.0
> > > > > > > > > > > > > >> also serves as a time to alter behavior and
> > platform
> > > > > > > support.
> > > > > > > > > Here
> > > > > > > > > > > is
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> current list of notable breaking changes that
> are
> > > > > planned
> > > > > > in
> > > > > > > > > > Impala
> > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Maintaining support for older operating systems
> > > > > requires a
> > > > > > > > > > > > considerable
> > > > > > > > > > > > > >> amount of effort, especially with changing
> > security
> > > > > > > > > requirements.
> > > > > > > > > > > Each
> > > > > > > > > > > > > >> extra operating system consumes valuable effort
> > and
> > > > can
> > > > > > > conflict
> > > > > > > > > > > with
> > > > > > > > > > > > > >> support for newer operating systems, so we have
> > > > decided
> > > > > to
> > > > > > > > > remove
> > > > > > > > > > > > > support
> > > > > > > > > > > > > >> for some older operating systems. Specifically,
> we
> > > > plan
> > > > > to
> > > > > > > drop
> > > > > > > > > > > > support
> > > > > > > > > > > > > for:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Each of these is approaching or past its end of
> > > life.
> > > > > > Since
> > > > > > > > > > Centos6
> > > > > > > > > > > > was
> > > > > > > > > > > > > >> the only operating system still needing Python
> 2.6
> > > and
> > > > > > > Python
> > > > > > > > > 2.6
> > > > > > > > > > is
> > > > > > > > > > > > > long
> > > > > > > > > > > > > >> past its end of life, Python 2.6 will no longer
> be
> > > > > > > supported. In
> > > > > > > > > > > > > addition,
> > > > > > > > > > > > > >> Centos 7 support will be focused on newer
> versions
> > > > such
> > > > > as
> > > > > > > 7.5
> > > > > > > > > and
> > > > > > > > > > > > > above.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Over the past year and a half, Impala's Ranger
> > > > > > authorization
> > > > > > > > > > > > > >> functionality has achieved parity and surpassed
> > the
> > > > > > existing
> > > > > > > > > > Sentry
> > > > > > > > > > > > > >> authorization functionality. Impala's Sentry
> > support
> > > > > > > requires
> > > > > > > > > > > ongoing
> > > > > > > > > > > > > >> effort to test and maintain, including
> maintenance
> > > on
> > > > > > Sentry
> > > > > > > > > > itself
> > > > > > > > > > > to
> > > > > > > > > > > > > >> address security updates. Given the diminished
> > > > activity
> > > > > in
> > > > > > > the
> > > > > > > > > > > Sentry
> > > > > > > > > > > > > >> community, there is no timeline for addressing
> > these
> > > > > > > security
> > > > > > > > > > > updates.
> > > > > > > > > > > > > >> Unless there is a maintainer for Sentry support,
> > > > Impala
> > > > > > > plans to
> > > > > > > > > > > focus
> > > > > > > > > > > > > its
> > > > > > > > > > > > > >> efforts on Ranger for its authorization system
> and
> > > > drop
> > > > > > > Sentry.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to read
> > the
> > > > LZO
> > > > > > > > > > compressed
> > > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why this
> > > support
> > > > > is
> > > > > > > not
> > > > > > > > > > > included
> > > > > > > > > > > > > >> directly. The Impala-lzo code interacts with
> > > internal
> > > > > > Impala
> > > > > > > > > code
> > > > > > > > > > > at a
> > > > > > > > > > > > > >> level that is error prone and intricate. Given
> the
> > > low
> > > > > > > adoption
> > > > > > > > > of
> > > > > > > > > > > LZO
> > > > > > > > > > > > > and
> > > > > > > > > > > > > >> the other compression options available, Impala
> > > plans
> > > > to
> > > > > > > remove
> > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > >> support along with the low level interface it
> > used.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> In addition, we also plan to deprecate several
> > > > existing
> > > > > > > features
> > > > > > > > > > so
> > > > > > > > > > > > that
> > > > > > > > > > > > > >> they can be removed in a future release. Here
> is a
> > > > > summary
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > most
> > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Impala will default to using on demand
> metadata
> > > > > > > management
> > > > > > > > > > (i.e.
> > > > > > > > > > > > the
> > > > > > > > > > > > > >> local catalog implementation) as described in
> > > this
> > > > > > > > > > documentation:
> > > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > >
> > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > >> The old metadata system is now deprecated and
> > may
> > > > be
> > > > > > > removed
> > > > > > > > > > in a
> > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Impala is deprecating the Beeswax client
> > protocol
> > > > > (i.e.
> > > > > > > > > clients
> > > > > > > > > > > > that
> > > > > > > > > > > > > >> connect via beeswax_port) in favor of the
> > > > HiveServer2
> > > > > > > client
> > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Impala is deprecating the old
> decimal_v2=false
> > > > > behavior
> > > > > > > from
> > > > > > > > > > > Impala
> > > > > > > > > > > > > >> 2.x. The decimal_v2 query option has
> defaulted
> > to
> > > > > true
> > > > > > > since
> > > > > > > > > > > Impala
> > > > > > > > > > > > > 3.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Impala is always open to new developers, and we
> > > > welcome
> > > > > > > feedback
> > > > > > > > > > on
> > > > > > > > > > > > > these
> > > > > > > > > > > > > >> plans. Further discussion of other changes and
> > > > > > deprecations
> > > > > > > is
> > > > > > > > > > > ongoing
> > > > > > > > > > > > > at
> > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Jim Apple <jb...@apache.org>.
This Sentry one I'm concerned about. The ticket has been open for a year.
We could do a 4.0 release without the docs, or we could maybe put a note on
the docs that Sentry support is no longer available.
Any other ideas on how to tackle this while the ticket waits for a patch?
On Thu, May 6, 2021 at 8:36 PM Joe McDonnell <jo...@cloudera.com>
wrote:
> I agree that it is a good idea for us to get going on the 4.0 release.
>
> I'm working on a change to fix IMPALA-9690. I will post an updated review
> once a test run completes. If everything is looking good, I would like to
> get this in the 4.0 release.
>
> There is also one documentation fix that I think belongs in the Impala 4.0
> release:
> https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry
> references
>
> Thanks,
> Joe
>
> On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com> wrote:
>
> > Sure thing! Let us know if you find owners for anything that someone
> wants
> > to champion as a 4.0 blocker.
> >
> > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <hu...@gmail.com>
> > wrote:
> >
> > > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > >
> > > But due to some other urgent works and the recent holiday, my progress
> is
> > > slow. Anyway, I think it's time to branch out. We've been waiting too
> > long.
> > > Thanks for creating the branch.
> > >
> > > Regards,
> > > Quanlong
> > >
> > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com> wrote:
> > >
> > > > I've created a 4.0 branch (with all changes through the end of April)
> > and
> > > > started testing a release candidate. Everyone who wants to vote,
> > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> is a
> > > > guide to both releasing and voting (at the bottom). I'll let you all
> > know
> > > > when I have something to vote on.
> > > >
> > > > Thanks!
> > > > Jim
> > > >
> > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > csringhofer@cloudera.com
> > > > >
> > > > wrote:
> > > >
> > > > > About IMPALA-9690 (AVX support):
> > > > > My preferred solution would be to deprecate support for x64 without
> > > AVX2
> > > > in
> > > > > 4.0, but not start removing the related logic yet.
> > > > > - We could even add a DCHECK + flag to crash by default if no AVX2
> is
> > > > > detected, and a message that points them to Impala mailing
> > > lists/related
> > > > > Jira.
> > > > > - After 4.0 is released people would be free to implement things
> that
> > > > > assume AVX2 support on x64, for example compiling Impala/the whole
> > > > > toolchain with -mavx2. I think that currently people are held back
> > from
> > > > > doing this - why put effort in doing+benchmarking a change if you
> > can't
> > > > > push it to master.
> > > > > - If a user appears with a valid use case that needs x64 without
> AVX2
> > > > then
> > > > > we can revise our plans, but we shouldn't ignore low hanging fruits
> > for
> > > > the
> > > > > sake of imaginary users :)
> > > > >
> > > > > -Csaba
> > > > >
> > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > shant@superdupershant.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Regarding IMPALA-9690 if someone has the time to clean up the old
> > > > > > conditional cpu logic it would be great, but if the ARM folks
> still
> > > > > > need some of that logic it's not critical enough to block, just a
> > > very
> > > > > > nice to have?
> > > > > >
> > > > > > -Shant
> > > > > >
> > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > tim.g.armstrong@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > I agree that it would be good to get this done even if it's not
> > > > > perfect,
> > > > > > > there's so much good stuff.
> > > > > > >
> > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > joemcdonnell@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I agree that we should wrap up Impala 4. A large amount of
> good
> > > > work
> > > > > > has
> > > > > > > > gone in and it belongs in a release.
> > > > > > > >
> > > > > > > > We need to decide which breaking changes are truly blockers
> for
> > > an
> > > > > > Impala 4
> > > > > > > > release. My feeling is that we can't hold the release for
> > > > > compatibility
> > > > > > > > breaking changes unless someone signs up to do those changes.
> > > > > Breaking
> > > > > > > > compatibility is useful sometimes, but it can't be an
> > indefinite
> > > > hold
> > > > > > on
> > > > > > > > releases. I think a release discussion thread is a good way
> to
> > > > > > stimulate
> > > > > > > > this discussion.
> > > > > > > >
> > > > > > > > Apart from the two JIRAs you listed, one other change
> mentioned
> > > in
> > > > > the
> > > > > > > > original email is switching to use the new on-demand metadata
> > by
> > > > > > default.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Joe
> > > > > > > >
> > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > huangquanlong@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Sure. If there are no objections, I'll raise a discussion
> > > thread
> > > > > for
> > > > > > the
> > > > > > > > > 4.0 release. There are still some unresolved breaking
> > changes,
> > > > e.g.
> > > > > > > > >
> > > > > > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Quanlong
> > > > > > > > >
> > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > jbapple@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for bringing this up again, Quanlong! I would love
> > to
> > > > see
> > > > > a
> > > > > > 4.0
> > > > > > > > > > release soon so that 4.1 releases and what not can be
> > > prepared.
> > > > > > Already
> > > > > > > > > the
> > > > > > > > > > changlist from the 3.x line is going to be enormous and
> > could
> > > > be
> > > > > > hard
> > > > > > > > for
> > > > > > > > > > users to digest.
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Reviving this thread. I think it's time to create the
> > 4.0.0
> > > > > > branch
> > > > > > > > and
> > > > > > > > > > > prepare for the release now. Any breaking change is
> > landing
> > > > and
> > > > > > we
> > > > > > > > > should
> > > > > > > > > > > wait?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Quanlong
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Another thing that we're looking into is changing the
> > > > encoded
> > > > > > > > runtime
> > > > > > > > > > > > profile representation to be more efficient - see
> > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> So
> > > that
> > > > > > might
> > > > > > > > be
> > > > > > > > > a
> > > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > An addendum to this - we're also considering
> whether
> > to
> > > > > > increase
> > > > > > > > > the
> > > > > > > > > > > > > minimum CPU version so that we can simplify
> > development
> > > > and
> > > > > > focus
> > > > > > > > > > more
> > > > > > > > > > > on
> > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > >
> > > > > > > > > > > > > Running on commodity hardware is important and we
> > want
> > > > > > people to
> > > > > > > > be
> > > > > > > > > > > able
> > > > > > > > > > > > > to develop on whatever machine they have access to.
> > At
> > > > most
> > > > > > we're
> > > > > > > > > > > likely
> > > > > > > > > > > > > going to require AVX2 support, which has been
> > supported
> > > > by
> > > > > > the
> > > > > > > > vast
> > > > > > > > > > > > > majority of CPUs for the better part of a decade.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Impala is starting development on Impala 4.0.
> There
> > > are
> > > > > many
> > > > > > > > > > exciting
> > > > > > > > > > > > >> projects underway including full support for Hive
> > ACID
> > > > > > tables,
> > > > > > > > > > > improved
> > > > > > > > > > > > >> multithreading support, and fault tolerance
> > > > improvements,
> > > > > > but
> > > > > > > > > Impala
> > > > > > > > > > > 4.0
> > > > > > > > > > > > >> also serves as a time to alter behavior and
> platform
> > > > > > support.
> > > > > > > > Here
> > > > > > > > > > is
> > > > > > > > > > > > the
> > > > > > > > > > > > >> current list of notable breaking changes that are
> > > > planned
> > > > > in
> > > > > > > > > Impala
> > > > > > > > > > > 4.0:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Maintaining support for older operating systems
> > > > requires a
> > > > > > > > > > > considerable
> > > > > > > > > > > > >> amount of effort, especially with changing
> security
> > > > > > > > requirements.
> > > > > > > > > > Each
> > > > > > > > > > > > >> extra operating system consumes valuable effort
> and
> > > can
> > > > > > conflict
> > > > > > > > > > with
> > > > > > > > > > > > >> support for newer operating systems, so we have
> > > decided
> > > > to
> > > > > > > > remove
> > > > > > > > > > > > support
> > > > > > > > > > > > >> for some older operating systems. Specifically, we
> > > plan
> > > > to
> > > > > > drop
> > > > > > > > > > > support
> > > > > > > > > > > > for:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Each of these is approaching or past its end of
> > life.
> > > > > Since
> > > > > > > > > Centos6
> > > > > > > > > > > was
> > > > > > > > > > > > >> the only operating system still needing Python 2.6
> > and
> > > > > > Python
> > > > > > > > 2.6
> > > > > > > > > is
> > > > > > > > > > > > long
> > > > > > > > > > > > >> past its end of life, Python 2.6 will no longer be
> > > > > > supported. In
> > > > > > > > > > > > addition,
> > > > > > > > > > > > >> Centos 7 support will be focused on newer versions
> > > such
> > > > as
> > > > > > 7.5
> > > > > > > > and
> > > > > > > > > > > > above.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Over the past year and a half, Impala's Ranger
> > > > > authorization
> > > > > > > > > > > > >> functionality has achieved parity and surpassed
> the
> > > > > existing
> > > > > > > > > Sentry
> > > > > > > > > > > > >> authorization functionality. Impala's Sentry
> support
> > > > > > requires
> > > > > > > > > > ongoing
> > > > > > > > > > > > >> effort to test and maintain, including maintenance
> > on
> > > > > Sentry
> > > > > > > > > itself
> > > > > > > > > > to
> > > > > > > > > > > > >> address security updates. Given the diminished
> > > activity
> > > > in
> > > > > > the
> > > > > > > > > > Sentry
> > > > > > > > > > > > >> community, there is no timeline for addressing
> these
> > > > > > security
> > > > > > > > > > updates.
> > > > > > > > > > > > >> Unless there is a maintainer for Sentry support,
> > > Impala
> > > > > > plans to
> > > > > > > > > > focus
> > > > > > > > > > > > its
> > > > > > > > > > > > >> efforts on Ranger for its authorization system and
> > > drop
> > > > > > Sentry.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to read
> the
> > > LZO
> > > > > > > > > compressed
> > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why this
> > support
> > > > is
> > > > > > not
> > > > > > > > > > included
> > > > > > > > > > > > >> directly. The Impala-lzo code interacts with
> > internal
> > > > > Impala
> > > > > > > > code
> > > > > > > > > > at a
> > > > > > > > > > > > >> level that is error prone and intricate. Given the
> > low
> > > > > > adoption
> > > > > > > > of
> > > > > > > > > > LZO
> > > > > > > > > > > > and
> > > > > > > > > > > > >> the other compression options available, Impala
> > plans
> > > to
> > > > > > remove
> > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > >> support along with the low level interface it
> used.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> In addition, we also plan to deprecate several
> > > existing
> > > > > > features
> > > > > > > > > so
> > > > > > > > > > > that
> > > > > > > > > > > > >> they can be removed in a future release. Here is a
> > > > summary
> > > > > > of
> > > > > > > > the
> > > > > > > > > > most
> > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala will default to using on demand metadata
> > > > > > management
> > > > > > > > > (i.e.
> > > > > > > > > > > the
> > > > > > > > > > > > >> local catalog implementation) as described in
> > this
> > > > > > > > > documentation:
> > > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > >
> > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > >> The old metadata system is now deprecated and
> may
> > > be
> > > > > > removed
> > > > > > > > > in a
> > > > > > > > > > > > >> future release.
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala is deprecating the Beeswax client
> protocol
> > > > (i.e.
> > > > > > > > clients
> > > > > > > > > > > that
> > > > > > > > > > > > >> connect via beeswax_port) in favor of the
> > > HiveServer2
> > > > > > client
> > > > > > > > > > > > protocol.
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala is deprecating the old decimal_v2=false
> > > > behavior
> > > > > > from
> > > > > > > > > > Impala
> > > > > > > > > > > > >> 2.x. The decimal_v2 query option has defaulted
> to
> > > > true
> > > > > > since
> > > > > > > > > > Impala
> > > > > > > > > > > > 3.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala is always open to new developers, and we
> > > welcome
> > > > > > feedback
> > > > > > > > > on
> > > > > > > > > > > > these
> > > > > > > > > > > > >> plans. Further discussion of other changes and
> > > > > deprecations
> > > > > > is
> > > > > > > > > > ongoing
> > > > > > > > > > > > at
> > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Quanlong Huang <hu...@gmail.com>.
Hi Jim,
There is another breaking change that we planned to make in 4.0:
- Don't allow ordinals in HAVING clause:
https://issues.apache.org/jira/browse/IMPALA-7844
Using ordinals in the HAVING clause is a non-standard feature. We never
document it but support it unintentionally. Paul already did the major
work. The missing part is adding a feature flag to disable it by default:
https://gerrit.cloudera.org/c/17415/
Thanks,
Quanlong
On Sun, May 9, 2021 at 1:12 AM Jim Apple <ap...@jbapple.com> wrote:
> On Fri, May 7, 2021 at 11:35 PM Csaba Ringhofer <cs...@cloudera.com>
> wrote:
>
> > I also have a commit that I would like to be merged to 4.0:
> >
> >
> https://github.com/apache/impala/commit/603091ed772f3f82511fd8fec355fe9b0126933b
> > It is a critical fix that is needed wherever the following commit also
> > exists:
> >
> >
> https://github.com/apache/impala/commit/c65d7861d9ae28f6fc592727ff699a8155dcda2c
>
>
> SG!
>
Re: Impala 4 Breaking Changes
Posted by Jim Apple <ap...@jbapple.com>.
On Fri, May 7, 2021 at 11:35 PM Csaba Ringhofer <cs...@cloudera.com>
wrote:
> I also have a commit that I would like to be merged to 4.0:
>
> https://github.com/apache/impala/commit/603091ed772f3f82511fd8fec355fe9b0126933b
> It is a critical fix that is needed wherever the following commit also
> exists:
>
> https://github.com/apache/impala/commit/c65d7861d9ae28f6fc592727ff699a8155dcda2c
SG!
Re: Impala 4 Breaking Changes
Posted by Csaba Ringhofer <cs...@cloudera.com>.
Hi Jim!
I also have a commit that I would like to be merged to 4.0:
https://github.com/apache/impala/commit/603091ed772f3f82511fd8fec355fe9b0126933b
It is a critical fix that is needed wherever the following commit also
exists:
https://github.com/apache/impala/commit/c65d7861d9ae28f6fc592727ff699a8155dcda2c
I have looked at the list of Jira's with target 4.0, one of them had a
trivial fix, uploaded it for review:
https://gerrit.cloudera.org/#/c/17412/
This is not a critical issue IMO, so I don't mind if it is not released in
4.0.
On Fri, May 7, 2021 at 5:41 PM Zoltán Borók-Nagy <bo...@apache.org>
wrote:
> Thank you, Jim!
>
> Jim Apple <ap...@jbapple.com> ezt írta (időpont: 2021. máj. 7., P 16:11):
>
> > Done!
> >
> > On Fri, May 7, 2021 at 1:42 AM Zoltán Borók-Nagy <bo...@apache.org>
> > wrote:
> >
> > > Hey Jim,
> > >
> > > My fix for IMPALA-10482 and IMPALA-10493 (the two issues are fixed in a
> > > single commit) just got merged after the 4.0 branching:
> > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=impala.git;a=commit;h=f0f083e45e2c77b1499fa6fa08ff8d9dc4a2785f
> > >
> > > Can we add this patch to the 4.0 release?
> > >
> > > Thanks,
> > > Zoltan
> > >
> > >
> > >
> > > On Fri, May 7, 2021 at 5:36 AM Joe McDonnell <
> joemcdonnell@cloudera.com>
> > > wrote:
> > >
> > > > I agree that it is a good idea for us to get going on the 4.0
> release.
> > > >
> > > > I'm working on a change to fix IMPALA-9690. I will post an updated
> > review
> > > > once a test run completes. If everything is looking good, I would
> like
> > to
> > > > get this in the 4.0 release.
> > > >
> > > > There is also one documentation fix that I think belongs in the
> Impala
> > > 4.0
> > > > release:
> > > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry
> > > > references
> > > >
> > > > Thanks,
> > > > Joe
> > > >
> > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com> wrote:
> > > >
> > > > > Sure thing! Let us know if you find owners for anything that
> someone
> > > > wants
> > > > > to champion as a 4.0 blocker.
> > > > >
> > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> > huangquanlong@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > > >
> > > > > > But due to some other urgent works and the recent holiday, my
> > > progress
> > > > is
> > > > > > slow. Anyway, I think it's time to branch out. We've been waiting
> > too
> > > > > long.
> > > > > > Thanks for creating the branch.
> > > > > >
> > > > > > Regards,
> > > > > > Quanlong
> > > > > >
> > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com>
> > wrote:
> > > > > >
> > > > > > > I've created a 4.0 branch (with all changes through the end of
> > > April)
> > > > > and
> > > > > > > started testing a release candidate. Everyone who wants to
> vote,
> > > > > > >
> > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > > is a
> > > > > > > guide to both releasing and voting (at the bottom). I'll let
> you
> > > all
> > > > > know
> > > > > > > when I have something to vote on.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Jim
> > > > > > >
> > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > > csringhofer@cloudera.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > > My preferred solution would be to deprecate support for x64
> > > without
> > > > > > AVX2
> > > > > > > in
> > > > > > > > 4.0, but not start removing the related logic yet.
> > > > > > > > - We could even add a DCHECK + flag to crash by default if no
> > > AVX2
> > > > is
> > > > > > > > detected, and a message that points them to Impala mailing
> > > > > > lists/related
> > > > > > > > Jira.
> > > > > > > > - After 4.0 is released people would be free to implement
> > things
> > > > that
> > > > > > > > assume AVX2 support on x64, for example compiling Impala/the
> > > whole
> > > > > > > > toolchain with -mavx2. I think that currently people are held
> > > back
> > > > > from
> > > > > > > > doing this - why put effort in doing+benchmarking a change if
> > you
> > > > > can't
> > > > > > > > push it to master.
> > > > > > > > - If a user appears with a valid use case that needs x64
> > without
> > > > AVX2
> > > > > > > then
> > > > > > > > we can revise our plans, but we shouldn't ignore low hanging
> > > fruits
> > > > > for
> > > > > > > the
> > > > > > > > sake of imaginary users :)
> > > > > > > >
> > > > > > > > -Csaba
> > > > > > > >
> > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > > shant@superdupershant.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > Regarding IMPALA-9690 if someone has the time to clean up
> the
> > > old
> > > > > > > > > conditional cpu logic it would be great, but if the ARM
> folks
> > > > still
> > > > > > > > > need some of that logic it's not critical enough to block,
> > > just a
> > > > > > very
> > > > > > > > > nice to have?
> > > > > > > > >
> > > > > > > > > -Shant
> > > > > > > > >
> > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > I agree that it would be good to get this done even if
> it's
> > > not
> > > > > > > > perfect,
> > > > > > > > > > there's so much good stuff.
> > > > > > > > > >
> > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I agree that we should wrap up Impala 4. A large amount
> > of
> > > > good
> > > > > > > work
> > > > > > > > > has
> > > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > > >
> > > > > > > > > > > We need to decide which breaking changes are truly
> > blockers
> > > > for
> > > > > > an
> > > > > > > > > Impala 4
> > > > > > > > > > > release. My feeling is that we can't hold the release
> for
> > > > > > > > compatibility
> > > > > > > > > > > breaking changes unless someone signs up to do those
> > > changes.
> > > > > > > > Breaking
> > > > > > > > > > > compatibility is useful sometimes, but it can't be an
> > > > > indefinite
> > > > > > > hold
> > > > > > > > > on
> > > > > > > > > > > releases. I think a release discussion thread is a good
> > way
> > > > to
> > > > > > > > > stimulate
> > > > > > > > > > > this discussion.
> > > > > > > > > > >
> > > > > > > > > > > Apart from the two JIRAs you listed, one other change
> > > > mentioned
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > original email is switching to use the new on-demand
> > > metadata
> > > > > by
> > > > > > > > > default.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Joe
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > > > huangquanlong@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Sure. If there are no objections, I'll raise a
> > discussion
> > > > > > thread
> > > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > 4.0 release. There are still some unresolved breaking
> > > > > changes,
> > > > > > > e.g.
> > > > > > > > > > > >
> > > > > > > > > > > > - IMPALA-2210: Make Parquet the default file
> format
> > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU
> requirements
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > > Quanlong
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > > > jbapple@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for bringing this up again, Quanlong! I
> would
> > > love
> > > > > to
> > > > > > > see
> > > > > > > > a
> > > > > > > > > 4.0
> > > > > > > > > > > > > release soon so that 4.1 releases and what not can
> be
> > > > > > prepared.
> > > > > > > > > Already
> > > > > > > > > > > > the
> > > > > > > > > > > > > changlist from the 3.x line is going to be enormous
> > and
> > > > > could
> > > > > > > be
> > > > > > > > > hard
> > > > > > > > > > > for
> > > > > > > > > > > > > users to digest.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Reviving this thread. I think it's time to create
> > the
> > > > > 4.0.0
> > > > > > > > > branch
> > > > > > > > > > > and
> > > > > > > > > > > > > > prepare for the release now. Any breaking change
> is
> > > > > landing
> > > > > > > and
> > > > > > > > > we
> > > > > > > > > > > > should
> > > > > > > > > > > > > > wait?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Another thing that we're looking into is
> changing
> > > the
> > > > > > > encoded
> > > > > > > > > > > runtime
> > > > > > > > > > > > > > > profile representation to be more efficient -
> see
> > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/IMPALA-9378.
> > > > So
> > > > > > that
> > > > > > > > > might
> > > > > > > > > > > be
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong
> <
> > > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > An addendum to this - we're also considering
> > > > whether
> > > > > to
> > > > > > > > > increase
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > minimum CPU version so that we can simplify
> > > > > development
> > > > > > > and
> > > > > > > > > focus
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Running on commodity hardware is important
> and
> > we
> > > > > want
> > > > > > > > > people to
> > > > > > > > > > > be
> > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > to develop on whatever machine they have
> access
> > > to.
> > > > > At
> > > > > > > most
> > > > > > > > > we're
> > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > going to require AVX2 support, which has been
> > > > > supported
> > > > > > > by
> > > > > > > > > the
> > > > > > > > > > > vast
> > > > > > > > > > > > > > > > majority of CPUs for the better part of a
> > decade.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe
> McDonnell <
> > > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Impala is starting development on Impala
> 4.0.
> > > > There
> > > > > > are
> > > > > > > > many
> > > > > > > > > > > > > exciting
> > > > > > > > > > > > > > > >> projects underway including full support for
> > > Hive
> > > > > ACID
> > > > > > > > > tables,
> > > > > > > > > > > > > > improved
> > > > > > > > > > > > > > > >> multithreading support, and fault tolerance
> > > > > > > improvements,
> > > > > > > > > but
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > > >> also serves as a time to alter behavior and
> > > > platform
> > > > > > > > > support.
> > > > > > > > > > > Here
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> current list of notable breaking changes
> that
> > > are
> > > > > > > planned
> > > > > > > > in
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Maintaining support for older operating
> > systems
> > > > > > > requires a
> > > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > > >> amount of effort, especially with changing
> > > > security
> > > > > > > > > > > requirements.
> > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > >> extra operating system consumes valuable
> > effort
> > > > and
> > > > > > can
> > > > > > > > > conflict
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > >> support for newer operating systems, so we
> > have
> > > > > > decided
> > > > > > > to
> > > > > > > > > > > remove
> > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > >> for some older operating systems.
> > Specifically,
> > > we
> > > > > > plan
> > > > > > > to
> > > > > > > > > drop
> > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Each of these is approaching or past its end
> > of
> > > > > life.
> > > > > > > > Since
> > > > > > > > > > > > Centos6
> > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > >> the only operating system still needing
> Python
> > > 2.6
> > > > > and
> > > > > > > > > Python
> > > > > > > > > > > 2.6
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > >> past its end of life, Python 2.6 will no
> > longer
> > > be
> > > > > > > > > supported. In
> > > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > > >> Centos 7 support will be focused on newer
> > > versions
> > > > > > such
> > > > > > > as
> > > > > > > > > 7.5
> > > > > > > > > > > and
> > > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Over the past year and a half, Impala's
> Ranger
> > > > > > > > authorization
> > > > > > > > > > > > > > > >> functionality has achieved parity and
> > surpassed
> > > > the
> > > > > > > > existing
> > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > >> authorization functionality. Impala's Sentry
> > > > support
> > > > > > > > > requires
> > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > >> effort to test and maintain, including
> > > maintenance
> > > > > on
> > > > > > > > Sentry
> > > > > > > > > > > > itself
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > >> address security updates. Given the
> diminished
> > > > > > activity
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > > >> community, there is no timeline for
> addressing
> > > > these
> > > > > > > > > security
> > > > > > > > > > > > > updates.
> > > > > > > > > > > > > > > >> Unless there is a maintainer for Sentry
> > support,
> > > > > > Impala
> > > > > > > > > plans to
> > > > > > > > > > > > > focus
> > > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > >> efforts on Ranger for its authorization
> system
> > > and
> > > > > > drop
> > > > > > > > > Sentry.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to
> > read
> > > > the
> > > > > > LZO
> > > > > > > > > > > > compressed
> > > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why
> this
> > > > > support
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > > > > included
> > > > > > > > > > > > > > > >> directly. The Impala-lzo code interacts with
> > > > > internal
> > > > > > > > Impala
> > > > > > > > > > > code
> > > > > > > > > > > > > at a
> > > > > > > > > > > > > > > >> level that is error prone and intricate.
> Given
> > > the
> > > > > low
> > > > > > > > > adoption
> > > > > > > > > > > of
> > > > > > > > > > > > > LZO
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> the other compression options available,
> > Impala
> > > > > plans
> > > > > > to
> > > > > > > > > remove
> > > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > > >> support along with the low level interface
> it
> > > > used.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> In addition, we also plan to deprecate
> several
> > > > > > existing
> > > > > > > > > features
> > > > > > > > > > > > so
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > >> they can be removed in a future release.
> Here
> > > is a
> > > > > > > summary
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > most
> > > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Impala will default to using on demand
> > > metadata
> > > > > > > > > management
> > > > > > > > > > > > (i.e.
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> local catalog implementation) as
> described
> > in
> > > > > this
> > > > > > > > > > > > documentation:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > > >> The old metadata system is now deprecated
> > and
> > > > may
> > > > > > be
> > > > > > > > > removed
> > > > > > > > > > > > in a
> > > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Impala is deprecating the Beeswax client
> > > > protocol
> > > > > > > (i.e.
> > > > > > > > > > > clients
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > >> connect via beeswax_port) in favor of the
> > > > > > HiveServer2
> > > > > > > > > client
> > > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Impala is deprecating the old
> > > decimal_v2=false
> > > > > > > behavior
> > > > > > > > > from
> > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query option has
> > > defaulted
> > > > to
> > > > > > > true
> > > > > > > > > since
> > > > > > > > > > > > > Impala
> > > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Impala is always open to new developers, and
> > we
> > > > > > welcome
> > > > > > > > > feedback
> > > > > > > > > > > > on
> > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > >> plans. Further discussion of other changes
> and
> > > > > > > > deprecations
> > > > > > > > > is
> > > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Zoltán Borók-Nagy <bo...@apache.org>.
Thank you, Jim!
Jim Apple <ap...@jbapple.com> ezt írta (időpont: 2021. máj. 7., P 16:11):
> Done!
>
> On Fri, May 7, 2021 at 1:42 AM Zoltán Borók-Nagy <bo...@apache.org>
> wrote:
>
> > Hey Jim,
> >
> > My fix for IMPALA-10482 and IMPALA-10493 (the two issues are fixed in a
> > single commit) just got merged after the 4.0 branching:
> >
> >
> https://gitbox.apache.org/repos/asf?p=impala.git;a=commit;h=f0f083e45e2c77b1499fa6fa08ff8d9dc4a2785f
> >
> > Can we add this patch to the 4.0 release?
> >
> > Thanks,
> > Zoltan
> >
> >
> >
> > On Fri, May 7, 2021 at 5:36 AM Joe McDonnell <jo...@cloudera.com>
> > wrote:
> >
> > > I agree that it is a good idea for us to get going on the 4.0 release.
> > >
> > > I'm working on a change to fix IMPALA-9690. I will post an updated
> review
> > > once a test run completes. If everything is looking good, I would like
> to
> > > get this in the 4.0 release.
> > >
> > > There is also one documentation fix that I think belongs in the Impala
> > 4.0
> > > release:
> > > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry
> > > references
> > >
> > > Thanks,
> > > Joe
> > >
> > > On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com> wrote:
> > >
> > > > Sure thing! Let us know if you find owners for anything that someone
> > > wants
> > > > to champion as a 4.0 blocker.
> > > >
> > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <
> huangquanlong@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > > >
> > > > > But due to some other urgent works and the recent holiday, my
> > progress
> > > is
> > > > > slow. Anyway, I think it's time to branch out. We've been waiting
> too
> > > > long.
> > > > > Thanks for creating the branch.
> > > > >
> > > > > Regards,
> > > > > Quanlong
> > > > >
> > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com>
> wrote:
> > > > >
> > > > > > I've created a 4.0 branch (with all changes through the end of
> > April)
> > > > and
> > > > > > started testing a release candidate. Everyone who wants to vote,
> > > > > >
> https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > > is a
> > > > > > guide to both releasing and voting (at the bottom). I'll let you
> > all
> > > > know
> > > > > > when I have something to vote on.
> > > > > >
> > > > > > Thanks!
> > > > > > Jim
> > > > > >
> > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > > csringhofer@cloudera.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > About IMPALA-9690 (AVX support):
> > > > > > > My preferred solution would be to deprecate support for x64
> > without
> > > > > AVX2
> > > > > > in
> > > > > > > 4.0, but not start removing the related logic yet.
> > > > > > > - We could even add a DCHECK + flag to crash by default if no
> > AVX2
> > > is
> > > > > > > detected, and a message that points them to Impala mailing
> > > > > lists/related
> > > > > > > Jira.
> > > > > > > - After 4.0 is released people would be free to implement
> things
> > > that
> > > > > > > assume AVX2 support on x64, for example compiling Impala/the
> > whole
> > > > > > > toolchain with -mavx2. I think that currently people are held
> > back
> > > > from
> > > > > > > doing this - why put effort in doing+benchmarking a change if
> you
> > > > can't
> > > > > > > push it to master.
> > > > > > > - If a user appears with a valid use case that needs x64
> without
> > > AVX2
> > > > > > then
> > > > > > > we can revise our plans, but we shouldn't ignore low hanging
> > fruits
> > > > for
> > > > > > the
> > > > > > > sake of imaginary users :)
> > > > > > >
> > > > > > > -Csaba
> > > > > > >
> > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > > shant@superdupershant.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Regarding IMPALA-9690 if someone has the time to clean up the
> > old
> > > > > > > > conditional cpu logic it would be great, but if the ARM folks
> > > still
> > > > > > > > need some of that logic it's not critical enough to block,
> > just a
> > > > > very
> > > > > > > > nice to have?
> > > > > > > >
> > > > > > > > -Shant
> > > > > > > >
> > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > > tim.g.armstrong@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I agree that it would be good to get this done even if it's
> > not
> > > > > > > perfect,
> > > > > > > > > there's so much good stuff.
> > > > > > > > >
> > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > > joemcdonnell@cloudera.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I agree that we should wrap up Impala 4. A large amount
> of
> > > good
> > > > > > work
> > > > > > > > has
> > > > > > > > > > gone in and it belongs in a release.
> > > > > > > > > >
> > > > > > > > > > We need to decide which breaking changes are truly
> blockers
> > > for
> > > > > an
> > > > > > > > Impala 4
> > > > > > > > > > release. My feeling is that we can't hold the release for
> > > > > > > compatibility
> > > > > > > > > > breaking changes unless someone signs up to do those
> > changes.
> > > > > > > Breaking
> > > > > > > > > > compatibility is useful sometimes, but it can't be an
> > > > indefinite
> > > > > > hold
> > > > > > > > on
> > > > > > > > > > releases. I think a release discussion thread is a good
> way
> > > to
> > > > > > > > stimulate
> > > > > > > > > > this discussion.
> > > > > > > > > >
> > > > > > > > > > Apart from the two JIRAs you listed, one other change
> > > mentioned
> > > > > in
> > > > > > > the
> > > > > > > > > > original email is switching to use the new on-demand
> > metadata
> > > > by
> > > > > > > > default.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Joe
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > > huangquanlong@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Sure. If there are no objections, I'll raise a
> discussion
> > > > > thread
> > > > > > > for
> > > > > > > > the
> > > > > > > > > > > 4.0 release. There are still some unresolved breaking
> > > > changes,
> > > > > > e.g.
> > > > > > > > > > >
> > > > > > > > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Quanlong
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > > jbapple@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for bringing this up again, Quanlong! I would
> > love
> > > > to
> > > > > > see
> > > > > > > a
> > > > > > > > 4.0
> > > > > > > > > > > > release soon so that 4.1 releases and what not can be
> > > > > prepared.
> > > > > > > > Already
> > > > > > > > > > > the
> > > > > > > > > > > > changlist from the 3.x line is going to be enormous
> and
> > > > could
> > > > > > be
> > > > > > > > hard
> > > > > > > > > > for
> > > > > > > > > > > > users to digest.
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Reviving this thread. I think it's time to create
> the
> > > > 4.0.0
> > > > > > > > branch
> > > > > > > > > > and
> > > > > > > > > > > > > prepare for the release now. Any breaking change is
> > > > landing
> > > > > > and
> > > > > > > > we
> > > > > > > > > > > should
> > > > > > > > > > > > > wait?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Quanlong
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Another thing that we're looking into is changing
> > the
> > > > > > encoded
> > > > > > > > > > runtime
> > > > > > > > > > > > > > profile representation to be more efficient - see
> > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/IMPALA-9378.
> > > So
> > > > > that
> > > > > > > > might
> > > > > > > > > > be
> > > > > > > > > > > a
> > > > > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > An addendum to this - we're also considering
> > > whether
> > > > to
> > > > > > > > increase
> > > > > > > > > > > the
> > > > > > > > > > > > > > > minimum CPU version so that we can simplify
> > > > development
> > > > > > and
> > > > > > > > focus
> > > > > > > > > > > > more
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Running on commodity hardware is important and
> we
> > > > want
> > > > > > > > people to
> > > > > > > > > > be
> > > > > > > > > > > > > able
> > > > > > > > > > > > > > > to develop on whatever machine they have access
> > to.
> > > > At
> > > > > > most
> > > > > > > > we're
> > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > going to require AVX2 support, which has been
> > > > supported
> > > > > > by
> > > > > > > > the
> > > > > > > > > > vast
> > > > > > > > > > > > > > > majority of CPUs for the better part of a
> decade.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Impala is starting development on Impala 4.0.
> > > There
> > > > > are
> > > > > > > many
> > > > > > > > > > > > exciting
> > > > > > > > > > > > > > >> projects underway including full support for
> > Hive
> > > > ACID
> > > > > > > > tables,
> > > > > > > > > > > > > improved
> > > > > > > > > > > > > > >> multithreading support, and fault tolerance
> > > > > > improvements,
> > > > > > > > but
> > > > > > > > > > > Impala
> > > > > > > > > > > > > 4.0
> > > > > > > > > > > > > > >> also serves as a time to alter behavior and
> > > platform
> > > > > > > > support.
> > > > > > > > > > Here
> > > > > > > > > > > > is
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> current list of notable breaking changes that
> > are
> > > > > > planned
> > > > > > > in
> > > > > > > > > > > Impala
> > > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Maintaining support for older operating
> systems
> > > > > > requires a
> > > > > > > > > > > > > considerable
> > > > > > > > > > > > > > >> amount of effort, especially with changing
> > > security
> > > > > > > > > > requirements.
> > > > > > > > > > > > Each
> > > > > > > > > > > > > > >> extra operating system consumes valuable
> effort
> > > and
> > > > > can
> > > > > > > > conflict
> > > > > > > > > > > > with
> > > > > > > > > > > > > > >> support for newer operating systems, so we
> have
> > > > > decided
> > > > > > to
> > > > > > > > > > remove
> > > > > > > > > > > > > > support
> > > > > > > > > > > > > > >> for some older operating systems.
> Specifically,
> > we
> > > > > plan
> > > > > > to
> > > > > > > > drop
> > > > > > > > > > > > > support
> > > > > > > > > > > > > > for:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Each of these is approaching or past its end
> of
> > > > life.
> > > > > > > Since
> > > > > > > > > > > Centos6
> > > > > > > > > > > > > was
> > > > > > > > > > > > > > >> the only operating system still needing Python
> > 2.6
> > > > and
> > > > > > > > Python
> > > > > > > > > > 2.6
> > > > > > > > > > > is
> > > > > > > > > > > > > > long
> > > > > > > > > > > > > > >> past its end of life, Python 2.6 will no
> longer
> > be
> > > > > > > > supported. In
> > > > > > > > > > > > > > addition,
> > > > > > > > > > > > > > >> Centos 7 support will be focused on newer
> > versions
> > > > > such
> > > > > > as
> > > > > > > > 7.5
> > > > > > > > > > and
> > > > > > > > > > > > > > above.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Over the past year and a half, Impala's Ranger
> > > > > > > authorization
> > > > > > > > > > > > > > >> functionality has achieved parity and
> surpassed
> > > the
> > > > > > > existing
> > > > > > > > > > > Sentry
> > > > > > > > > > > > > > >> authorization functionality. Impala's Sentry
> > > support
> > > > > > > > requires
> > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > >> effort to test and maintain, including
> > maintenance
> > > > on
> > > > > > > Sentry
> > > > > > > > > > > itself
> > > > > > > > > > > > to
> > > > > > > > > > > > > > >> address security updates. Given the diminished
> > > > > activity
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > Sentry
> > > > > > > > > > > > > > >> community, there is no timeline for addressing
> > > these
> > > > > > > > security
> > > > > > > > > > > > updates.
> > > > > > > > > > > > > > >> Unless there is a maintainer for Sentry
> support,
> > > > > Impala
> > > > > > > > plans to
> > > > > > > > > > > > focus
> > > > > > > > > > > > > > its
> > > > > > > > > > > > > > >> efforts on Ranger for its authorization system
> > and
> > > > > drop
> > > > > > > > Sentry.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to
> read
> > > the
> > > > > LZO
> > > > > > > > > > > compressed
> > > > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why this
> > > > support
> > > > > > is
> > > > > > > > not
> > > > > > > > > > > > included
> > > > > > > > > > > > > > >> directly. The Impala-lzo code interacts with
> > > > internal
> > > > > > > Impala
> > > > > > > > > > code
> > > > > > > > > > > > at a
> > > > > > > > > > > > > > >> level that is error prone and intricate. Given
> > the
> > > > low
> > > > > > > > adoption
> > > > > > > > > > of
> > > > > > > > > > > > LZO
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > >> the other compression options available,
> Impala
> > > > plans
> > > > > to
> > > > > > > > remove
> > > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > > >> support along with the low level interface it
> > > used.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> In addition, we also plan to deprecate several
> > > > > existing
> > > > > > > > features
> > > > > > > > > > > so
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > >> they can be removed in a future release. Here
> > is a
> > > > > > summary
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > most
> > > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Impala will default to using on demand
> > metadata
> > > > > > > > management
> > > > > > > > > > > (i.e.
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> local catalog implementation) as described
> in
> > > > this
> > > > > > > > > > > documentation:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > >
> > > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > > >> The old metadata system is now deprecated
> and
> > > may
> > > > > be
> > > > > > > > removed
> > > > > > > > > > > in a
> > > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Impala is deprecating the Beeswax client
> > > protocol
> > > > > > (i.e.
> > > > > > > > > > clients
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > >> connect via beeswax_port) in favor of the
> > > > > HiveServer2
> > > > > > > > client
> > > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > > >> -
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Impala is deprecating the old
> > decimal_v2=false
> > > > > > behavior
> > > > > > > > from
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > >> 2.x. The decimal_v2 query option has
> > defaulted
> > > to
> > > > > > true
> > > > > > > > since
> > > > > > > > > > > > Impala
> > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Impala is always open to new developers, and
> we
> > > > > welcome
> > > > > > > > feedback
> > > > > > > > > > > on
> > > > > > > > > > > > > > these
> > > > > > > > > > > > > > >> plans. Further discussion of other changes and
> > > > > > > deprecations
> > > > > > > > is
> > > > > > > > > > > > ongoing
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Jim Apple <ap...@jbapple.com>.
Done!
On Fri, May 7, 2021 at 1:42 AM Zoltán Borók-Nagy <bo...@apache.org>
wrote:
> Hey Jim,
>
> My fix for IMPALA-10482 and IMPALA-10493 (the two issues are fixed in a
> single commit) just got merged after the 4.0 branching:
>
> https://gitbox.apache.org/repos/asf?p=impala.git;a=commit;h=f0f083e45e2c77b1499fa6fa08ff8d9dc4a2785f
>
> Can we add this patch to the 4.0 release?
>
> Thanks,
> Zoltan
>
>
>
> On Fri, May 7, 2021 at 5:36 AM Joe McDonnell <jo...@cloudera.com>
> wrote:
>
> > I agree that it is a good idea for us to get going on the 4.0 release.
> >
> > I'm working on a change to fix IMPALA-9690. I will post an updated review
> > once a test run completes. If everything is looking good, I would like to
> > get this in the 4.0 release.
> >
> > There is also one documentation fix that I think belongs in the Impala
> 4.0
> > release:
> > https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry
> > references
> >
> > Thanks,
> > Joe
> >
> > On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com> wrote:
> >
> > > Sure thing! Let us know if you find owners for anything that someone
> > wants
> > > to champion as a 4.0 blocker.
> > >
> > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <huangquanlong@gmail.com
> >
> > > wrote:
> > >
> > > > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > > >
> > > > But due to some other urgent works and the recent holiday, my
> progress
> > is
> > > > slow. Anyway, I think it's time to branch out. We've been waiting too
> > > long.
> > > > Thanks for creating the branch.
> > > >
> > > > Regards,
> > > > Quanlong
> > > >
> > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com> wrote:
> > > >
> > > > > I've created a 4.0 branch (with all changes through the end of
> April)
> > > and
> > > > > started testing a release candidate. Everyone who wants to vote,
> > > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> > is a
> > > > > guide to both releasing and voting (at the bottom). I'll let you
> all
> > > know
> > > > > when I have something to vote on.
> > > > >
> > > > > Thanks!
> > > > > Jim
> > > > >
> > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > > csringhofer@cloudera.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > About IMPALA-9690 (AVX support):
> > > > > > My preferred solution would be to deprecate support for x64
> without
> > > > AVX2
> > > > > in
> > > > > > 4.0, but not start removing the related logic yet.
> > > > > > - We could even add a DCHECK + flag to crash by default if no
> AVX2
> > is
> > > > > > detected, and a message that points them to Impala mailing
> > > > lists/related
> > > > > > Jira.
> > > > > > - After 4.0 is released people would be free to implement things
> > that
> > > > > > assume AVX2 support on x64, for example compiling Impala/the
> whole
> > > > > > toolchain with -mavx2. I think that currently people are held
> back
> > > from
> > > > > > doing this - why put effort in doing+benchmarking a change if you
> > > can't
> > > > > > push it to master.
> > > > > > - If a user appears with a valid use case that needs x64 without
> > AVX2
> > > > > then
> > > > > > we can revise our plans, but we shouldn't ignore low hanging
> fruits
> > > for
> > > > > the
> > > > > > sake of imaginary users :)
> > > > > >
> > > > > > -Csaba
> > > > > >
> > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > > shant@superdupershant.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Regarding IMPALA-9690 if someone has the time to clean up the
> old
> > > > > > > conditional cpu logic it would be great, but if the ARM folks
> > still
> > > > > > > need some of that logic it's not critical enough to block,
> just a
> > > > very
> > > > > > > nice to have?
> > > > > > >
> > > > > > > -Shant
> > > > > > >
> > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > > tim.g.armstrong@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > I agree that it would be good to get this done even if it's
> not
> > > > > > perfect,
> > > > > > > > there's so much good stuff.
> > > > > > > >
> > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > > joemcdonnell@cloudera.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I agree that we should wrap up Impala 4. A large amount of
> > good
> > > > > work
> > > > > > > has
> > > > > > > > > gone in and it belongs in a release.
> > > > > > > > >
> > > > > > > > > We need to decide which breaking changes are truly blockers
> > for
> > > > an
> > > > > > > Impala 4
> > > > > > > > > release. My feeling is that we can't hold the release for
> > > > > > compatibility
> > > > > > > > > breaking changes unless someone signs up to do those
> changes.
> > > > > > Breaking
> > > > > > > > > compatibility is useful sometimes, but it can't be an
> > > indefinite
> > > > > hold
> > > > > > > on
> > > > > > > > > releases. I think a release discussion thread is a good way
> > to
> > > > > > > stimulate
> > > > > > > > > this discussion.
> > > > > > > > >
> > > > > > > > > Apart from the two JIRAs you listed, one other change
> > mentioned
> > > > in
> > > > > > the
> > > > > > > > > original email is switching to use the new on-demand
> metadata
> > > by
> > > > > > > default.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Joe
> > > > > > > > >
> > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > > huangquanlong@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Sure. If there are no objections, I'll raise a discussion
> > > > thread
> > > > > > for
> > > > > > > the
> > > > > > > > > > 4.0 release. There are still some unresolved breaking
> > > changes,
> > > > > e.g.
> > > > > > > > > >
> > > > > > > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Quanlong
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > > jbapple@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for bringing this up again, Quanlong! I would
> love
> > > to
> > > > > see
> > > > > > a
> > > > > > > 4.0
> > > > > > > > > > > release soon so that 4.1 releases and what not can be
> > > > prepared.
> > > > > > > Already
> > > > > > > > > > the
> > > > > > > > > > > changlist from the 3.x line is going to be enormous and
> > > could
> > > > > be
> > > > > > > hard
> > > > > > > > > for
> > > > > > > > > > > users to digest.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Reviving this thread. I think it's time to create the
> > > 4.0.0
> > > > > > > branch
> > > > > > > > > and
> > > > > > > > > > > > prepare for the release now. Any breaking change is
> > > landing
> > > > > and
> > > > > > > we
> > > > > > > > > > should
> > > > > > > > > > > > wait?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Quanlong
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Another thing that we're looking into is changing
> the
> > > > > encoded
> > > > > > > > > runtime
> > > > > > > > > > > > > profile representation to be more efficient - see
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> > So
> > > > that
> > > > > > > might
> > > > > > > > > be
> > > > > > > > > > a
> > > > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > An addendum to this - we're also considering
> > whether
> > > to
> > > > > > > increase
> > > > > > > > > > the
> > > > > > > > > > > > > > minimum CPU version so that we can simplify
> > > development
> > > > > and
> > > > > > > focus
> > > > > > > > > > > more
> > > > > > > > > > > > on
> > > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Running on commodity hardware is important and we
> > > want
> > > > > > > people to
> > > > > > > > > be
> > > > > > > > > > > > able
> > > > > > > > > > > > > > to develop on whatever machine they have access
> to.
> > > At
> > > > > most
> > > > > > > we're
> > > > > > > > > > > > likely
> > > > > > > > > > > > > > going to require AVX2 support, which has been
> > > supported
> > > > > by
> > > > > > > the
> > > > > > > > > vast
> > > > > > > > > > > > > > majority of CPUs for the better part of a decade.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Impala is starting development on Impala 4.0.
> > There
> > > > are
> > > > > > many
> > > > > > > > > > > exciting
> > > > > > > > > > > > > >> projects underway including full support for
> Hive
> > > ACID
> > > > > > > tables,
> > > > > > > > > > > > improved
> > > > > > > > > > > > > >> multithreading support, and fault tolerance
> > > > > improvements,
> > > > > > > but
> > > > > > > > > > Impala
> > > > > > > > > > > > 4.0
> > > > > > > > > > > > > >> also serves as a time to alter behavior and
> > platform
> > > > > > > support.
> > > > > > > > > Here
> > > > > > > > > > > is
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> current list of notable breaking changes that
> are
> > > > > planned
> > > > > > in
> > > > > > > > > > Impala
> > > > > > > > > > > > 4.0:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Maintaining support for older operating systems
> > > > > requires a
> > > > > > > > > > > > considerable
> > > > > > > > > > > > > >> amount of effort, especially with changing
> > security
> > > > > > > > > requirements.
> > > > > > > > > > > Each
> > > > > > > > > > > > > >> extra operating system consumes valuable effort
> > and
> > > > can
> > > > > > > conflict
> > > > > > > > > > > with
> > > > > > > > > > > > > >> support for newer operating systems, so we have
> > > > decided
> > > > > to
> > > > > > > > > remove
> > > > > > > > > > > > > support
> > > > > > > > > > > > > >> for some older operating systems. Specifically,
> we
> > > > plan
> > > > > to
> > > > > > > drop
> > > > > > > > > > > > support
> > > > > > > > > > > > > for:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Each of these is approaching or past its end of
> > > life.
> > > > > > Since
> > > > > > > > > > Centos6
> > > > > > > > > > > > was
> > > > > > > > > > > > > >> the only operating system still needing Python
> 2.6
> > > and
> > > > > > > Python
> > > > > > > > > 2.6
> > > > > > > > > > is
> > > > > > > > > > > > > long
> > > > > > > > > > > > > >> past its end of life, Python 2.6 will no longer
> be
> > > > > > > supported. In
> > > > > > > > > > > > > addition,
> > > > > > > > > > > > > >> Centos 7 support will be focused on newer
> versions
> > > > such
> > > > > as
> > > > > > > 7.5
> > > > > > > > > and
> > > > > > > > > > > > > above.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Over the past year and a half, Impala's Ranger
> > > > > > authorization
> > > > > > > > > > > > > >> functionality has achieved parity and surpassed
> > the
> > > > > > existing
> > > > > > > > > > Sentry
> > > > > > > > > > > > > >> authorization functionality. Impala's Sentry
> > support
> > > > > > > requires
> > > > > > > > > > > ongoing
> > > > > > > > > > > > > >> effort to test and maintain, including
> maintenance
> > > on
> > > > > > Sentry
> > > > > > > > > > itself
> > > > > > > > > > > to
> > > > > > > > > > > > > >> address security updates. Given the diminished
> > > > activity
> > > > > in
> > > > > > > the
> > > > > > > > > > > Sentry
> > > > > > > > > > > > > >> community, there is no timeline for addressing
> > these
> > > > > > > security
> > > > > > > > > > > updates.
> > > > > > > > > > > > > >> Unless there is a maintainer for Sentry support,
> > > > Impala
> > > > > > > plans to
> > > > > > > > > > > focus
> > > > > > > > > > > > > its
> > > > > > > > > > > > > >> efforts on Ranger for its authorization system
> and
> > > > drop
> > > > > > > Sentry.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to read
> > the
> > > > LZO
> > > > > > > > > > compressed
> > > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why this
> > > support
> > > > > is
> > > > > > > not
> > > > > > > > > > > included
> > > > > > > > > > > > > >> directly. The Impala-lzo code interacts with
> > > internal
> > > > > > Impala
> > > > > > > > > code
> > > > > > > > > > > at a
> > > > > > > > > > > > > >> level that is error prone and intricate. Given
> the
> > > low
> > > > > > > adoption
> > > > > > > > > of
> > > > > > > > > > > LZO
> > > > > > > > > > > > > and
> > > > > > > > > > > > > >> the other compression options available, Impala
> > > plans
> > > > to
> > > > > > > remove
> > > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > > >> support along with the low level interface it
> > used.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> In addition, we also plan to deprecate several
> > > > existing
> > > > > > > features
> > > > > > > > > > so
> > > > > > > > > > > > that
> > > > > > > > > > > > > >> they can be removed in a future release. Here
> is a
> > > > > summary
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > most
> > > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Impala will default to using on demand
> metadata
> > > > > > > management
> > > > > > > > > > (i.e.
> > > > > > > > > > > > the
> > > > > > > > > > > > > >> local catalog implementation) as described in
> > > this
> > > > > > > > > > documentation:
> > > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > >
> > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > > >> The old metadata system is now deprecated and
> > may
> > > > be
> > > > > > > removed
> > > > > > > > > > in a
> > > > > > > > > > > > > >> future release.
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Impala is deprecating the Beeswax client
> > protocol
> > > > > (i.e.
> > > > > > > > > clients
> > > > > > > > > > > > that
> > > > > > > > > > > > > >> connect via beeswax_port) in favor of the
> > > > HiveServer2
> > > > > > > client
> > > > > > > > > > > > > protocol.
> > > > > > > > > > > > > >> -
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Impala is deprecating the old
> decimal_v2=false
> > > > > behavior
> > > > > > > from
> > > > > > > > > > > Impala
> > > > > > > > > > > > > >> 2.x. The decimal_v2 query option has
> defaulted
> > to
> > > > > true
> > > > > > > since
> > > > > > > > > > > Impala
> > > > > > > > > > > > > 3.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Impala is always open to new developers, and we
> > > > welcome
> > > > > > > feedback
> > > > > > > > > > on
> > > > > > > > > > > > > these
> > > > > > > > > > > > > >> plans. Further discussion of other changes and
> > > > > > deprecations
> > > > > > > is
> > > > > > > > > > > ongoing
> > > > > > > > > > > > > at
> > > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Zoltán Borók-Nagy <bo...@apache.org>.
Hey Jim,
My fix for IMPALA-10482 and IMPALA-10493 (the two issues are fixed in a
single commit) just got merged after the 4.0 branching:
https://gitbox.apache.org/repos/asf?p=impala.git;a=commit;h=f0f083e45e2c77b1499fa6fa08ff8d9dc4a2785f
Can we add this patch to the 4.0 release?
Thanks,
Zoltan
On Fri, May 7, 2021 at 5:36 AM Joe McDonnell <jo...@cloudera.com>
wrote:
> I agree that it is a good idea for us to get going on the 4.0 release.
>
> I'm working on a change to fix IMPALA-9690. I will post an updated review
> once a test run completes. If everything is looking good, I would like to
> get this in the 4.0 release.
>
> There is also one documentation fix that I think belongs in the Impala 4.0
> release:
> https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry
> references
>
> Thanks,
> Joe
>
> On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com> wrote:
>
> > Sure thing! Let us know if you find owners for anything that someone
> wants
> > to champion as a 4.0 blocker.
> >
> > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <hu...@gmail.com>
> > wrote:
> >
> > > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > >
> > > But due to some other urgent works and the recent holiday, my progress
> is
> > > slow. Anyway, I think it's time to branch out. We've been waiting too
> > long.
> > > Thanks for creating the branch.
> > >
> > > Regards,
> > > Quanlong
> > >
> > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com> wrote:
> > >
> > > > I've created a 4.0 branch (with all changes through the end of April)
> > and
> > > > started testing a release candidate. Everyone who wants to vote,
> > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> is a
> > > > guide to both releasing and voting (at the bottom). I'll let you all
> > know
> > > > when I have something to vote on.
> > > >
> > > > Thanks!
> > > > Jim
> > > >
> > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > csringhofer@cloudera.com
> > > > >
> > > > wrote:
> > > >
> > > > > About IMPALA-9690 (AVX support):
> > > > > My preferred solution would be to deprecate support for x64 without
> > > AVX2
> > > > in
> > > > > 4.0, but not start removing the related logic yet.
> > > > > - We could even add a DCHECK + flag to crash by default if no AVX2
> is
> > > > > detected, and a message that points them to Impala mailing
> > > lists/related
> > > > > Jira.
> > > > > - After 4.0 is released people would be free to implement things
> that
> > > > > assume AVX2 support on x64, for example compiling Impala/the whole
> > > > > toolchain with -mavx2. I think that currently people are held back
> > from
> > > > > doing this - why put effort in doing+benchmarking a change if you
> > can't
> > > > > push it to master.
> > > > > - If a user appears with a valid use case that needs x64 without
> AVX2
> > > > then
> > > > > we can revise our plans, but we shouldn't ignore low hanging fruits
> > for
> > > > the
> > > > > sake of imaginary users :)
> > > > >
> > > > > -Csaba
> > > > >
> > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > shant@superdupershant.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Regarding IMPALA-9690 if someone has the time to clean up the old
> > > > > > conditional cpu logic it would be great, but if the ARM folks
> still
> > > > > > need some of that logic it's not critical enough to block, just a
> > > very
> > > > > > nice to have?
> > > > > >
> > > > > > -Shant
> > > > > >
> > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > tim.g.armstrong@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > I agree that it would be good to get this done even if it's not
> > > > > perfect,
> > > > > > > there's so much good stuff.
> > > > > > >
> > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > joemcdonnell@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I agree that we should wrap up Impala 4. A large amount of
> good
> > > > work
> > > > > > has
> > > > > > > > gone in and it belongs in a release.
> > > > > > > >
> > > > > > > > We need to decide which breaking changes are truly blockers
> for
> > > an
> > > > > > Impala 4
> > > > > > > > release. My feeling is that we can't hold the release for
> > > > > compatibility
> > > > > > > > breaking changes unless someone signs up to do those changes.
> > > > > Breaking
> > > > > > > > compatibility is useful sometimes, but it can't be an
> > indefinite
> > > > hold
> > > > > > on
> > > > > > > > releases. I think a release discussion thread is a good way
> to
> > > > > > stimulate
> > > > > > > > this discussion.
> > > > > > > >
> > > > > > > > Apart from the two JIRAs you listed, one other change
> mentioned
> > > in
> > > > > the
> > > > > > > > original email is switching to use the new on-demand metadata
> > by
> > > > > > default.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Joe
> > > > > > > >
> > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > huangquanlong@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Sure. If there are no objections, I'll raise a discussion
> > > thread
> > > > > for
> > > > > > the
> > > > > > > > > 4.0 release. There are still some unresolved breaking
> > changes,
> > > > e.g.
> > > > > > > > >
> > > > > > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Quanlong
> > > > > > > > >
> > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > jbapple@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for bringing this up again, Quanlong! I would love
> > to
> > > > see
> > > > > a
> > > > > > 4.0
> > > > > > > > > > release soon so that 4.1 releases and what not can be
> > > prepared.
> > > > > > Already
> > > > > > > > > the
> > > > > > > > > > changlist from the 3.x line is going to be enormous and
> > could
> > > > be
> > > > > > hard
> > > > > > > > for
> > > > > > > > > > users to digest.
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > > huangquanlong@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Reviving this thread. I think it's time to create the
> > 4.0.0
> > > > > > branch
> > > > > > > > and
> > > > > > > > > > > prepare for the release now. Any breaking change is
> > landing
> > > > and
> > > > > > we
> > > > > > > > > should
> > > > > > > > > > > wait?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Quanlong
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > > tarmstrong@cloudera.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Another thing that we're looking into is changing the
> > > > encoded
> > > > > > > > runtime
> > > > > > > > > > > > profile representation to be more efficient - see
> > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> So
> > > that
> > > > > > might
> > > > > > > > be
> > > > > > > > > a
> > > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > An addendum to this - we're also considering
> whether
> > to
> > > > > > increase
> > > > > > > > > the
> > > > > > > > > > > > > minimum CPU version so that we can simplify
> > development
> > > > and
> > > > > > focus
> > > > > > > > > > more
> > > > > > > > > > > on
> > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > >
> > > > > > > > > > > > > Running on commodity hardware is important and we
> > want
> > > > > > people to
> > > > > > > > be
> > > > > > > > > > > able
> > > > > > > > > > > > > to develop on whatever machine they have access to.
> > At
> > > > most
> > > > > > we're
> > > > > > > > > > > likely
> > > > > > > > > > > > > going to require AVX2 support, which has been
> > supported
> > > > by
> > > > > > the
> > > > > > > > vast
> > > > > > > > > > > > > majority of CPUs for the better part of a decade.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Impala is starting development on Impala 4.0.
> There
> > > are
> > > > > many
> > > > > > > > > > exciting
> > > > > > > > > > > > >> projects underway including full support for Hive
> > ACID
> > > > > > tables,
> > > > > > > > > > > improved
> > > > > > > > > > > > >> multithreading support, and fault tolerance
> > > > improvements,
> > > > > > but
> > > > > > > > > Impala
> > > > > > > > > > > 4.0
> > > > > > > > > > > > >> also serves as a time to alter behavior and
> platform
> > > > > > support.
> > > > > > > > Here
> > > > > > > > > > is
> > > > > > > > > > > > the
> > > > > > > > > > > > >> current list of notable breaking changes that are
> > > > planned
> > > > > in
> > > > > > > > > Impala
> > > > > > > > > > > 4.0:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Maintaining support for older operating systems
> > > > requires a
> > > > > > > > > > > considerable
> > > > > > > > > > > > >> amount of effort, especially with changing
> security
> > > > > > > > requirements.
> > > > > > > > > > Each
> > > > > > > > > > > > >> extra operating system consumes valuable effort
> and
> > > can
> > > > > > conflict
> > > > > > > > > > with
> > > > > > > > > > > > >> support for newer operating systems, so we have
> > > decided
> > > > to
> > > > > > > > remove
> > > > > > > > > > > > support
> > > > > > > > > > > > >> for some older operating systems. Specifically, we
> > > plan
> > > > to
> > > > > > drop
> > > > > > > > > > > support
> > > > > > > > > > > > for:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Each of these is approaching or past its end of
> > life.
> > > > > Since
> > > > > > > > > Centos6
> > > > > > > > > > > was
> > > > > > > > > > > > >> the only operating system still needing Python 2.6
> > and
> > > > > > Python
> > > > > > > > 2.6
> > > > > > > > > is
> > > > > > > > > > > > long
> > > > > > > > > > > > >> past its end of life, Python 2.6 will no longer be
> > > > > > supported. In
> > > > > > > > > > > > addition,
> > > > > > > > > > > > >> Centos 7 support will be focused on newer versions
> > > such
> > > > as
> > > > > > 7.5
> > > > > > > > and
> > > > > > > > > > > > above.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Over the past year and a half, Impala's Ranger
> > > > > authorization
> > > > > > > > > > > > >> functionality has achieved parity and surpassed
> the
> > > > > existing
> > > > > > > > > Sentry
> > > > > > > > > > > > >> authorization functionality. Impala's Sentry
> support
> > > > > > requires
> > > > > > > > > > ongoing
> > > > > > > > > > > > >> effort to test and maintain, including maintenance
> > on
> > > > > Sentry
> > > > > > > > > itself
> > > > > > > > > > to
> > > > > > > > > > > > >> address security updates. Given the diminished
> > > activity
> > > > in
> > > > > > the
> > > > > > > > > > Sentry
> > > > > > > > > > > > >> community, there is no timeline for addressing
> these
> > > > > > security
> > > > > > > > > > updates.
> > > > > > > > > > > > >> Unless there is a maintainer for Sentry support,
> > > Impala
> > > > > > plans to
> > > > > > > > > > focus
> > > > > > > > > > > > its
> > > > > > > > > > > > >> efforts on Ranger for its authorization system and
> > > drop
> > > > > > Sentry.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to read
> the
> > > LZO
> > > > > > > > > compressed
> > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why this
> > support
> > > > is
> > > > > > not
> > > > > > > > > > included
> > > > > > > > > > > > >> directly. The Impala-lzo code interacts with
> > internal
> > > > > Impala
> > > > > > > > code
> > > > > > > > > > at a
> > > > > > > > > > > > >> level that is error prone and intricate. Given the
> > low
> > > > > > adoption
> > > > > > > > of
> > > > > > > > > > LZO
> > > > > > > > > > > > and
> > > > > > > > > > > > >> the other compression options available, Impala
> > plans
> > > to
> > > > > > remove
> > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > >> support along with the low level interface it
> used.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> In addition, we also plan to deprecate several
> > > existing
> > > > > > features
> > > > > > > > > so
> > > > > > > > > > > that
> > > > > > > > > > > > >> they can be removed in a future release. Here is a
> > > > summary
> > > > > > of
> > > > > > > > the
> > > > > > > > > > most
> > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala will default to using on demand metadata
> > > > > > management
> > > > > > > > > (i.e.
> > > > > > > > > > > the
> > > > > > > > > > > > >> local catalog implementation) as described in
> > this
> > > > > > > > > documentation:
> > > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > >
> > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > >> The old metadata system is now deprecated and
> may
> > > be
> > > > > > removed
> > > > > > > > > in a
> > > > > > > > > > > > >> future release.
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala is deprecating the Beeswax client
> protocol
> > > > (i.e.
> > > > > > > > clients
> > > > > > > > > > > that
> > > > > > > > > > > > >> connect via beeswax_port) in favor of the
> > > HiveServer2
> > > > > > client
> > > > > > > > > > > > protocol.
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala is deprecating the old decimal_v2=false
> > > > behavior
> > > > > > from
> > > > > > > > > > Impala
> > > > > > > > > > > > >> 2.x. The decimal_v2 query option has defaulted
> to
> > > > true
> > > > > > since
> > > > > > > > > > Impala
> > > > > > > > > > > > 3.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala is always open to new developers, and we
> > > welcome
> > > > > > feedback
> > > > > > > > > on
> > > > > > > > > > > > these
> > > > > > > > > > > > >> plans. Further discussion of other changes and
> > > > > deprecations
> > > > > > is
> > > > > > > > > > ongoing
> > > > > > > > > > > > at
> > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Joe McDonnell <jo...@cloudera.com>.
I agree that it is a good idea for us to get going on the 4.0 release.
I'm working on a change to fix IMPALA-9690. I will post an updated review
once a test run completes. If everything is looking good, I would like to
get this in the 4.0 release.
There is also one documentation fix that I think belongs in the Impala 4.0
release:
https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry references
Thanks,
Joe
On Thu, May 6, 2021 at 7:55 PM Jim Apple <ap...@jbapple.com> wrote:
> Sure thing! Let us know if you find owners for anything that someone wants
> to champion as a 4.0 blocker.
>
> On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <hu...@gmail.com>
> wrote:
>
> > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> >
> > But due to some other urgent works and the recent holiday, my progress is
> > slow. Anyway, I think it's time to branch out. We've been waiting too
> long.
> > Thanks for creating the branch.
> >
> > Regards,
> > Quanlong
> >
> > On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com> wrote:
> >
> > > I've created a 4.0 branch (with all changes through the end of April)
> and
> > > started testing a release candidate. Everyone who wants to vote,
> > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release is a
> > > guide to both releasing and voting (at the bottom). I'll let you all
> know
> > > when I have something to vote on.
> > >
> > > Thanks!
> > > Jim
> > >
> > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > csringhofer@cloudera.com
> > > >
> > > wrote:
> > >
> > > > About IMPALA-9690 (AVX support):
> > > > My preferred solution would be to deprecate support for x64 without
> > AVX2
> > > in
> > > > 4.0, but not start removing the related logic yet.
> > > > - We could even add a DCHECK + flag to crash by default if no AVX2 is
> > > > detected, and a message that points them to Impala mailing
> > lists/related
> > > > Jira.
> > > > - After 4.0 is released people would be free to implement things that
> > > > assume AVX2 support on x64, for example compiling Impala/the whole
> > > > toolchain with -mavx2. I think that currently people are held back
> from
> > > > doing this - why put effort in doing+benchmarking a change if you
> can't
> > > > push it to master.
> > > > - If a user appears with a valid use case that needs x64 without AVX2
> > > then
> > > > we can revise our plans, but we shouldn't ignore low hanging fruits
> for
> > > the
> > > > sake of imaginary users :)
> > > >
> > > > -Csaba
> > > >
> > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > shant@superdupershant.com
> > > > >
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Regarding IMPALA-9690 if someone has the time to clean up the old
> > > > > conditional cpu logic it would be great, but if the ARM folks still
> > > > > need some of that logic it's not critical enough to block, just a
> > very
> > > > > nice to have?
> > > > >
> > > > > -Shant
> > > > >
> > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > tim.g.armstrong@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > I agree that it would be good to get this done even if it's not
> > > > perfect,
> > > > > > there's so much good stuff.
> > > > > >
> > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > joemcdonnell@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I agree that we should wrap up Impala 4. A large amount of good
> > > work
> > > > > has
> > > > > > > gone in and it belongs in a release.
> > > > > > >
> > > > > > > We need to decide which breaking changes are truly blockers for
> > an
> > > > > Impala 4
> > > > > > > release. My feeling is that we can't hold the release for
> > > > compatibility
> > > > > > > breaking changes unless someone signs up to do those changes.
> > > > Breaking
> > > > > > > compatibility is useful sometimes, but it can't be an
> indefinite
> > > hold
> > > > > on
> > > > > > > releases. I think a release discussion thread is a good way to
> > > > > stimulate
> > > > > > > this discussion.
> > > > > > >
> > > > > > > Apart from the two JIRAs you listed, one other change mentioned
> > in
> > > > the
> > > > > > > original email is switching to use the new on-demand metadata
> by
> > > > > default.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Joe
> > > > > > >
> > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > huangquanlong@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Sure. If there are no objections, I'll raise a discussion
> > thread
> > > > for
> > > > > the
> > > > > > > > 4.0 release. There are still some unresolved breaking
> changes,
> > > e.g.
> > > > > > > >
> > > > > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Quanlong
> > > > > > > >
> > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> jbapple@apache.org
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for bringing this up again, Quanlong! I would love
> to
> > > see
> > > > a
> > > > > 4.0
> > > > > > > > > release soon so that 4.1 releases and what not can be
> > prepared.
> > > > > Already
> > > > > > > > the
> > > > > > > > > changlist from the 3.x line is going to be enormous and
> could
> > > be
> > > > > hard
> > > > > > > for
> > > > > > > > > users to digest.
> > > > > > > > >
> > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > huangquanlong@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Reviving this thread. I think it's time to create the
> 4.0.0
> > > > > branch
> > > > > > > and
> > > > > > > > > > prepare for the release now. Any breaking change is
> landing
> > > and
> > > > > we
> > > > > > > > should
> > > > > > > > > > wait?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Quanlong
> > > > > > > > > >
> > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > tarmstrong@cloudera.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Another thing that we're looking into is changing the
> > > encoded
> > > > > > > runtime
> > > > > > > > > > > profile representation to be more efficient - see
> > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. So
> > that
> > > > > might
> > > > > > > be
> > > > > > > > a
> > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > An addendum to this - we're also considering whether
> to
> > > > > increase
> > > > > > > > the
> > > > > > > > > > > > minimum CPU version so that we can simplify
> development
> > > and
> > > > > focus
> > > > > > > > > more
> > > > > > > > > > on
> > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > >
> > > > > > > > > > > > Running on commodity hardware is important and we
> want
> > > > > people to
> > > > > > > be
> > > > > > > > > > able
> > > > > > > > > > > > to develop on whatever machine they have access to.
> At
> > > most
> > > > > we're
> > > > > > > > > > likely
> > > > > > > > > > > > going to require AVX2 support, which has been
> supported
> > > by
> > > > > the
> > > > > > > vast
> > > > > > > > > > > > majority of CPUs for the better part of a decade.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Impala is starting development on Impala 4.0. There
> > are
> > > > many
> > > > > > > > > exciting
> > > > > > > > > > > >> projects underway including full support for Hive
> ACID
> > > > > tables,
> > > > > > > > > > improved
> > > > > > > > > > > >> multithreading support, and fault tolerance
> > > improvements,
> > > > > but
> > > > > > > > Impala
> > > > > > > > > > 4.0
> > > > > > > > > > > >> also serves as a time to alter behavior and platform
> > > > > support.
> > > > > > > Here
> > > > > > > > > is
> > > > > > > > > > > the
> > > > > > > > > > > >> current list of notable breaking changes that are
> > > planned
> > > > in
> > > > > > > > Impala
> > > > > > > > > > 4.0:
> > > > > > > > > > > >>
> > > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > > >>
> > > > > > > > > > > >> Maintaining support for older operating systems
> > > requires a
> > > > > > > > > > considerable
> > > > > > > > > > > >> amount of effort, especially with changing security
> > > > > > > requirements.
> > > > > > > > > Each
> > > > > > > > > > > >> extra operating system consumes valuable effort and
> > can
> > > > > conflict
> > > > > > > > > with
> > > > > > > > > > > >> support for newer operating systems, so we have
> > decided
> > > to
> > > > > > > remove
> > > > > > > > > > > support
> > > > > > > > > > > >> for some older operating systems. Specifically, we
> > plan
> > > to
> > > > > drop
> > > > > > > > > > support
> > > > > > > > > > > for:
> > > > > > > > > > > >>
> > > > > > > > > > > >> -
> > > > > > > > > > > >>
> > > > > > > > > > > >> Centos 6
> > > > > > > > > > > >> -
> > > > > > > > > > > >>
> > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > >> -
> > > > > > > > > > > >>
> > > > > > > > > > > >> Debian 8
> > > > > > > > > > > >>
> > > > > > > > > > > >> Each of these is approaching or past its end of
> life.
> > > > Since
> > > > > > > > Centos6
> > > > > > > > > > was
> > > > > > > > > > > >> the only operating system still needing Python 2.6
> and
> > > > > Python
> > > > > > > 2.6
> > > > > > > > is
> > > > > > > > > > > long
> > > > > > > > > > > >> past its end of life, Python 2.6 will no longer be
> > > > > supported. In
> > > > > > > > > > > addition,
> > > > > > > > > > > >> Centos 7 support will be focused on newer versions
> > such
> > > as
> > > > > 7.5
> > > > > > > and
> > > > > > > > > > > above.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > >>
> > > > > > > > > > > >> Over the past year and a half, Impala's Ranger
> > > > authorization
> > > > > > > > > > > >> functionality has achieved parity and surpassed the
> > > > existing
> > > > > > > > Sentry
> > > > > > > > > > > >> authorization functionality. Impala's Sentry support
> > > > > requires
> > > > > > > > > ongoing
> > > > > > > > > > > >> effort to test and maintain, including maintenance
> on
> > > > Sentry
> > > > > > > > itself
> > > > > > > > > to
> > > > > > > > > > > >> address security updates. Given the diminished
> > activity
> > > in
> > > > > the
> > > > > > > > > Sentry
> > > > > > > > > > > >> community, there is no timeline for addressing these
> > > > > security
> > > > > > > > > updates.
> > > > > > > > > > > >> Unless there is a maintainer for Sentry support,
> > Impala
> > > > > plans to
> > > > > > > > > focus
> > > > > > > > > > > its
> > > > > > > > > > > >> efforts on Ranger for its authorization system and
> > drop
> > > > > Sentry.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > >>
> > > > > > > > > > > >> Impala-lzo provides code to allow Impala to read the
> > LZO
> > > > > > > > compressed
> > > > > > > > > > > >> tables. LZO is GPL licensed, which is why this
> support
> > > is
> > > > > not
> > > > > > > > > included
> > > > > > > > > > > >> directly. The Impala-lzo code interacts with
> internal
> > > > Impala
> > > > > > > code
> > > > > > > > > at a
> > > > > > > > > > > >> level that is error prone and intricate. Given the
> low
> > > > > adoption
> > > > > > > of
> > > > > > > > > LZO
> > > > > > > > > > > and
> > > > > > > > > > > >> the other compression options available, Impala
> plans
> > to
> > > > > remove
> > > > > > > > > > > Impala-lzo
> > > > > > > > > > > >> support along with the low level interface it used.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > >>
> > > > > > > > > > > >> In addition, we also plan to deprecate several
> > existing
> > > > > features
> > > > > > > > so
> > > > > > > > > > that
> > > > > > > > > > > >> they can be removed in a future release. Here is a
> > > summary
> > > > > of
> > > > > > > the
> > > > > > > > > most
> > > > > > > > > > > >> notable ones:
> > > > > > > > > > > >>
> > > > > > > > > > > >> -
> > > > > > > > > > > >>
> > > > > > > > > > > >> Impala will default to using on demand metadata
> > > > > management
> > > > > > > > (i.e.
> > > > > > > > > > the
> > > > > > > > > > > >> local catalog implementation) as described in
> this
> > > > > > > > documentation:
> > > > > > > > > > > >>
> > > > > > > > > >
> > > > >
> http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > >> The old metadata system is now deprecated and may
> > be
> > > > > removed
> > > > > > > > in a
> > > > > > > > > > > >> future release.
> > > > > > > > > > > >> -
> > > > > > > > > > > >>
> > > > > > > > > > > >> Impala is deprecating the Beeswax client protocol
> > > (i.e.
> > > > > > > clients
> > > > > > > > > > that
> > > > > > > > > > > >> connect via beeswax_port) in favor of the
> > HiveServer2
> > > > > client
> > > > > > > > > > > protocol.
> > > > > > > > > > > >> -
> > > > > > > > > > > >>
> > > > > > > > > > > >> Impala is deprecating the old decimal_v2=false
> > > behavior
> > > > > from
> > > > > > > > > Impala
> > > > > > > > > > > >> 2.x. The decimal_v2 query option has defaulted to
> > > true
> > > > > since
> > > > > > > > > Impala
> > > > > > > > > > > 3.
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> Impala is always open to new developers, and we
> > welcome
> > > > > feedback
> > > > > > > > on
> > > > > > > > > > > these
> > > > > > > > > > > >> plans. Further discussion of other changes and
> > > > deprecations
> > > > > is
> > > > > > > > > ongoing
> > > > > > > > > > > at
> > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks,
> > > > > > > > > > > >>
> > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Jim Apple <ap...@jbapple.com>.
Sure thing! Let us know if you find owners for anything that someone wants
to champion as a 4.0 blocker.
On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <hu...@gmail.com>
wrote:
> Thank Jim! I planned to go through all the JIRAs target at 4.0:
>
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
>
> But due to some other urgent works and the recent holiday, my progress is
> slow. Anyway, I think it's time to branch out. We've been waiting too long.
> Thanks for creating the branch.
>
> Regards,
> Quanlong
>
> On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com> wrote:
>
> > I've created a 4.0 branch (with all changes through the end of April) and
> > started testing a release candidate. Everyone who wants to vote,
> > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release is a
> > guide to both releasing and voting (at the bottom). I'll let you all know
> > when I have something to vote on.
> >
> > Thanks!
> > Jim
> >
> > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> csringhofer@cloudera.com
> > >
> > wrote:
> >
> > > About IMPALA-9690 (AVX support):
> > > My preferred solution would be to deprecate support for x64 without
> AVX2
> > in
> > > 4.0, but not start removing the related logic yet.
> > > - We could even add a DCHECK + flag to crash by default if no AVX2 is
> > > detected, and a message that points them to Impala mailing
> lists/related
> > > Jira.
> > > - After 4.0 is released people would be free to implement things that
> > > assume AVX2 support on x64, for example compiling Impala/the whole
> > > toolchain with -mavx2. I think that currently people are held back from
> > > doing this - why put effort in doing+benchmarking a change if you can't
> > > push it to master.
> > > - If a user appears with a valid use case that needs x64 without AVX2
> > then
> > > we can revise our plans, but we shouldn't ignore low hanging fruits for
> > the
> > > sake of imaginary users :)
> > >
> > > -Csaba
> > >
> > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > shant@superdupershant.com
> > > >
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Regarding IMPALA-9690 if someone has the time to clean up the old
> > > > conditional cpu logic it would be great, but if the ARM folks still
> > > > need some of that logic it's not critical enough to block, just a
> very
> > > > nice to have?
> > > >
> > > > -Shant
> > > >
> > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > tim.g.armstrong@gmail.com>
> > > > wrote:
> > > > >
> > > > > I agree that it would be good to get this done even if it's not
> > > perfect,
> > > > > there's so much good stuff.
> > > > >
> > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > joemcdonnell@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > I agree that we should wrap up Impala 4. A large amount of good
> > work
> > > > has
> > > > > > gone in and it belongs in a release.
> > > > > >
> > > > > > We need to decide which breaking changes are truly blockers for
> an
> > > > Impala 4
> > > > > > release. My feeling is that we can't hold the release for
> > > compatibility
> > > > > > breaking changes unless someone signs up to do those changes.
> > > Breaking
> > > > > > compatibility is useful sometimes, but it can't be an indefinite
> > hold
> > > > on
> > > > > > releases. I think a release discussion thread is a good way to
> > > > stimulate
> > > > > > this discussion.
> > > > > >
> > > > > > Apart from the two JIRAs you listed, one other change mentioned
> in
> > > the
> > > > > > original email is switching to use the new on-demand metadata by
> > > > default.
> > > > > >
> > > > > > Thanks,
> > > > > > Joe
> > > > > >
> > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > huangquanlong@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Sure. If there are no objections, I'll raise a discussion
> thread
> > > for
> > > > the
> > > > > > > 4.0 release. There are still some unresolved breaking changes,
> > e.g.
> > > > > > >
> > > > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > > > >
> > > > > > > Thanks
> > > > > > > Quanlong
> > > > > > >
> > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <jbapple@apache.org
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Thanks for bringing this up again, Quanlong! I would love to
> > see
> > > a
> > > > 4.0
> > > > > > > > release soon so that 4.1 releases and what not can be
> prepared.
> > > > Already
> > > > > > > the
> > > > > > > > changlist from the 3.x line is going to be enormous and could
> > be
> > > > hard
> > > > > > for
> > > > > > > > users to digest.
> > > > > > > >
> > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > huangquanlong@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Reviving this thread. I think it's time to create the 4.0.0
> > > > branch
> > > > > > and
> > > > > > > > > prepare for the release now. Any breaking change is landing
> > and
> > > > we
> > > > > > > should
> > > > > > > > > wait?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Quanlong
> > > > > > > > >
> > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > tarmstrong@cloudera.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Another thing that we're looking into is changing the
> > encoded
> > > > > > runtime
> > > > > > > > > > profile representation to be more efficient - see
> > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. So
> that
> > > > might
> > > > > > be
> > > > > > > a
> > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > An addendum to this - we're also considering whether to
> > > > increase
> > > > > > > the
> > > > > > > > > > > minimum CPU version so that we can simplify development
> > and
> > > > focus
> > > > > > > > more
> > > > > > > > > on
> > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > >
> > > > > > > > > > > Running on commodity hardware is important and we want
> > > > people to
> > > > > > be
> > > > > > > > > able
> > > > > > > > > > > to develop on whatever machine they have access to. At
> > most
> > > > we're
> > > > > > > > > likely
> > > > > > > > > > > going to require AVX2 support, which has been supported
> > by
> > > > the
> > > > > > vast
> > > > > > > > > > > majority of CPUs for the better part of a decade.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Impala is starting development on Impala 4.0. There
> are
> > > many
> > > > > > > > exciting
> > > > > > > > > > >> projects underway including full support for Hive ACID
> > > > tables,
> > > > > > > > > improved
> > > > > > > > > > >> multithreading support, and fault tolerance
> > improvements,
> > > > but
> > > > > > > Impala
> > > > > > > > > 4.0
> > > > > > > > > > >> also serves as a time to alter behavior and platform
> > > > support.
> > > > > > Here
> > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > >> current list of notable breaking changes that are
> > planned
> > > in
> > > > > > > Impala
> > > > > > > > > 4.0:
> > > > > > > > > > >>
> > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > >>
> > > > > > > > > > >> Maintaining support for older operating systems
> > requires a
> > > > > > > > > considerable
> > > > > > > > > > >> amount of effort, especially with changing security
> > > > > > requirements.
> > > > > > > > Each
> > > > > > > > > > >> extra operating system consumes valuable effort and
> can
> > > > conflict
> > > > > > > > with
> > > > > > > > > > >> support for newer operating systems, so we have
> decided
> > to
> > > > > > remove
> > > > > > > > > > support
> > > > > > > > > > >> for some older operating systems. Specifically, we
> plan
> > to
> > > > drop
> > > > > > > > > support
> > > > > > > > > > for:
> > > > > > > > > > >>
> > > > > > > > > > >> -
> > > > > > > > > > >>
> > > > > > > > > > >> Centos 6
> > > > > > > > > > >> -
> > > > > > > > > > >>
> > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > >> -
> > > > > > > > > > >>
> > > > > > > > > > >> Debian 8
> > > > > > > > > > >>
> > > > > > > > > > >> Each of these is approaching or past its end of life.
> > > Since
> > > > > > > Centos6
> > > > > > > > > was
> > > > > > > > > > >> the only operating system still needing Python 2.6 and
> > > > Python
> > > > > > 2.6
> > > > > > > is
> > > > > > > > > > long
> > > > > > > > > > >> past its end of life, Python 2.6 will no longer be
> > > > supported. In
> > > > > > > > > > addition,
> > > > > > > > > > >> Centos 7 support will be focused on newer versions
> such
> > as
> > > > 7.5
> > > > > > and
> > > > > > > > > > above.
> > > > > > > > > > >>
> > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > >>
> > > > > > > > > > >> Over the past year and a half, Impala's Ranger
> > > authorization
> > > > > > > > > > >> functionality has achieved parity and surpassed the
> > > existing
> > > > > > > Sentry
> > > > > > > > > > >> authorization functionality. Impala's Sentry support
> > > > requires
> > > > > > > > ongoing
> > > > > > > > > > >> effort to test and maintain, including maintenance on
> > > Sentry
> > > > > > > itself
> > > > > > > > to
> > > > > > > > > > >> address security updates. Given the diminished
> activity
> > in
> > > > the
> > > > > > > > Sentry
> > > > > > > > > > >> community, there is no timeline for addressing these
> > > > security
> > > > > > > > updates.
> > > > > > > > > > >> Unless there is a maintainer for Sentry support,
> Impala
> > > > plans to
> > > > > > > > focus
> > > > > > > > > > its
> > > > > > > > > > >> efforts on Ranger for its authorization system and
> drop
> > > > Sentry.
> > > > > > > > > > >>
> > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > >>
> > > > > > > > > > >> Impala-lzo provides code to allow Impala to read the
> LZO
> > > > > > > compressed
> > > > > > > > > > >> tables. LZO is GPL licensed, which is why this support
> > is
> > > > not
> > > > > > > > included
> > > > > > > > > > >> directly. The Impala-lzo code interacts with internal
> > > Impala
> > > > > > code
> > > > > > > > at a
> > > > > > > > > > >> level that is error prone and intricate. Given the low
> > > > adoption
> > > > > > of
> > > > > > > > LZO
> > > > > > > > > > and
> > > > > > > > > > >> the other compression options available, Impala plans
> to
> > > > remove
> > > > > > > > > > Impala-lzo
> > > > > > > > > > >> support along with the low level interface it used.
> > > > > > > > > > >>
> > > > > > > > > > >> Deprecations:
> > > > > > > > > > >>
> > > > > > > > > > >> In addition, we also plan to deprecate several
> existing
> > > > features
> > > > > > > so
> > > > > > > > > that
> > > > > > > > > > >> they can be removed in a future release. Here is a
> > summary
> > > > of
> > > > > > the
> > > > > > > > most
> > > > > > > > > > >> notable ones:
> > > > > > > > > > >>
> > > > > > > > > > >> -
> > > > > > > > > > >>
> > > > > > > > > > >> Impala will default to using on demand metadata
> > > > management
> > > > > > > (i.e.
> > > > > > > > > the
> > > > > > > > > > >> local catalog implementation) as described in this
> > > > > > > documentation:
> > > > > > > > > > >>
> > > > > > > > >
> > > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > >> The old metadata system is now deprecated and may
> be
> > > > removed
> > > > > > > in a
> > > > > > > > > > >> future release.
> > > > > > > > > > >> -
> > > > > > > > > > >>
> > > > > > > > > > >> Impala is deprecating the Beeswax client protocol
> > (i.e.
> > > > > > clients
> > > > > > > > > that
> > > > > > > > > > >> connect via beeswax_port) in favor of the
> HiveServer2
> > > > client
> > > > > > > > > > protocol.
> > > > > > > > > > >> -
> > > > > > > > > > >>
> > > > > > > > > > >> Impala is deprecating the old decimal_v2=false
> > behavior
> > > > from
> > > > > > > > Impala
> > > > > > > > > > >> 2.x. The decimal_v2 query option has defaulted to
> > true
> > > > since
> > > > > > > > Impala
> > > > > > > > > > 3.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> Impala is always open to new developers, and we
> welcome
> > > > feedback
> > > > > > > on
> > > > > > > > > > these
> > > > > > > > > > >> plans. Further discussion of other changes and
> > > deprecations
> > > > is
> > > > > > > > ongoing
> > > > > > > > > > at
> > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >>
> > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Quanlong Huang <hu...@gmail.com>.
Thank Jim! I planned to go through all the JIRAs target at 4.0:
https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
But due to some other urgent works and the recent holiday, my progress is
slow. Anyway, I think it's time to branch out. We've been waiting too long.
Thanks for creating the branch.
Regards,
Quanlong
On Fri, May 7, 2021 at 3:29 AM Jim Apple <ap...@jbapple.com> wrote:
> I've created a 4.0 branch (with all changes through the end of April) and
> started testing a release candidate. Everyone who wants to vote,
> https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release is a
> guide to both releasing and voting (at the bottom). I'll let you all know
> when I have something to vote on.
>
> Thanks!
> Jim
>
> On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <csringhofer@cloudera.com
> >
> wrote:
>
> > About IMPALA-9690 (AVX support):
> > My preferred solution would be to deprecate support for x64 without AVX2
> in
> > 4.0, but not start removing the related logic yet.
> > - We could even add a DCHECK + flag to crash by default if no AVX2 is
> > detected, and a message that points them to Impala mailing lists/related
> > Jira.
> > - After 4.0 is released people would be free to implement things that
> > assume AVX2 support on x64, for example compiling Impala/the whole
> > toolchain with -mavx2. I think that currently people are held back from
> > doing this - why put effort in doing+benchmarking a change if you can't
> > push it to master.
> > - If a user appears with a valid use case that needs x64 without AVX2
> then
> > we can revise our plans, but we shouldn't ignore low hanging fruits for
> the
> > sake of imaginary users :)
> >
> > -Csaba
> >
> > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> shant@superdupershant.com
> > >
> > wrote:
> >
> > > +1
> > >
> > > Regarding IMPALA-9690 if someone has the time to clean up the old
> > > conditional cpu logic it would be great, but if the ARM folks still
> > > need some of that logic it's not critical enough to block, just a very
> > > nice to have?
> > >
> > > -Shant
> > >
> > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> tim.g.armstrong@gmail.com>
> > > wrote:
> > > >
> > > > I agree that it would be good to get this done even if it's not
> > perfect,
> > > > there's so much good stuff.
> > > >
> > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> joemcdonnell@cloudera.com>
> > > > wrote:
> > > >
> > > > > I agree that we should wrap up Impala 4. A large amount of good
> work
> > > has
> > > > > gone in and it belongs in a release.
> > > > >
> > > > > We need to decide which breaking changes are truly blockers for an
> > > Impala 4
> > > > > release. My feeling is that we can't hold the release for
> > compatibility
> > > > > breaking changes unless someone signs up to do those changes.
> > Breaking
> > > > > compatibility is useful sometimes, but it can't be an indefinite
> hold
> > > on
> > > > > releases. I think a release discussion thread is a good way to
> > > stimulate
> > > > > this discussion.
> > > > >
> > > > > Apart from the two JIRAs you listed, one other change mentioned in
> > the
> > > > > original email is switching to use the new on-demand metadata by
> > > default.
> > > > >
> > > > > Thanks,
> > > > > Joe
> > > > >
> > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > huangquanlong@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Sure. If there are no objections, I'll raise a discussion thread
> > for
> > > the
> > > > > > 4.0 release. There are still some unresolved breaking changes,
> e.g.
> > > > > >
> > > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > > >
> > > > > > Thanks
> > > > > > Quanlong
> > > > > >
> > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <jb...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Thanks for bringing this up again, Quanlong! I would love to
> see
> > a
> > > 4.0
> > > > > > > release soon so that 4.1 releases and what not can be prepared.
> > > Already
> > > > > > the
> > > > > > > changlist from the 3.x line is going to be enormous and could
> be
> > > hard
> > > > > for
> > > > > > > users to digest.
> > > > > > >
> > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > huangquanlong@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Reviving this thread. I think it's time to create the 4.0.0
> > > branch
> > > > > and
> > > > > > > > prepare for the release now. Any breaking change is landing
> and
> > > we
> > > > > > should
> > > > > > > > wait?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Quanlong
> > > > > > > >
> > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > tarmstrong@cloudera.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Another thing that we're looking into is changing the
> encoded
> > > > > runtime
> > > > > > > > > profile representation to be more efficient - see
> > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. So that
> > > might
> > > > > be
> > > > > > a
> > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > >
> > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > > tarmstrong@cloudera.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > An addendum to this - we're also considering whether to
> > > increase
> > > > > > the
> > > > > > > > > > minimum CPU version so that we can simplify development
> and
> > > focus
> > > > > > > more
> > > > > > > > on
> > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > >
> > > > > > > > > > Running on commodity hardware is important and we want
> > > people to
> > > > > be
> > > > > > > > able
> > > > > > > > > > to develop on whatever machine they have access to. At
> most
> > > we're
> > > > > > > > likely
> > > > > > > > > > going to require AVX2 support, which has been supported
> by
> > > the
> > > > > vast
> > > > > > > > > > majority of CPUs for the better part of a decade.
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > > joemcdonnell@cloudera.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Impala is starting development on Impala 4.0. There are
> > many
> > > > > > > exciting
> > > > > > > > > >> projects underway including full support for Hive ACID
> > > tables,
> > > > > > > > improved
> > > > > > > > > >> multithreading support, and fault tolerance
> improvements,
> > > but
> > > > > > Impala
> > > > > > > > 4.0
> > > > > > > > > >> also serves as a time to alter behavior and platform
> > > support.
> > > > > Here
> > > > > > > is
> > > > > > > > > the
> > > > > > > > > >> current list of notable breaking changes that are
> planned
> > in
> > > > > > Impala
> > > > > > > > 4.0:
> > > > > > > > > >>
> > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > >>
> > > > > > > > > >> Maintaining support for older operating systems
> requires a
> > > > > > > > considerable
> > > > > > > > > >> amount of effort, especially with changing security
> > > > > requirements.
> > > > > > > Each
> > > > > > > > > >> extra operating system consumes valuable effort and can
> > > conflict
> > > > > > > with
> > > > > > > > > >> support for newer operating systems, so we have decided
> to
> > > > > remove
> > > > > > > > > support
> > > > > > > > > >> for some older operating systems. Specifically, we plan
> to
> > > drop
> > > > > > > > support
> > > > > > > > > for:
> > > > > > > > > >>
> > > > > > > > > >> -
> > > > > > > > > >>
> > > > > > > > > >> Centos 6
> > > > > > > > > >> -
> > > > > > > > > >>
> > > > > > > > > >> Ubuntu 14
> > > > > > > > > >> -
> > > > > > > > > >>
> > > > > > > > > >> Debian 8
> > > > > > > > > >>
> > > > > > > > > >> Each of these is approaching or past its end of life.
> > Since
> > > > > > Centos6
> > > > > > > > was
> > > > > > > > > >> the only operating system still needing Python 2.6 and
> > > Python
> > > > > 2.6
> > > > > > is
> > > > > > > > > long
> > > > > > > > > >> past its end of life, Python 2.6 will no longer be
> > > supported. In
> > > > > > > > > addition,
> > > > > > > > > >> Centos 7 support will be focused on newer versions such
> as
> > > 7.5
> > > > > and
> > > > > > > > > above.
> > > > > > > > > >>
> > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > >>
> > > > > > > > > >> Over the past year and a half, Impala's Ranger
> > authorization
> > > > > > > > > >> functionality has achieved parity and surpassed the
> > existing
> > > > > > Sentry
> > > > > > > > > >> authorization functionality. Impala's Sentry support
> > > requires
> > > > > > > ongoing
> > > > > > > > > >> effort to test and maintain, including maintenance on
> > Sentry
> > > > > > itself
> > > > > > > to
> > > > > > > > > >> address security updates. Given the diminished activity
> in
> > > the
> > > > > > > Sentry
> > > > > > > > > >> community, there is no timeline for addressing these
> > > security
> > > > > > > updates.
> > > > > > > > > >> Unless there is a maintainer for Sentry support, Impala
> > > plans to
> > > > > > > focus
> > > > > > > > > its
> > > > > > > > > >> efforts on Ranger for its authorization system and drop
> > > Sentry.
> > > > > > > > > >>
> > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > >>
> > > > > > > > > >> Impala-lzo provides code to allow Impala to read the LZO
> > > > > > compressed
> > > > > > > > > >> tables. LZO is GPL licensed, which is why this support
> is
> > > not
> > > > > > > included
> > > > > > > > > >> directly. The Impala-lzo code interacts with internal
> > Impala
> > > > > code
> > > > > > > at a
> > > > > > > > > >> level that is error prone and intricate. Given the low
> > > adoption
> > > > > of
> > > > > > > LZO
> > > > > > > > > and
> > > > > > > > > >> the other compression options available, Impala plans to
> > > remove
> > > > > > > > > Impala-lzo
> > > > > > > > > >> support along with the low level interface it used.
> > > > > > > > > >>
> > > > > > > > > >> Deprecations:
> > > > > > > > > >>
> > > > > > > > > >> In addition, we also plan to deprecate several existing
> > > features
> > > > > > so
> > > > > > > > that
> > > > > > > > > >> they can be removed in a future release. Here is a
> summary
> > > of
> > > > > the
> > > > > > > most
> > > > > > > > > >> notable ones:
> > > > > > > > > >>
> > > > > > > > > >> -
> > > > > > > > > >>
> > > > > > > > > >> Impala will default to using on demand metadata
> > > management
> > > > > > (i.e.
> > > > > > > > the
> > > > > > > > > >> local catalog implementation) as described in this
> > > > > > documentation:
> > > > > > > > > >>
> > > > > > > >
> > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > >> The old metadata system is now deprecated and may be
> > > removed
> > > > > > in a
> > > > > > > > > >> future release.
> > > > > > > > > >> -
> > > > > > > > > >>
> > > > > > > > > >> Impala is deprecating the Beeswax client protocol
> (i.e.
> > > > > clients
> > > > > > > > that
> > > > > > > > > >> connect via beeswax_port) in favor of the HiveServer2
> > > client
> > > > > > > > > protocol.
> > > > > > > > > >> -
> > > > > > > > > >>
> > > > > > > > > >> Impala is deprecating the old decimal_v2=false
> behavior
> > > from
> > > > > > > Impala
> > > > > > > > > >> 2.x. The decimal_v2 query option has defaulted to
> true
> > > since
> > > > > > > Impala
> > > > > > > > > 3.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> Impala is always open to new developers, and we welcome
> > > feedback
> > > > > > on
> > > > > > > > > these
> > > > > > > > > >> plans. Further discussion of other changes and
> > deprecations
> > > is
> > > > > > > ongoing
> > > > > > > > > at
> > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >>
> > > > > > > > > >> Joe McDonnell
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Jim Apple <ap...@jbapple.com>.
I've created a 4.0 branch (with all changes through the end of April) and
started testing a release candidate. Everyone who wants to vote,
https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release is a
guide to both releasing and voting (at the bottom). I'll let you all know
when I have something to vote on.
Thanks!
Jim
On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <cs...@cloudera.com>
wrote:
> About IMPALA-9690 (AVX support):
> My preferred solution would be to deprecate support for x64 without AVX2 in
> 4.0, but not start removing the related logic yet.
> - We could even add a DCHECK + flag to crash by default if no AVX2 is
> detected, and a message that points them to Impala mailing lists/related
> Jira.
> - After 4.0 is released people would be free to implement things that
> assume AVX2 support on x64, for example compiling Impala/the whole
> toolchain with -mavx2. I think that currently people are held back from
> doing this - why put effort in doing+benchmarking a change if you can't
> push it to master.
> - If a user appears with a valid use case that needs x64 without AVX2 then
> we can revise our plans, but we shouldn't ignore low hanging fruits for the
> sake of imaginary users :)
>
> -Csaba
>
> On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <shant@superdupershant.com
> >
> wrote:
>
> > +1
> >
> > Regarding IMPALA-9690 if someone has the time to clean up the old
> > conditional cpu logic it would be great, but if the ARM folks still
> > need some of that logic it's not critical enough to block, just a very
> > nice to have?
> >
> > -Shant
> >
> > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <ti...@gmail.com>
> > wrote:
> > >
> > > I agree that it would be good to get this done even if it's not
> perfect,
> > > there's so much good stuff.
> > >
> > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <jo...@cloudera.com>
> > > wrote:
> > >
> > > > I agree that we should wrap up Impala 4. A large amount of good work
> > has
> > > > gone in and it belongs in a release.
> > > >
> > > > We need to decide which breaking changes are truly blockers for an
> > Impala 4
> > > > release. My feeling is that we can't hold the release for
> compatibility
> > > > breaking changes unless someone signs up to do those changes.
> Breaking
> > > > compatibility is useful sometimes, but it can't be an indefinite hold
> > on
> > > > releases. I think a release discussion thread is a good way to
> > stimulate
> > > > this discussion.
> > > >
> > > > Apart from the two JIRAs you listed, one other change mentioned in
> the
> > > > original email is switching to use the new on-demand metadata by
> > default.
> > > >
> > > > Thanks,
> > > > Joe
> > > >
> > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> huangquanlong@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Sure. If there are no objections, I'll raise a discussion thread
> for
> > the
> > > > > 4.0 release. There are still some unresolved breaking changes, e.g.
> > > > >
> > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > >
> > > > > Thanks
> > > > > Quanlong
> > > > >
> > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <jb...@apache.org>
> > wrote:
> > > > >
> > > > > > Thanks for bringing this up again, Quanlong! I would love to see
> a
> > 4.0
> > > > > > release soon so that 4.1 releases and what not can be prepared.
> > Already
> > > > > the
> > > > > > changlist from the 3.x line is going to be enormous and could be
> > hard
> > > > for
> > > > > > users to digest.
> > > > > >
> > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > huangquanlong@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Reviving this thread. I think it's time to create the 4.0.0
> > branch
> > > > and
> > > > > > > prepare for the release now. Any breaking change is landing and
> > we
> > > > > should
> > > > > > > wait?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Quanlong
> > > > > > >
> > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > tarmstrong@cloudera.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Another thing that we're looking into is changing the encoded
> > > > runtime
> > > > > > > > profile representation to be more efficient - see
> > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. So that
> > might
> > > > be
> > > > > a
> > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > >
> > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > tarmstrong@cloudera.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > An addendum to this - we're also considering whether to
> > increase
> > > > > the
> > > > > > > > > minimum CPU version so that we can simplify development and
> > focus
> > > > > > more
> > > > > > > on
> > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > >
> > > > > > > > > Running on commodity hardware is important and we want
> > people to
> > > > be
> > > > > > > able
> > > > > > > > > to develop on whatever machine they have access to. At most
> > we're
> > > > > > > likely
> > > > > > > > > going to require AVX2 support, which has been supported by
> > the
> > > > vast
> > > > > > > > > majority of CPUs for the better part of a decade.
> > > > > > > > >
> > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > joemcdonnell@cloudera.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Impala is starting development on Impala 4.0. There are
> many
> > > > > > exciting
> > > > > > > > >> projects underway including full support for Hive ACID
> > tables,
> > > > > > > improved
> > > > > > > > >> multithreading support, and fault tolerance improvements,
> > but
> > > > > Impala
> > > > > > > 4.0
> > > > > > > > >> also serves as a time to alter behavior and platform
> > support.
> > > > Here
> > > > > > is
> > > > > > > > the
> > > > > > > > >> current list of notable breaking changes that are planned
> in
> > > > > Impala
> > > > > > > 4.0:
> > > > > > > > >>
> > > > > > > > >> Remove support for older operating systems:
> > > > > > > > >>
> > > > > > > > >> Maintaining support for older operating systems requires a
> > > > > > > considerable
> > > > > > > > >> amount of effort, especially with changing security
> > > > requirements.
> > > > > > Each
> > > > > > > > >> extra operating system consumes valuable effort and can
> > conflict
> > > > > > with
> > > > > > > > >> support for newer operating systems, so we have decided to
> > > > remove
> > > > > > > > support
> > > > > > > > >> for some older operating systems. Specifically, we plan to
> > drop
> > > > > > > support
> > > > > > > > for:
> > > > > > > > >>
> > > > > > > > >> -
> > > > > > > > >>
> > > > > > > > >> Centos 6
> > > > > > > > >> -
> > > > > > > > >>
> > > > > > > > >> Ubuntu 14
> > > > > > > > >> -
> > > > > > > > >>
> > > > > > > > >> Debian 8
> > > > > > > > >>
> > > > > > > > >> Each of these is approaching or past its end of life.
> Since
> > > > > Centos6
> > > > > > > was
> > > > > > > > >> the only operating system still needing Python 2.6 and
> > Python
> > > > 2.6
> > > > > is
> > > > > > > > long
> > > > > > > > >> past its end of life, Python 2.6 will no longer be
> > supported. In
> > > > > > > > addition,
> > > > > > > > >> Centos 7 support will be focused on newer versions such as
> > 7.5
> > > > and
> > > > > > > > above.
> > > > > > > > >>
> > > > > > > > >> Remove support for Sentry:
> > > > > > > > >>
> > > > > > > > >> Over the past year and a half, Impala's Ranger
> authorization
> > > > > > > > >> functionality has achieved parity and surpassed the
> existing
> > > > > Sentry
> > > > > > > > >> authorization functionality. Impala's Sentry support
> > requires
> > > > > > ongoing
> > > > > > > > >> effort to test and maintain, including maintenance on
> Sentry
> > > > > itself
> > > > > > to
> > > > > > > > >> address security updates. Given the diminished activity in
> > the
> > > > > > Sentry
> > > > > > > > >> community, there is no timeline for addressing these
> > security
> > > > > > updates.
> > > > > > > > >> Unless there is a maintainer for Sentry support, Impala
> > plans to
> > > > > > focus
> > > > > > > > its
> > > > > > > > >> efforts on Ranger for its authorization system and drop
> > Sentry.
> > > > > > > > >>
> > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > >>
> > > > > > > > >> Impala-lzo provides code to allow Impala to read the LZO
> > > > > compressed
> > > > > > > > >> tables. LZO is GPL licensed, which is why this support is
> > not
> > > > > > included
> > > > > > > > >> directly. The Impala-lzo code interacts with internal
> Impala
> > > > code
> > > > > > at a
> > > > > > > > >> level that is error prone and intricate. Given the low
> > adoption
> > > > of
> > > > > > LZO
> > > > > > > > and
> > > > > > > > >> the other compression options available, Impala plans to
> > remove
> > > > > > > > Impala-lzo
> > > > > > > > >> support along with the low level interface it used.
> > > > > > > > >>
> > > > > > > > >> Deprecations:
> > > > > > > > >>
> > > > > > > > >> In addition, we also plan to deprecate several existing
> > features
> > > > > so
> > > > > > > that
> > > > > > > > >> they can be removed in a future release. Here is a summary
> > of
> > > > the
> > > > > > most
> > > > > > > > >> notable ones:
> > > > > > > > >>
> > > > > > > > >> -
> > > > > > > > >>
> > > > > > > > >> Impala will default to using on demand metadata
> > management
> > > > > (i.e.
> > > > > > > the
> > > > > > > > >> local catalog implementation) as described in this
> > > > > documentation:
> > > > > > > > >>
> > > > > > >
> > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > >> The old metadata system is now deprecated and may be
> > removed
> > > > > in a
> > > > > > > > >> future release.
> > > > > > > > >> -
> > > > > > > > >>
> > > > > > > > >> Impala is deprecating the Beeswax client protocol (i.e.
> > > > clients
> > > > > > > that
> > > > > > > > >> connect via beeswax_port) in favor of the HiveServer2
> > client
> > > > > > > > protocol.
> > > > > > > > >> -
> > > > > > > > >>
> > > > > > > > >> Impala is deprecating the old decimal_v2=false behavior
> > from
> > > > > > Impala
> > > > > > > > >> 2.x. The decimal_v2 query option has defaulted to true
> > since
> > > > > > Impala
> > > > > > > > 3.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Impala is always open to new developers, and we welcome
> > feedback
> > > > > on
> > > > > > > > these
> > > > > > > > >> plans. Further discussion of other changes and
> deprecations
> > is
> > > > > > ongoing
> > > > > > > > at
> > > > > > > > >> dev@impala.apache.org.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >>
> > > > > > > > >> Joe McDonnell
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Csaba Ringhofer <cs...@cloudera.com>.
About IMPALA-9690 (AVX support):
My preferred solution would be to deprecate support for x64 without AVX2 in
4.0, but not start removing the related logic yet.
- We could even add a DCHECK + flag to crash by default if no AVX2 is
detected, and a message that points them to Impala mailing lists/related
Jira.
- After 4.0 is released people would be free to implement things that
assume AVX2 support on x64, for example compiling Impala/the whole
toolchain with -mavx2. I think that currently people are held back from
doing this - why put effort in doing+benchmarking a change if you can't
push it to master.
- If a user appears with a valid use case that needs x64 without AVX2 then
we can revise our plans, but we shouldn't ignore low hanging fruits for the
sake of imaginary users :)
-Csaba
On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <sh...@superdupershant.com>
wrote:
> +1
>
> Regarding IMPALA-9690 if someone has the time to clean up the old
> conditional cpu logic it would be great, but if the ARM folks still
> need some of that logic it's not critical enough to block, just a very
> nice to have?
>
> -Shant
>
> On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <ti...@gmail.com>
> wrote:
> >
> > I agree that it would be good to get this done even if it's not perfect,
> > there's so much good stuff.
> >
> > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <jo...@cloudera.com>
> > wrote:
> >
> > > I agree that we should wrap up Impala 4. A large amount of good work
> has
> > > gone in and it belongs in a release.
> > >
> > > We need to decide which breaking changes are truly blockers for an
> Impala 4
> > > release. My feeling is that we can't hold the release for compatibility
> > > breaking changes unless someone signs up to do those changes. Breaking
> > > compatibility is useful sometimes, but it can't be an indefinite hold
> on
> > > releases. I think a release discussion thread is a good way to
> stimulate
> > > this discussion.
> > >
> > > Apart from the two JIRAs you listed, one other change mentioned in the
> > > original email is switching to use the new on-demand metadata by
> default.
> > >
> > > Thanks,
> > > Joe
> > >
> > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <huangquanlong@gmail.com
> >
> > > wrote:
> > >
> > > > Sure. If there are no objections, I'll raise a discussion thread for
> the
> > > > 4.0 release. There are still some unresolved breaking changes, e.g.
> > > >
> > > > - IMPALA-2210: Make Parquet the default file format
> > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > >
> > > > Thanks
> > > > Quanlong
> > > >
> > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <jb...@apache.org>
> wrote:
> > > >
> > > > > Thanks for bringing this up again, Quanlong! I would love to see a
> 4.0
> > > > > release soon so that 4.1 releases and what not can be prepared.
> Already
> > > > the
> > > > > changlist from the 3.x line is going to be enormous and could be
> hard
> > > for
> > > > > users to digest.
> > > > >
> > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > huangquanlong@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Reviving this thread. I think it's time to create the 4.0.0
> branch
> > > and
> > > > > > prepare for the release now. Any breaking change is landing and
> we
> > > > should
> > > > > > wait?
> > > > > >
> > > > > > Thanks,
> > > > > > Quanlong
> > > > > >
> > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > tarmstrong@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Another thing that we're looking into is changing the encoded
> > > runtime
> > > > > > > profile representation to be more efficient - see
> > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. So that
> might
> > > be
> > > > a
> > > > > > > default we will try to change in Impala 4.0.
> > > > > > >
> > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > tarmstrong@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > An addendum to this - we're also considering whether to
> increase
> > > > the
> > > > > > > > minimum CPU version so that we can simplify development and
> focus
> > > > > more
> > > > > > on
> > > > > > > > optimising for the most common CPUs - see
> > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > >
> > > > > > > > Running on commodity hardware is important and we want
> people to
> > > be
> > > > > > able
> > > > > > > > to develop on whatever machine they have access to. At most
> we're
> > > > > > likely
> > > > > > > > going to require AVX2 support, which has been supported by
> the
> > > vast
> > > > > > > > majority of CPUs for the better part of a decade.
> > > > > > > >
> > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > joemcdonnell@cloudera.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Impala is starting development on Impala 4.0. There are many
> > > > > exciting
> > > > > > > >> projects underway including full support for Hive ACID
> tables,
> > > > > > improved
> > > > > > > >> multithreading support, and fault tolerance improvements,
> but
> > > > Impala
> > > > > > 4.0
> > > > > > > >> also serves as a time to alter behavior and platform
> support.
> > > Here
> > > > > is
> > > > > > > the
> > > > > > > >> current list of notable breaking changes that are planned in
> > > > Impala
> > > > > > 4.0:
> > > > > > > >>
> > > > > > > >> Remove support for older operating systems:
> > > > > > > >>
> > > > > > > >> Maintaining support for older operating systems requires a
> > > > > > considerable
> > > > > > > >> amount of effort, especially with changing security
> > > requirements.
> > > > > Each
> > > > > > > >> extra operating system consumes valuable effort and can
> conflict
> > > > > with
> > > > > > > >> support for newer operating systems, so we have decided to
> > > remove
> > > > > > > support
> > > > > > > >> for some older operating systems. Specifically, we plan to
> drop
> > > > > > support
> > > > > > > for:
> > > > > > > >>
> > > > > > > >> -
> > > > > > > >>
> > > > > > > >> Centos 6
> > > > > > > >> -
> > > > > > > >>
> > > > > > > >> Ubuntu 14
> > > > > > > >> -
> > > > > > > >>
> > > > > > > >> Debian 8
> > > > > > > >>
> > > > > > > >> Each of these is approaching or past its end of life. Since
> > > > Centos6
> > > > > > was
> > > > > > > >> the only operating system still needing Python 2.6 and
> Python
> > > 2.6
> > > > is
> > > > > > > long
> > > > > > > >> past its end of life, Python 2.6 will no longer be
> supported. In
> > > > > > > addition,
> > > > > > > >> Centos 7 support will be focused on newer versions such as
> 7.5
> > > and
> > > > > > > above.
> > > > > > > >>
> > > > > > > >> Remove support for Sentry:
> > > > > > > >>
> > > > > > > >> Over the past year and a half, Impala's Ranger authorization
> > > > > > > >> functionality has achieved parity and surpassed the existing
> > > > Sentry
> > > > > > > >> authorization functionality. Impala's Sentry support
> requires
> > > > > ongoing
> > > > > > > >> effort to test and maintain, including maintenance on Sentry
> > > > itself
> > > > > to
> > > > > > > >> address security updates. Given the diminished activity in
> the
> > > > > Sentry
> > > > > > > >> community, there is no timeline for addressing these
> security
> > > > > updates.
> > > > > > > >> Unless there is a maintainer for Sentry support, Impala
> plans to
> > > > > focus
> > > > > > > its
> > > > > > > >> efforts on Ranger for its authorization system and drop
> Sentry.
> > > > > > > >>
> > > > > > > >> Remove support for Impala-lzo:
> > > > > > > >>
> > > > > > > >> Impala-lzo provides code to allow Impala to read the LZO
> > > > compressed
> > > > > > > >> tables. LZO is GPL licensed, which is why this support is
> not
> > > > > included
> > > > > > > >> directly. The Impala-lzo code interacts with internal Impala
> > > code
> > > > > at a
> > > > > > > >> level that is error prone and intricate. Given the low
> adoption
> > > of
> > > > > LZO
> > > > > > > and
> > > > > > > >> the other compression options available, Impala plans to
> remove
> > > > > > > Impala-lzo
> > > > > > > >> support along with the low level interface it used.
> > > > > > > >>
> > > > > > > >> Deprecations:
> > > > > > > >>
> > > > > > > >> In addition, we also plan to deprecate several existing
> features
> > > > so
> > > > > > that
> > > > > > > >> they can be removed in a future release. Here is a summary
> of
> > > the
> > > > > most
> > > > > > > >> notable ones:
> > > > > > > >>
> > > > > > > >> -
> > > > > > > >>
> > > > > > > >> Impala will default to using on demand metadata
> management
> > > > (i.e.
> > > > > > the
> > > > > > > >> local catalog implementation) as described in this
> > > > documentation:
> > > > > > > >>
> > > > > >
> http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > >> The old metadata system is now deprecated and may be
> removed
> > > > in a
> > > > > > > >> future release.
> > > > > > > >> -
> > > > > > > >>
> > > > > > > >> Impala is deprecating the Beeswax client protocol (i.e.
> > > clients
> > > > > > that
> > > > > > > >> connect via beeswax_port) in favor of the HiveServer2
> client
> > > > > > > protocol.
> > > > > > > >> -
> > > > > > > >>
> > > > > > > >> Impala is deprecating the old decimal_v2=false behavior
> from
> > > > > Impala
> > > > > > > >> 2.x. The decimal_v2 query option has defaulted to true
> since
> > > > > Impala
> > > > > > > 3.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Impala is always open to new developers, and we welcome
> feedback
> > > > on
> > > > > > > these
> > > > > > > >> plans. Further discussion of other changes and deprecations
> is
> > > > > ongoing
> > > > > > > at
> > > > > > > >> dev@impala.apache.org.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >>
> > > > > > > >> Joe McDonnell
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: Impala 4 Breaking Changes
Posted by Shant Hovsepian <sh...@superdupershant.com>.
+1
Regarding IMPALA-9690 if someone has the time to clean up the old
conditional cpu logic it would be great, but if the ARM folks still
need some of that logic it's not critical enough to block, just a very
nice to have?
-Shant
On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <ti...@gmail.com> wrote:
>
> I agree that it would be good to get this done even if it's not perfect,
> there's so much good stuff.
>
> On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <jo...@cloudera.com>
> wrote:
>
> > I agree that we should wrap up Impala 4. A large amount of good work has
> > gone in and it belongs in a release.
> >
> > We need to decide which breaking changes are truly blockers for an Impala 4
> > release. My feeling is that we can't hold the release for compatibility
> > breaking changes unless someone signs up to do those changes. Breaking
> > compatibility is useful sometimes, but it can't be an indefinite hold on
> > releases. I think a release discussion thread is a good way to stimulate
> > this discussion.
> >
> > Apart from the two JIRAs you listed, one other change mentioned in the
> > original email is switching to use the new on-demand metadata by default.
> >
> > Thanks,
> > Joe
> >
> > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <hu...@gmail.com>
> > wrote:
> >
> > > Sure. If there are no objections, I'll raise a discussion thread for the
> > > 4.0 release. There are still some unresolved breaking changes, e.g.
> > >
> > > - IMPALA-2210: Make Parquet the default file format
> > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > >
> > > Thanks
> > > Quanlong
> > >
> > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <jb...@apache.org> wrote:
> > >
> > > > Thanks for bringing this up again, Quanlong! I would love to see a 4.0
> > > > release soon so that 4.1 releases and what not can be prepared. Already
> > > the
> > > > changlist from the 3.x line is going to be enormous and could be hard
> > for
> > > > users to digest.
> > > >
> > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > huangquanlong@gmail.com>
> > > > wrote:
> > > >
> > > > > Reviving this thread. I think it's time to create the 4.0.0 branch
> > and
> > > > > prepare for the release now. Any breaking change is landing and we
> > > should
> > > > > wait?
> > > > >
> > > > > Thanks,
> > > > > Quanlong
> > > > >
> > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > tarmstrong@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Another thing that we're looking into is changing the encoded
> > runtime
> > > > > > profile representation to be more efficient - see
> > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. So that might
> > be
> > > a
> > > > > > default we will try to change in Impala 4.0.
> > > > > >
> > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > tarmstrong@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > An addendum to this - we're also considering whether to increase
> > > the
> > > > > > > minimum CPU version so that we can simplify development and focus
> > > > more
> > > > > on
> > > > > > > optimising for the most common CPUs - see
> > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > >
> > > > > > > Running on commodity hardware is important and we want people to
> > be
> > > > > able
> > > > > > > to develop on whatever machine they have access to. At most we're
> > > > > likely
> > > > > > > going to require AVX2 support, which has been supported by the
> > vast
> > > > > > > majority of CPUs for the better part of a decade.
> > > > > > >
> > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > joemcdonnell@cloudera.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Impala is starting development on Impala 4.0. There are many
> > > > exciting
> > > > > > >> projects underway including full support for Hive ACID tables,
> > > > > improved
> > > > > > >> multithreading support, and fault tolerance improvements, but
> > > Impala
> > > > > 4.0
> > > > > > >> also serves as a time to alter behavior and platform support.
> > Here
> > > > is
> > > > > > the
> > > > > > >> current list of notable breaking changes that are planned in
> > > Impala
> > > > > 4.0:
> > > > > > >>
> > > > > > >> Remove support for older operating systems:
> > > > > > >>
> > > > > > >> Maintaining support for older operating systems requires a
> > > > > considerable
> > > > > > >> amount of effort, especially with changing security
> > requirements.
> > > > Each
> > > > > > >> extra operating system consumes valuable effort and can conflict
> > > > with
> > > > > > >> support for newer operating systems, so we have decided to
> > remove
> > > > > > support
> > > > > > >> for some older operating systems. Specifically, we plan to drop
> > > > > support
> > > > > > for:
> > > > > > >>
> > > > > > >> -
> > > > > > >>
> > > > > > >> Centos 6
> > > > > > >> -
> > > > > > >>
> > > > > > >> Ubuntu 14
> > > > > > >> -
> > > > > > >>
> > > > > > >> Debian 8
> > > > > > >>
> > > > > > >> Each of these is approaching or past its end of life. Since
> > > Centos6
> > > > > was
> > > > > > >> the only operating system still needing Python 2.6 and Python
> > 2.6
> > > is
> > > > > > long
> > > > > > >> past its end of life, Python 2.6 will no longer be supported. In
> > > > > > addition,
> > > > > > >> Centos 7 support will be focused on newer versions such as 7.5
> > and
> > > > > > above.
> > > > > > >>
> > > > > > >> Remove support for Sentry:
> > > > > > >>
> > > > > > >> Over the past year and a half, Impala's Ranger authorization
> > > > > > >> functionality has achieved parity and surpassed the existing
> > > Sentry
> > > > > > >> authorization functionality. Impala's Sentry support requires
> > > > ongoing
> > > > > > >> effort to test and maintain, including maintenance on Sentry
> > > itself
> > > > to
> > > > > > >> address security updates. Given the diminished activity in the
> > > > Sentry
> > > > > > >> community, there is no timeline for addressing these security
> > > > updates.
> > > > > > >> Unless there is a maintainer for Sentry support, Impala plans to
> > > > focus
> > > > > > its
> > > > > > >> efforts on Ranger for its authorization system and drop Sentry.
> > > > > > >>
> > > > > > >> Remove support for Impala-lzo:
> > > > > > >>
> > > > > > >> Impala-lzo provides code to allow Impala to read the LZO
> > > compressed
> > > > > > >> tables. LZO is GPL licensed, which is why this support is not
> > > > included
> > > > > > >> directly. The Impala-lzo code interacts with internal Impala
> > code
> > > > at a
> > > > > > >> level that is error prone and intricate. Given the low adoption
> > of
> > > > LZO
> > > > > > and
> > > > > > >> the other compression options available, Impala plans to remove
> > > > > > Impala-lzo
> > > > > > >> support along with the low level interface it used.
> > > > > > >>
> > > > > > >> Deprecations:
> > > > > > >>
> > > > > > >> In addition, we also plan to deprecate several existing features
> > > so
> > > > > that
> > > > > > >> they can be removed in a future release. Here is a summary of
> > the
> > > > most
> > > > > > >> notable ones:
> > > > > > >>
> > > > > > >> -
> > > > > > >>
> > > > > > >> Impala will default to using on demand metadata management
> > > (i.e.
> > > > > the
> > > > > > >> local catalog implementation) as described in this
> > > documentation:
> > > > > > >>
> > > > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > >> The old metadata system is now deprecated and may be removed
> > > in a
> > > > > > >> future release.
> > > > > > >> -
> > > > > > >>
> > > > > > >> Impala is deprecating the Beeswax client protocol (i.e.
> > clients
> > > > > that
> > > > > > >> connect via beeswax_port) in favor of the HiveServer2 client
> > > > > > protocol.
> > > > > > >> -
> > > > > > >>
> > > > > > >> Impala is deprecating the old decimal_v2=false behavior from
> > > > Impala
> > > > > > >> 2.x. The decimal_v2 query option has defaulted to true since
> > > > Impala
> > > > > > 3.
> > > > > > >>
> > > > > > >>
> > > > > > >> Impala is always open to new developers, and we welcome feedback
> > > on
> > > > > > these
> > > > > > >> plans. Further discussion of other changes and deprecations is
> > > > ongoing
> > > > > > at
> > > > > > >> dev@impala.apache.org.
> > > > > > >>
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >>
> > > > > > >> Joe McDonnell
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
Re: Impala 4 Breaking Changes
Posted by Tim Armstrong <ti...@gmail.com>.
I agree that it would be good to get this done even if it's not perfect,
there's so much good stuff.
On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <jo...@cloudera.com>
wrote:
> I agree that we should wrap up Impala 4. A large amount of good work has
> gone in and it belongs in a release.
>
> We need to decide which breaking changes are truly blockers for an Impala 4
> release. My feeling is that we can't hold the release for compatibility
> breaking changes unless someone signs up to do those changes. Breaking
> compatibility is useful sometimes, but it can't be an indefinite hold on
> releases. I think a release discussion thread is a good way to stimulate
> this discussion.
>
> Apart from the two JIRAs you listed, one other change mentioned in the
> original email is switching to use the new on-demand metadata by default.
>
> Thanks,
> Joe
>
> On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <hu...@gmail.com>
> wrote:
>
> > Sure. If there are no objections, I'll raise a discussion thread for the
> > 4.0 release. There are still some unresolved breaking changes, e.g.
> >
> > - IMPALA-2210: Make Parquet the default file format
> > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> >
> > Thanks
> > Quanlong
> >
> > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <jb...@apache.org> wrote:
> >
> > > Thanks for bringing this up again, Quanlong! I would love to see a 4.0
> > > release soon so that 4.1 releases and what not can be prepared. Already
> > the
> > > changlist from the 3.x line is going to be enormous and could be hard
> for
> > > users to digest.
> > >
> > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> huangquanlong@gmail.com>
> > > wrote:
> > >
> > > > Reviving this thread. I think it's time to create the 4.0.0 branch
> and
> > > > prepare for the release now. Any breaking change is landing and we
> > should
> > > > wait?
> > > >
> > > > Thanks,
> > > > Quanlong
> > > >
> > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> tarmstrong@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Another thing that we're looking into is changing the encoded
> runtime
> > > > > profile representation to be more efficient - see
> > > > > https://issues.apache.org/jira/browse/IMPALA-9378. So that might
> be
> > a
> > > > > default we will try to change in Impala 4.0.
> > > > >
> > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > tarmstrong@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > An addendum to this - we're also considering whether to increase
> > the
> > > > > > minimum CPU version so that we can simplify development and focus
> > > more
> > > > on
> > > > > > optimising for the most common CPUs - see
> > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > >
> > > > > > Running on commodity hardware is important and we want people to
> be
> > > > able
> > > > > > to develop on whatever machine they have access to. At most we're
> > > > likely
> > > > > > going to require AVX2 support, which has been supported by the
> vast
> > > > > > majority of CPUs for the better part of a decade.
> > > > > >
> > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > joemcdonnell@cloudera.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Impala is starting development on Impala 4.0. There are many
> > > exciting
> > > > > >> projects underway including full support for Hive ACID tables,
> > > > improved
> > > > > >> multithreading support, and fault tolerance improvements, but
> > Impala
> > > > 4.0
> > > > > >> also serves as a time to alter behavior and platform support.
> Here
> > > is
> > > > > the
> > > > > >> current list of notable breaking changes that are planned in
> > Impala
> > > > 4.0:
> > > > > >>
> > > > > >> Remove support for older operating systems:
> > > > > >>
> > > > > >> Maintaining support for older operating systems requires a
> > > > considerable
> > > > > >> amount of effort, especially with changing security
> requirements.
> > > Each
> > > > > >> extra operating system consumes valuable effort and can conflict
> > > with
> > > > > >> support for newer operating systems, so we have decided to
> remove
> > > > > support
> > > > > >> for some older operating systems. Specifically, we plan to drop
> > > > support
> > > > > for:
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Centos 6
> > > > > >> -
> > > > > >>
> > > > > >> Ubuntu 14
> > > > > >> -
> > > > > >>
> > > > > >> Debian 8
> > > > > >>
> > > > > >> Each of these is approaching or past its end of life. Since
> > Centos6
> > > > was
> > > > > >> the only operating system still needing Python 2.6 and Python
> 2.6
> > is
> > > > > long
> > > > > >> past its end of life, Python 2.6 will no longer be supported. In
> > > > > addition,
> > > > > >> Centos 7 support will be focused on newer versions such as 7.5
> and
> > > > > above.
> > > > > >>
> > > > > >> Remove support for Sentry:
> > > > > >>
> > > > > >> Over the past year and a half, Impala's Ranger authorization
> > > > > >> functionality has achieved parity and surpassed the existing
> > Sentry
> > > > > >> authorization functionality. Impala's Sentry support requires
> > > ongoing
> > > > > >> effort to test and maintain, including maintenance on Sentry
> > itself
> > > to
> > > > > >> address security updates. Given the diminished activity in the
> > > Sentry
> > > > > >> community, there is no timeline for addressing these security
> > > updates.
> > > > > >> Unless there is a maintainer for Sentry support, Impala plans to
> > > focus
> > > > > its
> > > > > >> efforts on Ranger for its authorization system and drop Sentry.
> > > > > >>
> > > > > >> Remove support for Impala-lzo:
> > > > > >>
> > > > > >> Impala-lzo provides code to allow Impala to read the LZO
> > compressed
> > > > > >> tables. LZO is GPL licensed, which is why this support is not
> > > included
> > > > > >> directly. The Impala-lzo code interacts with internal Impala
> code
> > > at a
> > > > > >> level that is error prone and intricate. Given the low adoption
> of
> > > LZO
> > > > > and
> > > > > >> the other compression options available, Impala plans to remove
> > > > > Impala-lzo
> > > > > >> support along with the low level interface it used.
> > > > > >>
> > > > > >> Deprecations:
> > > > > >>
> > > > > >> In addition, we also plan to deprecate several existing features
> > so
> > > > that
> > > > > >> they can be removed in a future release. Here is a summary of
> the
> > > most
> > > > > >> notable ones:
> > > > > >>
> > > > > >> -
> > > > > >>
> > > > > >> Impala will default to using on demand metadata management
> > (i.e.
> > > > the
> > > > > >> local catalog implementation) as described in this
> > documentation:
> > > > > >>
> > > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > >> The old metadata system is now deprecated and may be removed
> > in a
> > > > > >> future release.
> > > > > >> -
> > > > > >>
> > > > > >> Impala is deprecating the Beeswax client protocol (i.e.
> clients
> > > > that
> > > > > >> connect via beeswax_port) in favor of the HiveServer2 client
> > > > > protocol.
> > > > > >> -
> > > > > >>
> > > > > >> Impala is deprecating the old decimal_v2=false behavior from
> > > Impala
> > > > > >> 2.x. The decimal_v2 query option has defaulted to true since
> > > Impala
> > > > > 3.
> > > > > >>
> > > > > >>
> > > > > >> Impala is always open to new developers, and we welcome feedback
> > on
> > > > > these
> > > > > >> plans. Further discussion of other changes and deprecations is
> > > ongoing
> > > > > at
> > > > > >> dev@impala.apache.org.
> > > > > >>
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Joe McDonnell
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Joe McDonnell <jo...@cloudera.com>.
I agree that we should wrap up Impala 4. A large amount of good work has
gone in and it belongs in a release.
We need to decide which breaking changes are truly blockers for an Impala 4
release. My feeling is that we can't hold the release for compatibility
breaking changes unless someone signs up to do those changes. Breaking
compatibility is useful sometimes, but it can't be an indefinite hold on
releases. I think a release discussion thread is a good way to stimulate
this discussion.
Apart from the two JIRAs you listed, one other change mentioned in the
original email is switching to use the new on-demand metadata by default.
Thanks,
Joe
On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <hu...@gmail.com>
wrote:
> Sure. If there are no objections, I'll raise a discussion thread for the
> 4.0 release. There are still some unresolved breaking changes, e.g.
>
> - IMPALA-2210: Make Parquet the default file format
> - IMPALA-9690: Bump minimum x86-64 CPU requirements
>
> Thanks
> Quanlong
>
> On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <jb...@apache.org> wrote:
>
> > Thanks for bringing this up again, Quanlong! I would love to see a 4.0
> > release soon so that 4.1 releases and what not can be prepared. Already
> the
> > changlist from the 3.x line is going to be enormous and could be hard for
> > users to digest.
> >
> > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <hu...@gmail.com>
> > wrote:
> >
> > > Reviving this thread. I think it's time to create the 4.0.0 branch and
> > > prepare for the release now. Any breaking change is landing and we
> should
> > > wait?
> > >
> > > Thanks,
> > > Quanlong
> > >
> > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <tarmstrong@cloudera.com
> >
> > > wrote:
> > >
> > > > Another thing that we're looking into is changing the encoded runtime
> > > > profile representation to be more efficient - see
> > > > https://issues.apache.org/jira/browse/IMPALA-9378. So that might be
> a
> > > > default we will try to change in Impala 4.0.
> > > >
> > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > tarmstrong@cloudera.com>
> > > > wrote:
> > > >
> > > > > An addendum to this - we're also considering whether to increase
> the
> > > > > minimum CPU version so that we can simplify development and focus
> > more
> > > on
> > > > > optimising for the most common CPUs - see
> > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > >
> > > > > Running on commodity hardware is important and we want people to be
> > > able
> > > > > to develop on whatever machine they have access to. At most we're
> > > likely
> > > > > going to require AVX2 support, which has been supported by the vast
> > > > > majority of CPUs for the better part of a decade.
> > > > >
> > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > joemcdonnell@cloudera.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Impala is starting development on Impala 4.0. There are many
> > exciting
> > > > >> projects underway including full support for Hive ACID tables,
> > > improved
> > > > >> multithreading support, and fault tolerance improvements, but
> Impala
> > > 4.0
> > > > >> also serves as a time to alter behavior and platform support. Here
> > is
> > > > the
> > > > >> current list of notable breaking changes that are planned in
> Impala
> > > 4.0:
> > > > >>
> > > > >> Remove support for older operating systems:
> > > > >>
> > > > >> Maintaining support for older operating systems requires a
> > > considerable
> > > > >> amount of effort, especially with changing security requirements.
> > Each
> > > > >> extra operating system consumes valuable effort and can conflict
> > with
> > > > >> support for newer operating systems, so we have decided to remove
> > > > support
> > > > >> for some older operating systems. Specifically, we plan to drop
> > > support
> > > > for:
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Centos 6
> > > > >> -
> > > > >>
> > > > >> Ubuntu 14
> > > > >> -
> > > > >>
> > > > >> Debian 8
> > > > >>
> > > > >> Each of these is approaching or past its end of life. Since
> Centos6
> > > was
> > > > >> the only operating system still needing Python 2.6 and Python 2.6
> is
> > > > long
> > > > >> past its end of life, Python 2.6 will no longer be supported. In
> > > > addition,
> > > > >> Centos 7 support will be focused on newer versions such as 7.5 and
> > > > above.
> > > > >>
> > > > >> Remove support for Sentry:
> > > > >>
> > > > >> Over the past year and a half, Impala's Ranger authorization
> > > > >> functionality has achieved parity and surpassed the existing
> Sentry
> > > > >> authorization functionality. Impala's Sentry support requires
> > ongoing
> > > > >> effort to test and maintain, including maintenance on Sentry
> itself
> > to
> > > > >> address security updates. Given the diminished activity in the
> > Sentry
> > > > >> community, there is no timeline for addressing these security
> > updates.
> > > > >> Unless there is a maintainer for Sentry support, Impala plans to
> > focus
> > > > its
> > > > >> efforts on Ranger for its authorization system and drop Sentry.
> > > > >>
> > > > >> Remove support for Impala-lzo:
> > > > >>
> > > > >> Impala-lzo provides code to allow Impala to read the LZO
> compressed
> > > > >> tables. LZO is GPL licensed, which is why this support is not
> > included
> > > > >> directly. The Impala-lzo code interacts with internal Impala code
> > at a
> > > > >> level that is error prone and intricate. Given the low adoption of
> > LZO
> > > > and
> > > > >> the other compression options available, Impala plans to remove
> > > > Impala-lzo
> > > > >> support along with the low level interface it used.
> > > > >>
> > > > >> Deprecations:
> > > > >>
> > > > >> In addition, we also plan to deprecate several existing features
> so
> > > that
> > > > >> they can be removed in a future release. Here is a summary of the
> > most
> > > > >> notable ones:
> > > > >>
> > > > >> -
> > > > >>
> > > > >> Impala will default to using on demand metadata management
> (i.e.
> > > the
> > > > >> local catalog implementation) as described in this
> documentation:
> > > > >>
> > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > >> The old metadata system is now deprecated and may be removed
> in a
> > > > >> future release.
> > > > >> -
> > > > >>
> > > > >> Impala is deprecating the Beeswax client protocol (i.e. clients
> > > that
> > > > >> connect via beeswax_port) in favor of the HiveServer2 client
> > > > protocol.
> > > > >> -
> > > > >>
> > > > >> Impala is deprecating the old decimal_v2=false behavior from
> > Impala
> > > > >> 2.x. The decimal_v2 query option has defaulted to true since
> > Impala
> > > > 3.
> > > > >>
> > > > >>
> > > > >> Impala is always open to new developers, and we welcome feedback
> on
> > > > these
> > > > >> plans. Further discussion of other changes and deprecations is
> > ongoing
> > > > at
> > > > >> dev@impala.apache.org.
> > > > >>
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Joe McDonnell
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Quanlong Huang <hu...@gmail.com>.
Sure. If there are no objections, I'll raise a discussion thread for the
4.0 release. There are still some unresolved breaking changes, e.g.
- IMPALA-2210: Make Parquet the default file format
- IMPALA-9690: Bump minimum x86-64 CPU requirements
Thanks
Quanlong
On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <jb...@apache.org> wrote:
> Thanks for bringing this up again, Quanlong! I would love to see a 4.0
> release soon so that 4.1 releases and what not can be prepared. Already the
> changlist from the 3.x line is going to be enormous and could be hard for
> users to digest.
>
> On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <hu...@gmail.com>
> wrote:
>
> > Reviving this thread. I think it's time to create the 4.0.0 branch and
> > prepare for the release now. Any breaking change is landing and we should
> > wait?
> >
> > Thanks,
> > Quanlong
> >
> > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > Another thing that we're looking into is changing the encoded runtime
> > > profile representation to be more efficient - see
> > > https://issues.apache.org/jira/browse/IMPALA-9378. So that might be a
> > > default we will try to change in Impala 4.0.
> > >
> > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> tarmstrong@cloudera.com>
> > > wrote:
> > >
> > > > An addendum to this - we're also considering whether to increase the
> > > > minimum CPU version so that we can simplify development and focus
> more
> > on
> > > > optimising for the most common CPUs - see
> > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > >
> > > > Running on commodity hardware is important and we want people to be
> > able
> > > > to develop on whatever machine they have access to. At most we're
> > likely
> > > > going to require AVX2 support, which has been supported by the vast
> > > > majority of CPUs for the better part of a decade.
> > > >
> > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > joemcdonnell@cloudera.com
> > > >
> > > > wrote:
> > > >
> > > >> Impala is starting development on Impala 4.0. There are many
> exciting
> > > >> projects underway including full support for Hive ACID tables,
> > improved
> > > >> multithreading support, and fault tolerance improvements, but Impala
> > 4.0
> > > >> also serves as a time to alter behavior and platform support. Here
> is
> > > the
> > > >> current list of notable breaking changes that are planned in Impala
> > 4.0:
> > > >>
> > > >> Remove support for older operating systems:
> > > >>
> > > >> Maintaining support for older operating systems requires a
> > considerable
> > > >> amount of effort, especially with changing security requirements.
> Each
> > > >> extra operating system consumes valuable effort and can conflict
> with
> > > >> support for newer operating systems, so we have decided to remove
> > > support
> > > >> for some older operating systems. Specifically, we plan to drop
> > support
> > > for:
> > > >>
> > > >> -
> > > >>
> > > >> Centos 6
> > > >> -
> > > >>
> > > >> Ubuntu 14
> > > >> -
> > > >>
> > > >> Debian 8
> > > >>
> > > >> Each of these is approaching or past its end of life. Since Centos6
> > was
> > > >> the only operating system still needing Python 2.6 and Python 2.6 is
> > > long
> > > >> past its end of life, Python 2.6 will no longer be supported. In
> > > addition,
> > > >> Centos 7 support will be focused on newer versions such as 7.5 and
> > > above.
> > > >>
> > > >> Remove support for Sentry:
> > > >>
> > > >> Over the past year and a half, Impala's Ranger authorization
> > > >> functionality has achieved parity and surpassed the existing Sentry
> > > >> authorization functionality. Impala's Sentry support requires
> ongoing
> > > >> effort to test and maintain, including maintenance on Sentry itself
> to
> > > >> address security updates. Given the diminished activity in the
> Sentry
> > > >> community, there is no timeline for addressing these security
> updates.
> > > >> Unless there is a maintainer for Sentry support, Impala plans to
> focus
> > > its
> > > >> efforts on Ranger for its authorization system and drop Sentry.
> > > >>
> > > >> Remove support for Impala-lzo:
> > > >>
> > > >> Impala-lzo provides code to allow Impala to read the LZO compressed
> > > >> tables. LZO is GPL licensed, which is why this support is not
> included
> > > >> directly. The Impala-lzo code interacts with internal Impala code
> at a
> > > >> level that is error prone and intricate. Given the low adoption of
> LZO
> > > and
> > > >> the other compression options available, Impala plans to remove
> > > Impala-lzo
> > > >> support along with the low level interface it used.
> > > >>
> > > >> Deprecations:
> > > >>
> > > >> In addition, we also plan to deprecate several existing features so
> > that
> > > >> they can be removed in a future release. Here is a summary of the
> most
> > > >> notable ones:
> > > >>
> > > >> -
> > > >>
> > > >> Impala will default to using on demand metadata management (i.e.
> > the
> > > >> local catalog implementation) as described in this documentation:
> > > >>
> > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > >> The old metadata system is now deprecated and may be removed in a
> > > >> future release.
> > > >> -
> > > >>
> > > >> Impala is deprecating the Beeswax client protocol (i.e. clients
> > that
> > > >> connect via beeswax_port) in favor of the HiveServer2 client
> > > protocol.
> > > >> -
> > > >>
> > > >> Impala is deprecating the old decimal_v2=false behavior from
> Impala
> > > >> 2.x. The decimal_v2 query option has defaulted to true since
> Impala
> > > 3.
> > > >>
> > > >>
> > > >> Impala is always open to new developers, and we welcome feedback on
> > > these
> > > >> plans. Further discussion of other changes and deprecations is
> ongoing
> > > at
> > > >> dev@impala.apache.org.
> > > >>
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Joe McDonnell
> > > >>
> > > >>
> > >
> >
>
Re: Impala 4 Breaking Changes
Posted by Jim Apple <jb...@apache.org>.
Thanks for bringing this up again, Quanlong! I would love to see a 4.0
release soon so that 4.1 releases and what not can be prepared. Already the
changlist from the 3.x line is going to be enormous and could be hard for
users to digest.
On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <hu...@gmail.com>
wrote:
> Reviving this thread. I think it's time to create the 4.0.0 branch and
> prepare for the release now. Any breaking change is landing and we should
> wait?
>
> Thanks,
> Quanlong
>
> On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> > Another thing that we're looking into is changing the encoded runtime
> > profile representation to be more efficient - see
> > https://issues.apache.org/jira/browse/IMPALA-9378. So that might be a
> > default we will try to change in Impala 4.0.
> >
> > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > An addendum to this - we're also considering whether to increase the
> > > minimum CPU version so that we can simplify development and focus more
> on
> > > optimising for the most common CPUs - see
> > > https://issues.apache.org/jira/browse/IMPALA-9690
> > >
> > > Running on commodity hardware is important and we want people to be
> able
> > > to develop on whatever machine they have access to. At most we're
> likely
> > > going to require AVX2 support, which has been supported by the vast
> > > majority of CPUs for the better part of a decade.
> > >
> > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> joemcdonnell@cloudera.com
> > >
> > > wrote:
> > >
> > >> Impala is starting development on Impala 4.0. There are many exciting
> > >> projects underway including full support for Hive ACID tables,
> improved
> > >> multithreading support, and fault tolerance improvements, but Impala
> 4.0
> > >> also serves as a time to alter behavior and platform support. Here is
> > the
> > >> current list of notable breaking changes that are planned in Impala
> 4.0:
> > >>
> > >> Remove support for older operating systems:
> > >>
> > >> Maintaining support for older operating systems requires a
> considerable
> > >> amount of effort, especially with changing security requirements. Each
> > >> extra operating system consumes valuable effort and can conflict with
> > >> support for newer operating systems, so we have decided to remove
> > support
> > >> for some older operating systems. Specifically, we plan to drop
> support
> > for:
> > >>
> > >> -
> > >>
> > >> Centos 6
> > >> -
> > >>
> > >> Ubuntu 14
> > >> -
> > >>
> > >> Debian 8
> > >>
> > >> Each of these is approaching or past its end of life. Since Centos6
> was
> > >> the only operating system still needing Python 2.6 and Python 2.6 is
> > long
> > >> past its end of life, Python 2.6 will no longer be supported. In
> > addition,
> > >> Centos 7 support will be focused on newer versions such as 7.5 and
> > above.
> > >>
> > >> Remove support for Sentry:
> > >>
> > >> Over the past year and a half, Impala's Ranger authorization
> > >> functionality has achieved parity and surpassed the existing Sentry
> > >> authorization functionality. Impala's Sentry support requires ongoing
> > >> effort to test and maintain, including maintenance on Sentry itself to
> > >> address security updates. Given the diminished activity in the Sentry
> > >> community, there is no timeline for addressing these security updates.
> > >> Unless there is a maintainer for Sentry support, Impala plans to focus
> > its
> > >> efforts on Ranger for its authorization system and drop Sentry.
> > >>
> > >> Remove support for Impala-lzo:
> > >>
> > >> Impala-lzo provides code to allow Impala to read the LZO compressed
> > >> tables. LZO is GPL licensed, which is why this support is not included
> > >> directly. The Impala-lzo code interacts with internal Impala code at a
> > >> level that is error prone and intricate. Given the low adoption of LZO
> > and
> > >> the other compression options available, Impala plans to remove
> > Impala-lzo
> > >> support along with the low level interface it used.
> > >>
> > >> Deprecations:
> > >>
> > >> In addition, we also plan to deprecate several existing features so
> that
> > >> they can be removed in a future release. Here is a summary of the most
> > >> notable ones:
> > >>
> > >> -
> > >>
> > >> Impala will default to using on demand metadata management (i.e.
> the
> > >> local catalog implementation) as described in this documentation:
> > >>
> http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > >> The old metadata system is now deprecated and may be removed in a
> > >> future release.
> > >> -
> > >>
> > >> Impala is deprecating the Beeswax client protocol (i.e. clients
> that
> > >> connect via beeswax_port) in favor of the HiveServer2 client
> > protocol.
> > >> -
> > >>
> > >> Impala is deprecating the old decimal_v2=false behavior from Impala
> > >> 2.x. The decimal_v2 query option has defaulted to true since Impala
> > 3.
> > >>
> > >>
> > >> Impala is always open to new developers, and we welcome feedback on
> > these
> > >> plans. Further discussion of other changes and deprecations is ongoing
> > at
> > >> dev@impala.apache.org.
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Joe McDonnell
> > >>
> > >>
> >
>
Re: Impala 4 Breaking Changes
Posted by Quanlong Huang <hu...@gmail.com>.
Reviving this thread. I think it's time to create the 4.0.0 branch and
prepare for the release now. Any breaking change is landing and we should
wait?
Thanks,
Quanlong
On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <ta...@cloudera.com>
wrote:
> Another thing that we're looking into is changing the encoded runtime
> profile representation to be more efficient - see
> https://issues.apache.org/jira/browse/IMPALA-9378. So that might be a
> default we will try to change in Impala 4.0.
>
> On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> > An addendum to this - we're also considering whether to increase the
> > minimum CPU version so that we can simplify development and focus more on
> > optimising for the most common CPUs - see
> > https://issues.apache.org/jira/browse/IMPALA-9690
> >
> > Running on commodity hardware is important and we want people to be able
> > to develop on whatever machine they have access to. At most we're likely
> > going to require AVX2 support, which has been supported by the vast
> > majority of CPUs for the better part of a decade.
> >
> > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <joemcdonnell@cloudera.com
> >
> > wrote:
> >
> >> Impala is starting development on Impala 4.0. There are many exciting
> >> projects underway including full support for Hive ACID tables, improved
> >> multithreading support, and fault tolerance improvements, but Impala 4.0
> >> also serves as a time to alter behavior and platform support. Here is
> the
> >> current list of notable breaking changes that are planned in Impala 4.0:
> >>
> >> Remove support for older operating systems:
> >>
> >> Maintaining support for older operating systems requires a considerable
> >> amount of effort, especially with changing security requirements. Each
> >> extra operating system consumes valuable effort and can conflict with
> >> support for newer operating systems, so we have decided to remove
> support
> >> for some older operating systems. Specifically, we plan to drop support
> for:
> >>
> >> -
> >>
> >> Centos 6
> >> -
> >>
> >> Ubuntu 14
> >> -
> >>
> >> Debian 8
> >>
> >> Each of these is approaching or past its end of life. Since Centos6 was
> >> the only operating system still needing Python 2.6 and Python 2.6 is
> long
> >> past its end of life, Python 2.6 will no longer be supported. In
> addition,
> >> Centos 7 support will be focused on newer versions such as 7.5 and
> above.
> >>
> >> Remove support for Sentry:
> >>
> >> Over the past year and a half, Impala's Ranger authorization
> >> functionality has achieved parity and surpassed the existing Sentry
> >> authorization functionality. Impala's Sentry support requires ongoing
> >> effort to test and maintain, including maintenance on Sentry itself to
> >> address security updates. Given the diminished activity in the Sentry
> >> community, there is no timeline for addressing these security updates.
> >> Unless there is a maintainer for Sentry support, Impala plans to focus
> its
> >> efforts on Ranger for its authorization system and drop Sentry.
> >>
> >> Remove support for Impala-lzo:
> >>
> >> Impala-lzo provides code to allow Impala to read the LZO compressed
> >> tables. LZO is GPL licensed, which is why this support is not included
> >> directly. The Impala-lzo code interacts with internal Impala code at a
> >> level that is error prone and intricate. Given the low adoption of LZO
> and
> >> the other compression options available, Impala plans to remove
> Impala-lzo
> >> support along with the low level interface it used.
> >>
> >> Deprecations:
> >>
> >> In addition, we also plan to deprecate several existing features so that
> >> they can be removed in a future release. Here is a summary of the most
> >> notable ones:
> >>
> >> -
> >>
> >> Impala will default to using on demand metadata management (i.e. the
> >> local catalog implementation) as described in this documentation:
> >> http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> >> The old metadata system is now deprecated and may be removed in a
> >> future release.
> >> -
> >>
> >> Impala is deprecating the Beeswax client protocol (i.e. clients that
> >> connect via beeswax_port) in favor of the HiveServer2 client
> protocol.
> >> -
> >>
> >> Impala is deprecating the old decimal_v2=false behavior from Impala
> >> 2.x. The decimal_v2 query option has defaulted to true since Impala
> 3.
> >>
> >>
> >> Impala is always open to new developers, and we welcome feedback on
> these
> >> plans. Further discussion of other changes and deprecations is ongoing
> at
> >> dev@impala.apache.org.
> >>
> >>
> >> Thanks,
> >>
> >> Joe McDonnell
> >>
> >>
>