You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Joe McDonnell <jo...@cloudera.com> on 2020/03/16 17:07:13 UTC

Impala 4.0 breaking changes

Now that Impala 3.4 is branched and master is Impala 4.0, we need to decide
what breaking changes will happen in Impala 4.0. I have provided a series
of proposals below. I welcome feedback on them. Other proposals are also
welcome.

Thanks,
Joe

Proposal 0: Hadoop component versions

Switch to CDP versions of components by default. This means that Impala
will use Hive 3+ (which is already essentially Hive 4 and may change names
to being Hive 4).
Remove support for CDH versions of components.
This was already discussed in the original thread for Impala 4, so this is
not new.

Proposal 1: OS support

Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
Centos 7 development will be focused on newer Centos 7 versions such as 7.6
and 7.7.
Add support for Centos 8
Move main development from Ubuntu 16 to Ubuntu 18 over time.

Proposal 2: Python support

Drop support for Python 2.6
Add support for Python 3 over time.

Proposal 3: Impala-lzo

Drop support for Impala-lzo/hadoop-lzo

Proposal 4: Clients

Deprecate beeswax protocol. This means that it can be removed in the next
major version number, but it would not be removed in Impala 4. Current
users of beeswax would need to start migrating to HS2.

Proposal 5: Sentry

Drop support for Sentry in favor of Ranger.

Proposal 6: Metadata

Metadata V2 will become the default. Metadata V1 will be deprecated.

Thanks,
Joe

Re: Impala 4.0 breaking changes

Posted by Csaba Ringhofer <cs...@cloudera.com>.
> What do you think about dateless timestamps? AFAIK that is not supported
+1, I think that dateless timestamps are just confusing both in the code
and for the users
I created a Jira to drop it: IMPALA-9531
A number of issues with them are listed in this jira: IMPALA-5942

On Wed, Mar 18, 2020 at 3:16 PM Gabor Kaszab <ga...@apache.org> wrote:

> What do you think about dateless timestamps? AFAIK that is not supported
> ATM, shouldn't we drop it?
>
> Gabor
>
> On Wed, Mar 18, 2020 at 1:46 AM Shant Hovsepian <sh...@cloudera.com>
> wrote:
>
> > +1 on RUNTIME_FILTER_WAIT_TIME_MS increasing.
> >
> > On Tue, Mar 17, 2020 at 5:43 PM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> > >
> > > I think we should consider changing a couple more defaults, after
> having
> > an
> > > offline conversion with Shant.
> > >
> > > We could change COMPRESSION_CODEC to LZ4 or ZSTD as the default. I
> think
> > > LZ4 is the safest option perf-wise, because it will be faster across
> the
> > > board and the decompression is now one of the main CPU bottlenecks for
> > > Parquet scanning. We might need to double-check that enough of the
> > > ecosystem supports LZ4, but this seems like it would be a good
> > improvement.
> > >
> > > It *might* we worth enabled compute stats table sampling by default,
> but
> > I
> > > think that could be open for discussion.
> > >
> > > We could also consider bumping RUNTIME_FILTER_WAIT_TIME_MS to a higher
> > > value, since I think generally higher values have proven to be more
> > robust
> > > for complex queries (TPC-DS, etc).
> > >
> > > On Tue, Mar 17, 2020 at 11:56 AM Tim Armstrong <
> tarmstrong@cloudera.com>
> > > wrote:
> > >
> > > > >   - Do we still need the DECIMAL_V2 query option? Seems like this
> has
> > > > been  true for a while. Maybe we can add it to the list of deprecated
> > flags?
> > > > Maybe we could officially deprecate it and phase it out soonish? It
> > really
> > > > only exists as a workaround for people upgrading from the old
> > behaviour in
> > > > 2.x. It hasn't been terribly bad maintaining the two code paths, but
> it
> > > > would be nice to simplify it.
> > > >
> > > > >   - Deprecate support for ADLS, since it has effectively been
> > replaced
> > > > by ABFS
> > > > Makes sense. It probably isn't too much overhead to keep the old code
> > > > around for a while, is it? Just in case users have a bunch of data
> > still
> > > > sitting in the old ADLS.
> > > >
> > > > >   - Deprecate (or even remove) support for HDFS cacheing? Not sure
> > how
> > > > extensively this is used, removing the code would be nice as it
> > simplifies
> > > > part of the HDFS read path
> > > > Anecdotally I do see it used, but a lot of times it's to affect
> > scheduling
> > > > rather than because saving memcpy() makes a real difference (with
> > > > compressed parquet, that's rarely the bottleneck) . A compromise or
> > > > in-between step would be to remove the special-casing of the
> zero-copy
> > code
> > > > path in the backend, but keep the scheduling behaviour.
> > > >
> > > > On Tue, Mar 17, 2020 at 11:50 AM Tim Armstrong <
> > tarmstrong@cloudera.com>
> > > > wrote:
> > > >
> > > >> I think I generally support this. A few specific comments.
> > > >>
> > > >> > Proposal 3: Impala-lzo
> > > >> > Drop support for Impala-lzo/hadoop-lzo
> > > >>
> > > >> Does this mean dropping the plugin text scanner interface entirely?
> > LZO
> > > >> is the only implementation of that that I'm aware of (and we rely on
> > it to
> > > >> test the interface) so seems reasonable to me to remove something
> > that has
> > > >> minimal adoption and not cleanly separated from the scanner
> > implementation
> > > >> of core Impala.
> > > >>
> > > >> > Proposal 5: Sentry
> > > >> > Drop support for Sentry in favor of Ranger.
> > > >>
> > > >> I think moving this direction makes a lot of sense given that
> > activity in
> > > >> the Sentry project has declined a lot (just look at the activity
> > level on
> > > >> the two projects, it's dramatically different), unless someone in
> the
> > > >> community wants to step up and maintain the integration.
> > > >>
> > > >> > Proposal 6: Metadata
> > > >> > Metadata V2 will become the default. Metadata V1 will be
> deprecated.
> > > >> Maybe we should set a goal of removing the support in Impala 4.1 or
> > 4.2?
> > > >> That would allow us to remove a lot of complex code
> > > >>
> > > >> On Mon, Mar 16, 2020 at 10:07 AM Joe McDonnell <
> > joemcdonnell@cloudera.com>
> > > >> wrote:
> > > >>
> > > >>> Now that Impala 3.4 is branched and master is Impala 4.0, we need
> to
> > > >>> decide
> > > >>> what breaking changes will happen in Impala 4.0. I have provided a
> > series
> > > >>> of proposals below. I welcome feedback on them. Other proposals are
> > also
> > > >>> welcome.
> > > >>>
> > > >>> Thanks,
> > > >>> Joe
> > > >>>
> > > >>> Proposal 0: Hadoop component versions
> > > >>>
> > > >>> Switch to CDP versions of components by default. This means that
> > Impala
> > > >>> will use Hive 3+ (which is already essentially Hive 4 and may
> change
> > > >>> names
> > > >>> to being Hive 4).
> > > >>> Remove support for CDH versions of components.
> > > >>> This was already discussed in the original thread for Impala 4, so
> > this
> > > >>> is
> > > >>> not new.
> > > >>>
> > > >>> Proposal 1: OS support
> > > >>>
> > > >>> Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> > > >>> Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> > > >>> Centos 7 development will be focused on newer Centos 7 versions
> such
> > as
> > > >>> 7.6
> > > >>> and 7.7.
> > > >>> Add support for Centos 8
> > > >>> Move main development from Ubuntu 16 to Ubuntu 18 over time.
> > > >>>
> > > >>> Proposal 2: Python support
> > > >>>
> > > >>> Drop support for Python 2.6
> > > >>> Add support for Python 3 over time.
> > > >>>
> > > >>> Proposal 3: Impala-lzo
> > > >>>
> > > >>> Drop support for Impala-lzo/hadoop-lzo
> > > >>>
> > > >>> Proposal 4: Clients
> > > >>>
> > > >>> Deprecate beeswax protocol. This means that it can be removed in
> the
> > next
> > > >>> major version number, but it would not be removed in Impala 4.
> > Current
> > > >>> users of beeswax would need to start migrating to HS2.
> > > >>>
> > > >>> Proposal 5: Sentry
> > > >>>
> > > >>> Drop support for Sentry in favor of Ranger.
> > > >>>
> > > >>> Proposal 6: Metadata
> > > >>>
> > > >>> Metadata V2 will become the default. Metadata V1 will be
> deprecated.
> > > >>>
> > > >>> Thanks,
> > > >>> Joe
> > > >>>
> > > >>
> >
>

Re: Impala 4.0 breaking changes

Posted by Gabor Kaszab <ga...@apache.org>.
What do you think about dateless timestamps? AFAIK that is not supported
ATM, shouldn't we drop it?

Gabor

On Wed, Mar 18, 2020 at 1:46 AM Shant Hovsepian <sh...@cloudera.com> wrote:

> +1 on RUNTIME_FILTER_WAIT_TIME_MS increasing.
>
> On Tue, Mar 17, 2020 at 5:43 PM Tim Armstrong <ta...@cloudera.com>
> wrote:
> >
> > I think we should consider changing a couple more defaults, after having
> an
> > offline conversion with Shant.
> >
> > We could change COMPRESSION_CODEC to LZ4 or ZSTD as the default. I think
> > LZ4 is the safest option perf-wise, because it will be faster across the
> > board and the decompression is now one of the main CPU bottlenecks for
> > Parquet scanning. We might need to double-check that enough of the
> > ecosystem supports LZ4, but this seems like it would be a good
> improvement.
> >
> > It *might* we worth enabled compute stats table sampling by default, but
> I
> > think that could be open for discussion.
> >
> > We could also consider bumping RUNTIME_FILTER_WAIT_TIME_MS to a higher
> > value, since I think generally higher values have proven to be more
> robust
> > for complex queries (TPC-DS, etc).
> >
> > On Tue, Mar 17, 2020 at 11:56 AM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > >   - Do we still need the DECIMAL_V2 query option? Seems like this has
> > > been  true for a while. Maybe we can add it to the list of deprecated
> flags?
> > > Maybe we could officially deprecate it and phase it out soonish? It
> really
> > > only exists as a workaround for people upgrading from the old
> behaviour in
> > > 2.x. It hasn't been terribly bad maintaining the two code paths, but it
> > > would be nice to simplify it.
> > >
> > > >   - Deprecate support for ADLS, since it has effectively been
> replaced
> > > by ABFS
> > > Makes sense. It probably isn't too much overhead to keep the old code
> > > around for a while, is it? Just in case users have a bunch of data
> still
> > > sitting in the old ADLS.
> > >
> > > >   - Deprecate (or even remove) support for HDFS cacheing? Not sure
> how
> > > extensively this is used, removing the code would be nice as it
> simplifies
> > > part of the HDFS read path
> > > Anecdotally I do see it used, but a lot of times it's to affect
> scheduling
> > > rather than because saving memcpy() makes a real difference (with
> > > compressed parquet, that's rarely the bottleneck) . A compromise or
> > > in-between step would be to remove the special-casing of the zero-copy
> code
> > > path in the backend, but keep the scheduling behaviour.
> > >
> > > On Tue, Mar 17, 2020 at 11:50 AM Tim Armstrong <
> tarmstrong@cloudera.com>
> > > wrote:
> > >
> > >> I think I generally support this. A few specific comments.
> > >>
> > >> > Proposal 3: Impala-lzo
> > >> > Drop support for Impala-lzo/hadoop-lzo
> > >>
> > >> Does this mean dropping the plugin text scanner interface entirely?
> LZO
> > >> is the only implementation of that that I'm aware of (and we rely on
> it to
> > >> test the interface) so seems reasonable to me to remove something
> that has
> > >> minimal adoption and not cleanly separated from the scanner
> implementation
> > >> of core Impala.
> > >>
> > >> > Proposal 5: Sentry
> > >> > Drop support for Sentry in favor of Ranger.
> > >>
> > >> I think moving this direction makes a lot of sense given that
> activity in
> > >> the Sentry project has declined a lot (just look at the activity
> level on
> > >> the two projects, it's dramatically different), unless someone in the
> > >> community wants to step up and maintain the integration.
> > >>
> > >> > Proposal 6: Metadata
> > >> > Metadata V2 will become the default. Metadata V1 will be deprecated.
> > >> Maybe we should set a goal of removing the support in Impala 4.1 or
> 4.2?
> > >> That would allow us to remove a lot of complex code
> > >>
> > >> On Mon, Mar 16, 2020 at 10:07 AM Joe McDonnell <
> joemcdonnell@cloudera.com>
> > >> wrote:
> > >>
> > >>> Now that Impala 3.4 is branched and master is Impala 4.0, we need to
> > >>> decide
> > >>> what breaking changes will happen in Impala 4.0. I have provided a
> series
> > >>> of proposals below. I welcome feedback on them. Other proposals are
> also
> > >>> welcome.
> > >>>
> > >>> Thanks,
> > >>> Joe
> > >>>
> > >>> Proposal 0: Hadoop component versions
> > >>>
> > >>> Switch to CDP versions of components by default. This means that
> Impala
> > >>> will use Hive 3+ (which is already essentially Hive 4 and may change
> > >>> names
> > >>> to being Hive 4).
> > >>> Remove support for CDH versions of components.
> > >>> This was already discussed in the original thread for Impala 4, so
> this
> > >>> is
> > >>> not new.
> > >>>
> > >>> Proposal 1: OS support
> > >>>
> > >>> Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> > >>> Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> > >>> Centos 7 development will be focused on newer Centos 7 versions such
> as
> > >>> 7.6
> > >>> and 7.7.
> > >>> Add support for Centos 8
> > >>> Move main development from Ubuntu 16 to Ubuntu 18 over time.
> > >>>
> > >>> Proposal 2: Python support
> > >>>
> > >>> Drop support for Python 2.6
> > >>> Add support for Python 3 over time.
> > >>>
> > >>> Proposal 3: Impala-lzo
> > >>>
> > >>> Drop support for Impala-lzo/hadoop-lzo
> > >>>
> > >>> Proposal 4: Clients
> > >>>
> > >>> Deprecate beeswax protocol. This means that it can be removed in the
> next
> > >>> major version number, but it would not be removed in Impala 4.
> Current
> > >>> users of beeswax would need to start migrating to HS2.
> > >>>
> > >>> Proposal 5: Sentry
> > >>>
> > >>> Drop support for Sentry in favor of Ranger.
> > >>>
> > >>> Proposal 6: Metadata
> > >>>
> > >>> Metadata V2 will become the default. Metadata V1 will be deprecated.
> > >>>
> > >>> Thanks,
> > >>> Joe
> > >>>
> > >>
>

Re: Impala 4.0 breaking changes

Posted by Shant Hovsepian <sh...@cloudera.com>.
+1 on RUNTIME_FILTER_WAIT_TIME_MS increasing.

On Tue, Mar 17, 2020 at 5:43 PM Tim Armstrong <ta...@cloudera.com> wrote:
>
> I think we should consider changing a couple more defaults, after having an
> offline conversion with Shant.
>
> We could change COMPRESSION_CODEC to LZ4 or ZSTD as the default. I think
> LZ4 is the safest option perf-wise, because it will be faster across the
> board and the decompression is now one of the main CPU bottlenecks for
> Parquet scanning. We might need to double-check that enough of the
> ecosystem supports LZ4, but this seems like it would be a good improvement.
>
> It *might* we worth enabled compute stats table sampling by default, but I
> think that could be open for discussion.
>
> We could also consider bumping RUNTIME_FILTER_WAIT_TIME_MS to a higher
> value, since I think generally higher values have proven to be more robust
> for complex queries (TPC-DS, etc).
>
> On Tue, Mar 17, 2020 at 11:56 AM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> > >   - Do we still need the DECIMAL_V2 query option? Seems like this has
> > been  true for a while. Maybe we can add it to the list of deprecated flags?
> > Maybe we could officially deprecate it and phase it out soonish? It really
> > only exists as a workaround for people upgrading from the old behaviour in
> > 2.x. It hasn't been terribly bad maintaining the two code paths, but it
> > would be nice to simplify it.
> >
> > >   - Deprecate support for ADLS, since it has effectively been replaced
> > by ABFS
> > Makes sense. It probably isn't too much overhead to keep the old code
> > around for a while, is it? Just in case users have a bunch of data still
> > sitting in the old ADLS.
> >
> > >   - Deprecate (or even remove) support for HDFS cacheing? Not sure how
> > extensively this is used, removing the code would be nice as it simplifies
> > part of the HDFS read path
> > Anecdotally I do see it used, but a lot of times it's to affect scheduling
> > rather than because saving memcpy() makes a real difference (with
> > compressed parquet, that's rarely the bottleneck) . A compromise or
> > in-between step would be to remove the special-casing of the zero-copy code
> > path in the backend, but keep the scheduling behaviour.
> >
> > On Tue, Mar 17, 2020 at 11:50 AM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> >> I think I generally support this. A few specific comments.
> >>
> >> > Proposal 3: Impala-lzo
> >> > Drop support for Impala-lzo/hadoop-lzo
> >>
> >> Does this mean dropping the plugin text scanner interface entirely? LZO
> >> is the only implementation of that that I'm aware of (and we rely on it to
> >> test the interface) so seems reasonable to me to remove something that has
> >> minimal adoption and not cleanly separated from the scanner implementation
> >> of core Impala.
> >>
> >> > Proposal 5: Sentry
> >> > Drop support for Sentry in favor of Ranger.
> >>
> >> I think moving this direction makes a lot of sense given that activity in
> >> the Sentry project has declined a lot (just look at the activity level on
> >> the two projects, it's dramatically different), unless someone in the
> >> community wants to step up and maintain the integration.
> >>
> >> > Proposal 6: Metadata
> >> > Metadata V2 will become the default. Metadata V1 will be deprecated.
> >> Maybe we should set a goal of removing the support in Impala 4.1 or 4.2?
> >> That would allow us to remove a lot of complex code
> >>
> >> On Mon, Mar 16, 2020 at 10:07 AM Joe McDonnell <jo...@cloudera.com>
> >> wrote:
> >>
> >>> Now that Impala 3.4 is branched and master is Impala 4.0, we need to
> >>> decide
> >>> what breaking changes will happen in Impala 4.0. I have provided a series
> >>> of proposals below. I welcome feedback on them. Other proposals are also
> >>> welcome.
> >>>
> >>> Thanks,
> >>> Joe
> >>>
> >>> Proposal 0: Hadoop component versions
> >>>
> >>> Switch to CDP versions of components by default. This means that Impala
> >>> will use Hive 3+ (which is already essentially Hive 4 and may change
> >>> names
> >>> to being Hive 4).
> >>> Remove support for CDH versions of components.
> >>> This was already discussed in the original thread for Impala 4, so this
> >>> is
> >>> not new.
> >>>
> >>> Proposal 1: OS support
> >>>
> >>> Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> >>> Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> >>> Centos 7 development will be focused on newer Centos 7 versions such as
> >>> 7.6
> >>> and 7.7.
> >>> Add support for Centos 8
> >>> Move main development from Ubuntu 16 to Ubuntu 18 over time.
> >>>
> >>> Proposal 2: Python support
> >>>
> >>> Drop support for Python 2.6
> >>> Add support for Python 3 over time.
> >>>
> >>> Proposal 3: Impala-lzo
> >>>
> >>> Drop support for Impala-lzo/hadoop-lzo
> >>>
> >>> Proposal 4: Clients
> >>>
> >>> Deprecate beeswax protocol. This means that it can be removed in the next
> >>> major version number, but it would not be removed in Impala 4. Current
> >>> users of beeswax would need to start migrating to HS2.
> >>>
> >>> Proposal 5: Sentry
> >>>
> >>> Drop support for Sentry in favor of Ranger.
> >>>
> >>> Proposal 6: Metadata
> >>>
> >>> Metadata V2 will become the default. Metadata V1 will be deprecated.
> >>>
> >>> Thanks,
> >>> Joe
> >>>
> >>

Re: Impala 4.0 breaking changes

Posted by Tim Armstrong <ta...@cloudera.com>.
I think we should consider changing a couple more defaults, after having an
offline conversion with Shant.

We could change COMPRESSION_CODEC to LZ4 or ZSTD as the default. I think
LZ4 is the safest option perf-wise, because it will be faster across the
board and the decompression is now one of the main CPU bottlenecks for
Parquet scanning. We might need to double-check that enough of the
ecosystem supports LZ4, but this seems like it would be a good improvement.

It *might* we worth enabled compute stats table sampling by default, but I
think that could be open for discussion.

We could also consider bumping RUNTIME_FILTER_WAIT_TIME_MS to a higher
value, since I think generally higher values have proven to be more robust
for complex queries (TPC-DS, etc).

On Tue, Mar 17, 2020 at 11:56 AM Tim Armstrong <ta...@cloudera.com>
wrote:

> >   - Do we still need the DECIMAL_V2 query option? Seems like this has
> been  true for a while. Maybe we can add it to the list of deprecated flags?
> Maybe we could officially deprecate it and phase it out soonish? It really
> only exists as a workaround for people upgrading from the old behaviour in
> 2.x. It hasn't been terribly bad maintaining the two code paths, but it
> would be nice to simplify it.
>
> >   - Deprecate support for ADLS, since it has effectively been replaced
> by ABFS
> Makes sense. It probably isn't too much overhead to keep the old code
> around for a while, is it? Just in case users have a bunch of data still
> sitting in the old ADLS.
>
> >   - Deprecate (or even remove) support for HDFS cacheing? Not sure how
> extensively this is used, removing the code would be nice as it simplifies
> part of the HDFS read path
> Anecdotally I do see it used, but a lot of times it's to affect scheduling
> rather than because saving memcpy() makes a real difference (with
> compressed parquet, that's rarely the bottleneck) . A compromise or
> in-between step would be to remove the special-casing of the zero-copy code
> path in the backend, but keep the scheduling behaviour.
>
> On Tue, Mar 17, 2020 at 11:50 AM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
>> I think I generally support this. A few specific comments.
>>
>> > Proposal 3: Impala-lzo
>> > Drop support for Impala-lzo/hadoop-lzo
>>
>> Does this mean dropping the plugin text scanner interface entirely? LZO
>> is the only implementation of that that I'm aware of (and we rely on it to
>> test the interface) so seems reasonable to me to remove something that has
>> minimal adoption and not cleanly separated from the scanner implementation
>> of core Impala.
>>
>> > Proposal 5: Sentry
>> > Drop support for Sentry in favor of Ranger.
>>
>> I think moving this direction makes a lot of sense given that activity in
>> the Sentry project has declined a lot (just look at the activity level on
>> the two projects, it's dramatically different), unless someone in the
>> community wants to step up and maintain the integration.
>>
>> > Proposal 6: Metadata
>> > Metadata V2 will become the default. Metadata V1 will be deprecated.
>> Maybe we should set a goal of removing the support in Impala 4.1 or 4.2?
>> That would allow us to remove a lot of complex code
>>
>> On Mon, Mar 16, 2020 at 10:07 AM Joe McDonnell <jo...@cloudera.com>
>> wrote:
>>
>>> Now that Impala 3.4 is branched and master is Impala 4.0, we need to
>>> decide
>>> what breaking changes will happen in Impala 4.0. I have provided a series
>>> of proposals below. I welcome feedback on them. Other proposals are also
>>> welcome.
>>>
>>> Thanks,
>>> Joe
>>>
>>> Proposal 0: Hadoop component versions
>>>
>>> Switch to CDP versions of components by default. This means that Impala
>>> will use Hive 3+ (which is already essentially Hive 4 and may change
>>> names
>>> to being Hive 4).
>>> Remove support for CDH versions of components.
>>> This was already discussed in the original thread for Impala 4, so this
>>> is
>>> not new.
>>>
>>> Proposal 1: OS support
>>>
>>> Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
>>> Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
>>> Centos 7 development will be focused on newer Centos 7 versions such as
>>> 7.6
>>> and 7.7.
>>> Add support for Centos 8
>>> Move main development from Ubuntu 16 to Ubuntu 18 over time.
>>>
>>> Proposal 2: Python support
>>>
>>> Drop support for Python 2.6
>>> Add support for Python 3 over time.
>>>
>>> Proposal 3: Impala-lzo
>>>
>>> Drop support for Impala-lzo/hadoop-lzo
>>>
>>> Proposal 4: Clients
>>>
>>> Deprecate beeswax protocol. This means that it can be removed in the next
>>> major version number, but it would not be removed in Impala 4. Current
>>> users of beeswax would need to start migrating to HS2.
>>>
>>> Proposal 5: Sentry
>>>
>>> Drop support for Sentry in favor of Ranger.
>>>
>>> Proposal 6: Metadata
>>>
>>> Metadata V2 will become the default. Metadata V1 will be deprecated.
>>>
>>> Thanks,
>>> Joe
>>>
>>

Re: Impala 4.0 breaking changes

Posted by Tim Armstrong <ta...@cloudera.com>.
>   - Do we still need the DECIMAL_V2 query option? Seems like this has
been  true for a while. Maybe we can add it to the list of deprecated flags?
Maybe we could officially deprecate it and phase it out soonish? It really
only exists as a workaround for people upgrading from the old behaviour in
2.x. It hasn't been terribly bad maintaining the two code paths, but it
would be nice to simplify it.

>   - Deprecate support for ADLS, since it has effectively been replaced by
ABFS
Makes sense. It probably isn't too much overhead to keep the old code
around for a while, is it? Just in case users have a bunch of data still
sitting in the old ADLS.

>   - Deprecate (or even remove) support for HDFS cacheing? Not sure how
extensively this is used, removing the code would be nice as it simplifies
part of the HDFS read path
Anecdotally I do see it used, but a lot of times it's to affect scheduling
rather than because saving memcpy() makes a real difference (with
compressed parquet, that's rarely the bottleneck) . A compromise or
in-between step would be to remove the special-casing of the zero-copy code
path in the backend, but keep the scheduling behaviour.

On Tue, Mar 17, 2020 at 11:50 AM Tim Armstrong <ta...@cloudera.com>
wrote:

> I think I generally support this. A few specific comments.
>
> > Proposal 3: Impala-lzo
> > Drop support for Impala-lzo/hadoop-lzo
>
> Does this mean dropping the plugin text scanner interface entirely? LZO is
> the only implementation of that that I'm aware of (and we rely on it to
> test the interface) so seems reasonable to me to remove something that has
> minimal adoption and not cleanly separated from the scanner implementation
> of core Impala.
>
> > Proposal 5: Sentry
> > Drop support for Sentry in favor of Ranger.
>
> I think moving this direction makes a lot of sense given that activity in
> the Sentry project has declined a lot (just look at the activity level on
> the two projects, it's dramatically different), unless someone in the
> community wants to step up and maintain the integration.
>
> > Proposal 6: Metadata
> > Metadata V2 will become the default. Metadata V1 will be deprecated.
> Maybe we should set a goal of removing the support in Impala 4.1 or 4.2?
> That would allow us to remove a lot of complex code
>
> On Mon, Mar 16, 2020 at 10:07 AM Joe McDonnell <jo...@cloudera.com>
> wrote:
>
>> Now that Impala 3.4 is branched and master is Impala 4.0, we need to
>> decide
>> what breaking changes will happen in Impala 4.0. I have provided a series
>> of proposals below. I welcome feedback on them. Other proposals are also
>> welcome.
>>
>> Thanks,
>> Joe
>>
>> Proposal 0: Hadoop component versions
>>
>> Switch to CDP versions of components by default. This means that Impala
>> will use Hive 3+ (which is already essentially Hive 4 and may change names
>> to being Hive 4).
>> Remove support for CDH versions of components.
>> This was already discussed in the original thread for Impala 4, so this is
>> not new.
>>
>> Proposal 1: OS support
>>
>> Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
>> Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
>> Centos 7 development will be focused on newer Centos 7 versions such as
>> 7.6
>> and 7.7.
>> Add support for Centos 8
>> Move main development from Ubuntu 16 to Ubuntu 18 over time.
>>
>> Proposal 2: Python support
>>
>> Drop support for Python 2.6
>> Add support for Python 3 over time.
>>
>> Proposal 3: Impala-lzo
>>
>> Drop support for Impala-lzo/hadoop-lzo
>>
>> Proposal 4: Clients
>>
>> Deprecate beeswax protocol. This means that it can be removed in the next
>> major version number, but it would not be removed in Impala 4. Current
>> users of beeswax would need to start migrating to HS2.
>>
>> Proposal 5: Sentry
>>
>> Drop support for Sentry in favor of Ranger.
>>
>> Proposal 6: Metadata
>>
>> Metadata V2 will become the default. Metadata V1 will be deprecated.
>>
>> Thanks,
>> Joe
>>
>

Re: Impala 4.0 breaking changes

Posted by Joe McDonnell <jo...@cloudera.com>.
Response inline:

On Tue, Mar 17, 2020 at 11:51 AM Tim Armstrong <ta...@cloudera.com>
wrote:

> I think I generally support this. A few specific comments.
>
> > Proposal 3: Impala-lzo
> > Drop support for Impala-lzo/hadoop-lzo
>
> Does this mean dropping the plugin text scanner interface entirely? LZO is
> the only implementation of that that I'm aware of (and we rely on it to
> test the interface) so seems reasonable to me to remove something that has
> minimal adoption and not cleanly separated from the scanner implementation
> of core Impala.
>

I think it makes sense to make a clean break and remove the plugin text
scanner interface unless there is a clear user. The interaction with the
scanner side is unstable and intricate.
I think this would be done immediately in Impala 4.0.


>
> > Proposal 5: Sentry
> > Drop support for Sentry in favor of Ranger.
>
> I think moving this direction makes a lot of sense given that activity in
> the Sentry project has declined a lot (just look at the activity level on
> the two projects, it's dramatically different), unless someone in the
> community wants to step up and maintain the integration.


> > Proposal 6: Metadata
> > Metadata V2 will become the default. Metadata V1 will be deprecated.
> Maybe we should set a goal of removing the support in Impala 4.1 or 4.2?
> That would allow us to remove a lot of complex code
>

I think it would be reasonable to decide that some version of Impala 4 will
remove Metadata V1. The sooner the better, but it will depend on our
assessment of V2.


>
> On Mon, Mar 16, 2020 at 10:07 AM Joe McDonnell <jo...@cloudera.com>
> wrote:
>
> > Now that Impala 3.4 is branched and master is Impala 4.0, we need to
> decide
> > what breaking changes will happen in Impala 4.0. I have provided a series
> > of proposals below. I welcome feedback on them. Other proposals are also
> > welcome.
> >
> > Thanks,
> > Joe
> >
> > Proposal 0: Hadoop component versions
> >
> > Switch to CDP versions of components by default. This means that Impala
> > will use Hive 3+ (which is already essentially Hive 4 and may change
> names
> > to being Hive 4).
> > Remove support for CDH versions of components.
> > This was already discussed in the original thread for Impala 4, so this
> is
> > not new.
> >
> > Proposal 1: OS support
> >
> > Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> > Centos 7 development will be focused on newer Centos 7 versions such as
> 7.6
> > and 7.7.
> > Add support for Centos 8
> > Move main development from Ubuntu 16 to Ubuntu 18 over time.
> >
> > Proposal 2: Python support
> >
> > Drop support for Python 2.6
> > Add support for Python 3 over time.
> >
> > Proposal 3: Impala-lzo
> >
> > Drop support for Impala-lzo/hadoop-lzo
> >
> > Proposal 4: Clients
> >
> > Deprecate beeswax protocol. This means that it can be removed in the next
> > major version number, but it would not be removed in Impala 4. Current
> > users of beeswax would need to start migrating to HS2.
> >
> > Proposal 5: Sentry
> >
> > Drop support for Sentry in favor of Ranger.
> >
> > Proposal 6: Metadata
> >
> > Metadata V2 will become the default. Metadata V1 will be deprecated.
> >
> > Thanks,
> > Joe
> >
>

Re: Impala 4.0 breaking changes

Posted by Tim Armstrong <ta...@cloudera.com>.
I think I generally support this. A few specific comments.

> Proposal 3: Impala-lzo
> Drop support for Impala-lzo/hadoop-lzo

Does this mean dropping the plugin text scanner interface entirely? LZO is
the only implementation of that that I'm aware of (and we rely on it to
test the interface) so seems reasonable to me to remove something that has
minimal adoption and not cleanly separated from the scanner implementation
of core Impala.

> Proposal 5: Sentry
> Drop support for Sentry in favor of Ranger.

I think moving this direction makes a lot of sense given that activity in
the Sentry project has declined a lot (just look at the activity level on
the two projects, it's dramatically different), unless someone in the
community wants to step up and maintain the integration.

> Proposal 6: Metadata
> Metadata V2 will become the default. Metadata V1 will be deprecated.
Maybe we should set a goal of removing the support in Impala 4.1 or 4.2?
That would allow us to remove a lot of complex code

On Mon, Mar 16, 2020 at 10:07 AM Joe McDonnell <jo...@cloudera.com>
wrote:

> Now that Impala 3.4 is branched and master is Impala 4.0, we need to decide
> what breaking changes will happen in Impala 4.0. I have provided a series
> of proposals below. I welcome feedback on them. Other proposals are also
> welcome.
>
> Thanks,
> Joe
>
> Proposal 0: Hadoop component versions
>
> Switch to CDP versions of components by default. This means that Impala
> will use Hive 3+ (which is already essentially Hive 4 and may change names
> to being Hive 4).
> Remove support for CDH versions of components.
> This was already discussed in the original thread for Impala 4, so this is
> not new.
>
> Proposal 1: OS support
>
> Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> Centos 7 development will be focused on newer Centos 7 versions such as 7.6
> and 7.7.
> Add support for Centos 8
> Move main development from Ubuntu 16 to Ubuntu 18 over time.
>
> Proposal 2: Python support
>
> Drop support for Python 2.6
> Add support for Python 3 over time.
>
> Proposal 3: Impala-lzo
>
> Drop support for Impala-lzo/hadoop-lzo
>
> Proposal 4: Clients
>
> Deprecate beeswax protocol. This means that it can be removed in the next
> major version number, but it would not be removed in Impala 4. Current
> users of beeswax would need to start migrating to HS2.
>
> Proposal 5: Sentry
>
> Drop support for Sentry in favor of Ranger.
>
> Proposal 6: Metadata
>
> Metadata V2 will become the default. Metadata V1 will be deprecated.
>
> Thanks,
> Joe
>

Re: Impala 4.0 breaking changes

Posted by Tim Armstrong <ta...@cloudera.com>.
I marked that as a blocker. Did you have a specific approach in mind? E.g.
changing the behaviour, having it controlled by a flag, etc?

On Sat, May 30, 2020 at 7:52 PM Shant Hovsepian <sh...@superdupershant.com>
wrote:

> Here's another one regarding support of ordinals in HAVING clauses.
>
>
> https://github.com/apache/impala/blame/master/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java#L54
>
> https://issues.apache.org/jira/browse/IMPALA-7844
>
> -Shant
>
>
> On Fri, May 8, 2020 at 10:35 AM Sahil Takiar <ta...@gmail.com>
> wrote:
>
> > Another aspect is that ACID-inserts are probably faster, especially on
> > object stores like S3.
> >
> >
> > Note that
> >
> >
> https://impala.apache.org/docs/build/html/topics/impala_s3_skip_insert_staging.html
> > allows
> > for direct-writes to S3 (no staging directory). Although this does not
> work
> > for insert overwrite queries.
> >
> > On Fri, May 8, 2020 at 1:44 AM Zoltán Borók-Nagy <bo...@apache.org>
> > wrote:
> >
> > > About transactional tables:
> > > If there's an ACID base directory in the table (due to compaction or
> > INSERT
> > > OVERWRITE), then files at table/partition-root level will be ignored.
> > > So in that case Spark would need to do ACID-aware inserts.
> > >
> > > Another aspect is that ACID-inserts are probably faster, especially on
> > > object stores like S3.
> > > The reason for this is that we don't need to create a staging directory
> > and
> > > move (which is a copy on S3) files to their final location.
> > > However, read amplification is definitely greater for ACID tables.
> > >
> > > Btw, do we want to achieve consistent default behavior with an upstream
> > > Hive version?
> > >
> > > That said, I think creating non-transactional tables is a good default.
> > > Especially because Impala will probably support Hudi and Iceberg in the
> > > future, so it's probably better to let the users choose explicitly.
> > >
> > > - Zoltan
> > >
> > >
> > > On Thu, May 7, 2020 at 11:46 PM Tim Armstrong <tarmstrong@cloudera.com
> >
> > > wrote:
> > >
> > > > That's a pretty good argument against defaulting to transactional
> > tables.
> > > > You are right that it doesn't work out-of-the box with most other
> > > engines -
> > > > writing files into the base directory of the table/partition will not
> > > work
> > > > as intended afaik.
> > > >
> > > > On Thu, May 7, 2020 at 1:10 PM Shant Hovsepian <sh...@cloudera.com>
> > > wrote:
> > > >
> > > > > How compatible with other engines is the insert only transaction
> > type.
> > > > >
> > > > > Very often data is loaded with spark, especially for cases with
> > complex
> > > > > types where it's the only option. Will landing parquet files in the
> > > table
> > > > > path just work even if we don't get consistent inserts or does
> spark
> > > need
> > > > > to be aware of the table format in either case?
> > > > >
> > > > > -Shant
> > > > >
> > > > > On Thu, May 7, 2020 at 3:09 PM Sahil Takiar <
> takiar.sahil@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1 on query results spooling, I've been thinking about enabling
> it
> > by
> > > > > > default recently since it seems to be relatively stable.
> > > > > >
> > > > > > On Thu, May 7, 2020 at 11:41 AM Tim Armstrong <
> > > tarmstrong@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I'm going to revive this thread. I thought of a few more
> defaults
> > > > that
> > > > > we
> > > > > > > might want to change. These are default changes we (putting on
> > > > Cloudera
> > > > > > hat
> > > > > > > temporarily) have made for some new production deployments and
> > have
> > > > > been
> > > > > > > happy with.
> > > > > > >
> > > > > > > Query result spooling has a bunch of advantages for resource
> > > > > consumption
> > > > > > > and fetch speed. It uses a bounded amount of memory and scratch
> > > > space,
> > > > > > but
> > > > > > > I think it's overall a better default. We've been using it in
> > > > > production
> > > > > > > for a while now and haven't had any issues.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_spool_query_results.html
> > > > > > >
> > > > > > > I think we should also switch the default file format to
> parquet,
> > > > > because
> > > > > > > it's more correct (default text has some issues with escaping)
> > and
> > > > > > because
> > > > > > > it's more performant.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_file_format.html
> > > > > > >
> > > > > > > We could also consider creating insert_only transactional
> tables
> > by
> > > > > > default
> > > > > > > -
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_transactional_type.html
> > > > > > > .
> > > > > > > The pros and cons here are more complex - we get more
> consistent
> > > > > > behaviour
> > > > > > > by default, but there can be perf/scalability consequences.
> > > > > > >
> > > > > > > Any objections or thoughts on these?
> > > > > > >
> > > > > > > On Thu, Mar 19, 2020 at 4:44 PM Tim Armstrong <
> > > > tarmstrong@cloudera.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I think ARM support can ship in whatever release it's reading
> > in,
> > > > > since
> > > > > > > > it's not a breaking change.
> > > > > > > >
> > > > > > > > On Wed, Mar 18, 2020 at 9:43 PM 赵 仁海 <zhaorenhai@hotmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks
> > > > > > > >> I will work hard on this ^_^
> > > > > > > >>
> > > > > > > >> ________________________________
> > > > > > > >> 发件人: Jim Apple <ap...@jbapple.com>
> > > > > > > >> 发送时间: 2020年3月19日 10:21
> > > > > > > >> 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > > > > > >> 主题: Re: Impala 4.0 breaking changes
> > > > > > > >>
> > > > > > > >> I agree. I don’t know how far we are from having arm64
> > support,
> > > > > > though,
> > > > > > > >> and
> > > > > > > >> we might not get there for a 4.0 release, I’d guess. But
> that
> > > > > doesn’t
> > > > > > > mean
> > > > > > > >> it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or
> > > whatever.
> > > > > > > >>
> > > > > > > >> On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <
> > > > > > > joemcdonnell@cloudera.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Patches to add support for arm64 are definitely welcome in
> > any
> > > > > > > release.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Joe
> > > > > > > >> >
> > > > > > > >> > On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <
> > zhaorenhai@hotmail.com>
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi
> > > > > > > >> > >
> > > > > > > >> > > Could we  add support for arm64?
> > > > > > > >> > >
> > > > > > > >> > > Thanks
> > > > > > > >> > > Zhao Renhai
> > > > > > > >> > >
> > > > > > > >> > > ________________________________
> > > > > > > >> > > 发件人: Joe McDonnell <jo...@cloudera.com>
> > > > > > > >> > > 发送时间: 2020年3月17日 1:07
> > > > > > > >> > > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > > > > > >> > > 主题: Impala 4.0 breaking changes
> > > > > > > >> > >
> > > > > > > >> > > Now that Impala 3.4 is branched and master is Impala
> 4.0,
> > we
> > > > > need
> > > > > > to
> > > > > > > >> > decide
> > > > > > > >> > > what breaking changes will happen in Impala 4.0. I have
> > > > > provided a
> > > > > > > >> series
> > > > > > > >> > > of proposals below. I welcome feedback on them. Other
> > > > proposals
> > > > > > are
> > > > > > > >> also
> > > > > > > >> > > welcome.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Joe
> > > > > > > >> > >
> > > > > > > >> > > Proposal 0: Hadoop component versions
> > > > > > > >> > >
> > > > > > > >> > > Switch to CDP versions of components by default. This
> > means
> > > > that
> > > > > > > >> Impala
> > > > > > > >> > > will use Hive 3+ (which is already essentially Hive 4
> and
> > > may
> > > > > > change
> > > > > > > >> > names
> > > > > > > >> > > to being Hive 4).
> > > > > > > >> > > Remove support for CDH versions of components.
> > > > > > > >> > > This was already discussed in the original thread for
> > Impala
> > > > 4,
> > > > > so
> > > > > > > >> this
> > > > > > > >> > is
> > > > > > > >> > > not new.
> > > > > > > >> > >
> > > > > > > >> > > Proposal 1: OS support
> > > > > > > >> > >
> > > > > > > >> > > Drop support for Centos 6, Ubuntu 14, and Debian (all
> > > > versions)
> > > > > > > >> > > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and
> > SLES
> > > 12
> > > > > > > >> > > Centos 7 development will be focused on newer Centos 7
> > > > versions
> > > > > > such
> > > > > > > >> as
> > > > > > > >> > 7.6
> > > > > > > >> > > and 7.7.
> > > > > > > >> > > Add support for Centos 8
> > > > > > > >> > > Move main development from Ubuntu 16 to Ubuntu 18 over
> > time.
> > > > > > > >> > >
> > > > > > > >> > > Proposal 2: Python support
> > > > > > > >> > >
> > > > > > > >> > > Drop support for Python 2.6
> > > > > > > >> > > Add support for Python 3 over time.
> > > > > > > >> > >
> > > > > > > >> > > Proposal 3: Impala-lzo
> > > > > > > >> > >
> > > > > > > >> > > Drop support for Impala-lzo/hadoop-lzo
> > > > > > > >> > >
> > > > > > > >> > > Proposal 4: Clients
> > > > > > > >> > >
> > > > > > > >> > > Deprecate beeswax protocol. This means that it can be
> > > removed
> > > > in
> > > > > > the
> > > > > > > >> next
> > > > > > > >> > > major version number, but it would not be removed in
> > Impala
> > > 4.
> > > > > > > Current
> > > > > > > >> > > users of beeswax would need to start migrating to HS2.
> > > > > > > >> > >
> > > > > > > >> > > Proposal 5: Sentry
> > > > > > > >> > >
> > > > > > > >> > > Drop support for Sentry in favor of Ranger.
> > > > > > > >> > >
> > > > > > > >> > > Proposal 6: Metadata
> > > > > > > >> > >
> > > > > > > >> > > Metadata V2 will become the default. Metadata V1 will be
> > > > > > deprecated.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Joe
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sahil Takiar
> > > > > > Software Engineer
> > > > > > takiar.sahil@gmail.com | (510) 673-0309
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Sahil Takiar
> > Software Engineer
> > takiar.sahil@gmail.com | (510) 673-0309
> >
>

Re: Impala 4.0 breaking changes

Posted by Shant Hovsepian <sh...@superdupershant.com>.
Here's another one regarding support of ordinals in HAVING clauses.

https://github.com/apache/impala/blame/master/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java#L54

https://issues.apache.org/jira/browse/IMPALA-7844

-Shant


On Fri, May 8, 2020 at 10:35 AM Sahil Takiar <ta...@gmail.com> wrote:

> Another aspect is that ACID-inserts are probably faster, especially on
> object stores like S3.
>
>
> Note that
>
> https://impala.apache.org/docs/build/html/topics/impala_s3_skip_insert_staging.html
> allows
> for direct-writes to S3 (no staging directory). Although this does not work
> for insert overwrite queries.
>
> On Fri, May 8, 2020 at 1:44 AM Zoltán Borók-Nagy <bo...@apache.org>
> wrote:
>
> > About transactional tables:
> > If there's an ACID base directory in the table (due to compaction or
> INSERT
> > OVERWRITE), then files at table/partition-root level will be ignored.
> > So in that case Spark would need to do ACID-aware inserts.
> >
> > Another aspect is that ACID-inserts are probably faster, especially on
> > object stores like S3.
> > The reason for this is that we don't need to create a staging directory
> and
> > move (which is a copy on S3) files to their final location.
> > However, read amplification is definitely greater for ACID tables.
> >
> > Btw, do we want to achieve consistent default behavior with an upstream
> > Hive version?
> >
> > That said, I think creating non-transactional tables is a good default.
> > Especially because Impala will probably support Hudi and Iceberg in the
> > future, so it's probably better to let the users choose explicitly.
> >
> > - Zoltan
> >
> >
> > On Thu, May 7, 2020 at 11:46 PM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > That's a pretty good argument against defaulting to transactional
> tables.
> > > You are right that it doesn't work out-of-the box with most other
> > engines -
> > > writing files into the base directory of the table/partition will not
> > work
> > > as intended afaik.
> > >
> > > On Thu, May 7, 2020 at 1:10 PM Shant Hovsepian <sh...@cloudera.com>
> > wrote:
> > >
> > > > How compatible with other engines is the insert only transaction
> type.
> > > >
> > > > Very often data is loaded with spark, especially for cases with
> complex
> > > > types where it's the only option. Will landing parquet files in the
> > table
> > > > path just work even if we don't get consistent inserts or does spark
> > need
> > > > to be aware of the table format in either case?
> > > >
> > > > -Shant
> > > >
> > > > On Thu, May 7, 2020 at 3:09 PM Sahil Takiar <ta...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 on query results spooling, I've been thinking about enabling it
> by
> > > > > default recently since it seems to be relatively stable.
> > > > >
> > > > > On Thu, May 7, 2020 at 11:41 AM Tim Armstrong <
> > tarmstrong@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > I'm going to revive this thread. I thought of a few more defaults
> > > that
> > > > we
> > > > > > might want to change. These are default changes we (putting on
> > > Cloudera
> > > > > hat
> > > > > > temporarily) have made for some new production deployments and
> have
> > > > been
> > > > > > happy with.
> > > > > >
> > > > > > Query result spooling has a bunch of advantages for resource
> > > > consumption
> > > > > > and fetch speed. It uses a bounded amount of memory and scratch
> > > space,
> > > > > but
> > > > > > I think it's overall a better default. We've been using it in
> > > > production
> > > > > > for a while now and haven't had any issues.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_spool_query_results.html
> > > > > >
> > > > > > I think we should also switch the default file format to parquet,
> > > > because
> > > > > > it's more correct (default text has some issues with escaping)
> and
> > > > > because
> > > > > > it's more performant.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_file_format.html
> > > > > >
> > > > > > We could also consider creating insert_only transactional tables
> by
> > > > > default
> > > > > > -
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_transactional_type.html
> > > > > > .
> > > > > > The pros and cons here are more complex - we get more consistent
> > > > > behaviour
> > > > > > by default, but there can be perf/scalability consequences.
> > > > > >
> > > > > > Any objections or thoughts on these?
> > > > > >
> > > > > > On Thu, Mar 19, 2020 at 4:44 PM Tim Armstrong <
> > > tarmstrong@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I think ARM support can ship in whatever release it's reading
> in,
> > > > since
> > > > > > > it's not a breaking change.
> > > > > > >
> > > > > > > On Wed, Mar 18, 2020 at 9:43 PM 赵 仁海 <zh...@hotmail.com>
> > > wrote:
> > > > > > >
> > > > > > >> Thanks
> > > > > > >> I will work hard on this ^_^
> > > > > > >>
> > > > > > >> ________________________________
> > > > > > >> 发件人: Jim Apple <ap...@jbapple.com>
> > > > > > >> 发送时间: 2020年3月19日 10:21
> > > > > > >> 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > > > > >> 主题: Re: Impala 4.0 breaking changes
> > > > > > >>
> > > > > > >> I agree. I don’t know how far we are from having arm64
> support,
> > > > > though,
> > > > > > >> and
> > > > > > >> we might not get there for a 4.0 release, I’d guess. But that
> > > > doesn’t
> > > > > > mean
> > > > > > >> it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or
> > whatever.
> > > > > > >>
> > > > > > >> On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <
> > > > > > joemcdonnell@cloudera.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Patches to add support for arm64 are definitely welcome in
> any
> > > > > > release.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Joe
> > > > > > >> >
> > > > > > >> > On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <
> zhaorenhai@hotmail.com>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Hi
> > > > > > >> > >
> > > > > > >> > > Could we  add support for arm64?
> > > > > > >> > >
> > > > > > >> > > Thanks
> > > > > > >> > > Zhao Renhai
> > > > > > >> > >
> > > > > > >> > > ________________________________
> > > > > > >> > > 发件人: Joe McDonnell <jo...@cloudera.com>
> > > > > > >> > > 发送时间: 2020年3月17日 1:07
> > > > > > >> > > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > > > > >> > > 主题: Impala 4.0 breaking changes
> > > > > > >> > >
> > > > > > >> > > Now that Impala 3.4 is branched and master is Impala 4.0,
> we
> > > > need
> > > > > to
> > > > > > >> > decide
> > > > > > >> > > what breaking changes will happen in Impala 4.0. I have
> > > > provided a
> > > > > > >> series
> > > > > > >> > > of proposals below. I welcome feedback on them. Other
> > > proposals
> > > > > are
> > > > > > >> also
> > > > > > >> > > welcome.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Joe
> > > > > > >> > >
> > > > > > >> > > Proposal 0: Hadoop component versions
> > > > > > >> > >
> > > > > > >> > > Switch to CDP versions of components by default. This
> means
> > > that
> > > > > > >> Impala
> > > > > > >> > > will use Hive 3+ (which is already essentially Hive 4 and
> > may
> > > > > change
> > > > > > >> > names
> > > > > > >> > > to being Hive 4).
> > > > > > >> > > Remove support for CDH versions of components.
> > > > > > >> > > This was already discussed in the original thread for
> Impala
> > > 4,
> > > > so
> > > > > > >> this
> > > > > > >> > is
> > > > > > >> > > not new.
> > > > > > >> > >
> > > > > > >> > > Proposal 1: OS support
> > > > > > >> > >
> > > > > > >> > > Drop support for Centos 6, Ubuntu 14, and Debian (all
> > > versions)
> > > > > > >> > > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and
> SLES
> > 12
> > > > > > >> > > Centos 7 development will be focused on newer Centos 7
> > > versions
> > > > > such
> > > > > > >> as
> > > > > > >> > 7.6
> > > > > > >> > > and 7.7.
> > > > > > >> > > Add support for Centos 8
> > > > > > >> > > Move main development from Ubuntu 16 to Ubuntu 18 over
> time.
> > > > > > >> > >
> > > > > > >> > > Proposal 2: Python support
> > > > > > >> > >
> > > > > > >> > > Drop support for Python 2.6
> > > > > > >> > > Add support for Python 3 over time.
> > > > > > >> > >
> > > > > > >> > > Proposal 3: Impala-lzo
> > > > > > >> > >
> > > > > > >> > > Drop support for Impala-lzo/hadoop-lzo
> > > > > > >> > >
> > > > > > >> > > Proposal 4: Clients
> > > > > > >> > >
> > > > > > >> > > Deprecate beeswax protocol. This means that it can be
> > removed
> > > in
> > > > > the
> > > > > > >> next
> > > > > > >> > > major version number, but it would not be removed in
> Impala
> > 4.
> > > > > > Current
> > > > > > >> > > users of beeswax would need to start migrating to HS2.
> > > > > > >> > >
> > > > > > >> > > Proposal 5: Sentry
> > > > > > >> > >
> > > > > > >> > > Drop support for Sentry in favor of Ranger.
> > > > > > >> > >
> > > > > > >> > > Proposal 6: Metadata
> > > > > > >> > >
> > > > > > >> > > Metadata V2 will become the default. Metadata V1 will be
> > > > > deprecated.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Joe
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sahil Takiar
> > > > > Software Engineer
> > > > > takiar.sahil@gmail.com | (510) 673-0309
> > > > >
> > > >
> > >
> >
>
>
> --
> Sahil Takiar
> Software Engineer
> takiar.sahil@gmail.com | (510) 673-0309
>

Re: Impala 4.0 breaking changes

Posted by Sahil Takiar <ta...@gmail.com>.
Another aspect is that ACID-inserts are probably faster, especially on
object stores like S3.


Note that
https://impala.apache.org/docs/build/html/topics/impala_s3_skip_insert_staging.html
allows
for direct-writes to S3 (no staging directory). Although this does not work
for insert overwrite queries.

On Fri, May 8, 2020 at 1:44 AM Zoltán Borók-Nagy <bo...@apache.org>
wrote:

> About transactional tables:
> If there's an ACID base directory in the table (due to compaction or INSERT
> OVERWRITE), then files at table/partition-root level will be ignored.
> So in that case Spark would need to do ACID-aware inserts.
>
> Another aspect is that ACID-inserts are probably faster, especially on
> object stores like S3.
> The reason for this is that we don't need to create a staging directory and
> move (which is a copy on S3) files to their final location.
> However, read amplification is definitely greater for ACID tables.
>
> Btw, do we want to achieve consistent default behavior with an upstream
> Hive version?
>
> That said, I think creating non-transactional tables is a good default.
> Especially because Impala will probably support Hudi and Iceberg in the
> future, so it's probably better to let the users choose explicitly.
>
> - Zoltan
>
>
> On Thu, May 7, 2020 at 11:46 PM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> > That's a pretty good argument against defaulting to transactional tables.
> > You are right that it doesn't work out-of-the box with most other
> engines -
> > writing files into the base directory of the table/partition will not
> work
> > as intended afaik.
> >
> > On Thu, May 7, 2020 at 1:10 PM Shant Hovsepian <sh...@cloudera.com>
> wrote:
> >
> > > How compatible with other engines is the insert only transaction type.
> > >
> > > Very often data is loaded with spark, especially for cases with complex
> > > types where it's the only option. Will landing parquet files in the
> table
> > > path just work even if we don't get consistent inserts or does spark
> need
> > > to be aware of the table format in either case?
> > >
> > > -Shant
> > >
> > > On Thu, May 7, 2020 at 3:09 PM Sahil Takiar <ta...@gmail.com>
> > > wrote:
> > >
> > > > +1 on query results spooling, I've been thinking about enabling it by
> > > > default recently since it seems to be relatively stable.
> > > >
> > > > On Thu, May 7, 2020 at 11:41 AM Tim Armstrong <
> tarmstrong@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > I'm going to revive this thread. I thought of a few more defaults
> > that
> > > we
> > > > > might want to change. These are default changes we (putting on
> > Cloudera
> > > > hat
> > > > > temporarily) have made for some new production deployments and have
> > > been
> > > > > happy with.
> > > > >
> > > > > Query result spooling has a bunch of advantages for resource
> > > consumption
> > > > > and fetch speed. It uses a bounded amount of memory and scratch
> > space,
> > > > but
> > > > > I think it's overall a better default. We've been using it in
> > > production
> > > > > for a while now and haven't had any issues.
> > > > >
> > > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_spool_query_results.html
> > > > >
> > > > > I think we should also switch the default file format to parquet,
> > > because
> > > > > it's more correct (default text has some issues with escaping) and
> > > > because
> > > > > it's more performant.
> > > > >
> > > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_file_format.html
> > > > >
> > > > > We could also consider creating insert_only transactional tables by
> > > > default
> > > > > -
> > > > >
> > > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_transactional_type.html
> > > > > .
> > > > > The pros and cons here are more complex - we get more consistent
> > > > behaviour
> > > > > by default, but there can be perf/scalability consequences.
> > > > >
> > > > > Any objections or thoughts on these?
> > > > >
> > > > > On Thu, Mar 19, 2020 at 4:44 PM Tim Armstrong <
> > tarmstrong@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > I think ARM support can ship in whatever release it's reading in,
> > > since
> > > > > > it's not a breaking change.
> > > > > >
> > > > > > On Wed, Mar 18, 2020 at 9:43 PM 赵 仁海 <zh...@hotmail.com>
> > wrote:
> > > > > >
> > > > > >> Thanks
> > > > > >> I will work hard on this ^_^
> > > > > >>
> > > > > >> ________________________________
> > > > > >> 发件人: Jim Apple <ap...@jbapple.com>
> > > > > >> 发送时间: 2020年3月19日 10:21
> > > > > >> 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > > > >> 主题: Re: Impala 4.0 breaking changes
> > > > > >>
> > > > > >> I agree. I don’t know how far we are from having arm64 support,
> > > > though,
> > > > > >> and
> > > > > >> we might not get there for a 4.0 release, I’d guess. But that
> > > doesn’t
> > > > > mean
> > > > > >> it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or
> whatever.
> > > > > >>
> > > > > >> On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <
> > > > > joemcdonnell@cloudera.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Patches to add support for arm64 are definitely welcome in any
> > > > > release.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Joe
> > > > > >> >
> > > > > >> > On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com>
> > > > wrote:
> > > > > >> >
> > > > > >> > > Hi
> > > > > >> > >
> > > > > >> > > Could we  add support for arm64?
> > > > > >> > >
> > > > > >> > > Thanks
> > > > > >> > > Zhao Renhai
> > > > > >> > >
> > > > > >> > > ________________________________
> > > > > >> > > 发件人: Joe McDonnell <jo...@cloudera.com>
> > > > > >> > > 发送时间: 2020年3月17日 1:07
> > > > > >> > > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > > > >> > > 主题: Impala 4.0 breaking changes
> > > > > >> > >
> > > > > >> > > Now that Impala 3.4 is branched and master is Impala 4.0, we
> > > need
> > > > to
> > > > > >> > decide
> > > > > >> > > what breaking changes will happen in Impala 4.0. I have
> > > provided a
> > > > > >> series
> > > > > >> > > of proposals below. I welcome feedback on them. Other
> > proposals
> > > > are
> > > > > >> also
> > > > > >> > > welcome.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Joe
> > > > > >> > >
> > > > > >> > > Proposal 0: Hadoop component versions
> > > > > >> > >
> > > > > >> > > Switch to CDP versions of components by default. This means
> > that
> > > > > >> Impala
> > > > > >> > > will use Hive 3+ (which is already essentially Hive 4 and
> may
> > > > change
> > > > > >> > names
> > > > > >> > > to being Hive 4).
> > > > > >> > > Remove support for CDH versions of components.
> > > > > >> > > This was already discussed in the original thread for Impala
> > 4,
> > > so
> > > > > >> this
> > > > > >> > is
> > > > > >> > > not new.
> > > > > >> > >
> > > > > >> > > Proposal 1: OS support
> > > > > >> > >
> > > > > >> > > Drop support for Centos 6, Ubuntu 14, and Debian (all
> > versions)
> > > > > >> > > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES
> 12
> > > > > >> > > Centos 7 development will be focused on newer Centos 7
> > versions
> > > > such
> > > > > >> as
> > > > > >> > 7.6
> > > > > >> > > and 7.7.
> > > > > >> > > Add support for Centos 8
> > > > > >> > > Move main development from Ubuntu 16 to Ubuntu 18 over time.
> > > > > >> > >
> > > > > >> > > Proposal 2: Python support
> > > > > >> > >
> > > > > >> > > Drop support for Python 2.6
> > > > > >> > > Add support for Python 3 over time.
> > > > > >> > >
> > > > > >> > > Proposal 3: Impala-lzo
> > > > > >> > >
> > > > > >> > > Drop support for Impala-lzo/hadoop-lzo
> > > > > >> > >
> > > > > >> > > Proposal 4: Clients
> > > > > >> > >
> > > > > >> > > Deprecate beeswax protocol. This means that it can be
> removed
> > in
> > > > the
> > > > > >> next
> > > > > >> > > major version number, but it would not be removed in Impala
> 4.
> > > > > Current
> > > > > >> > > users of beeswax would need to start migrating to HS2.
> > > > > >> > >
> > > > > >> > > Proposal 5: Sentry
> > > > > >> > >
> > > > > >> > > Drop support for Sentry in favor of Ranger.
> > > > > >> > >
> > > > > >> > > Proposal 6: Metadata
> > > > > >> > >
> > > > > >> > > Metadata V2 will become the default. Metadata V1 will be
> > > > deprecated.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Joe
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Sahil Takiar
> > > > Software Engineer
> > > > takiar.sahil@gmail.com | (510) 673-0309
> > > >
> > >
> >
>


-- 
Sahil Takiar
Software Engineer
takiar.sahil@gmail.com | (510) 673-0309

Re: Impala 4.0 breaking changes

Posted by Zoltán Borók-Nagy <bo...@apache.org>.
About transactional tables:
If there's an ACID base directory in the table (due to compaction or INSERT
OVERWRITE), then files at table/partition-root level will be ignored.
So in that case Spark would need to do ACID-aware inserts.

Another aspect is that ACID-inserts are probably faster, especially on
object stores like S3.
The reason for this is that we don't need to create a staging directory and
move (which is a copy on S3) files to their final location.
However, read amplification is definitely greater for ACID tables.

Btw, do we want to achieve consistent default behavior with an upstream
Hive version?

That said, I think creating non-transactional tables is a good default.
Especially because Impala will probably support Hudi and Iceberg in the
future, so it's probably better to let the users choose explicitly.

- Zoltan


On Thu, May 7, 2020 at 11:46 PM Tim Armstrong <ta...@cloudera.com>
wrote:

> That's a pretty good argument against defaulting to transactional tables.
> You are right that it doesn't work out-of-the box with most other engines -
> writing files into the base directory of the table/partition will not work
> as intended afaik.
>
> On Thu, May 7, 2020 at 1:10 PM Shant Hovsepian <sh...@cloudera.com> wrote:
>
> > How compatible with other engines is the insert only transaction type.
> >
> > Very often data is loaded with spark, especially for cases with complex
> > types where it's the only option. Will landing parquet files in the table
> > path just work even if we don't get consistent inserts or does spark need
> > to be aware of the table format in either case?
> >
> > -Shant
> >
> > On Thu, May 7, 2020 at 3:09 PM Sahil Takiar <ta...@gmail.com>
> > wrote:
> >
> > > +1 on query results spooling, I've been thinking about enabling it by
> > > default recently since it seems to be relatively stable.
> > >
> > > On Thu, May 7, 2020 at 11:41 AM Tim Armstrong <tarmstrong@cloudera.com
> >
> > > wrote:
> > >
> > > > I'm going to revive this thread. I thought of a few more defaults
> that
> > we
> > > > might want to change. These are default changes we (putting on
> Cloudera
> > > hat
> > > > temporarily) have made for some new production deployments and have
> > been
> > > > happy with.
> > > >
> > > > Query result spooling has a bunch of advantages for resource
> > consumption
> > > > and fetch speed. It uses a bounded amount of memory and scratch
> space,
> > > but
> > > > I think it's overall a better default. We've been using it in
> > production
> > > > for a while now and haven't had any issues.
> > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_spool_query_results.html
> > > >
> > > > I think we should also switch the default file format to parquet,
> > because
> > > > it's more correct (default text has some issues with escaping) and
> > > because
> > > > it's more performant.
> > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_file_format.html
> > > >
> > > > We could also consider creating insert_only transactional tables by
> > > default
> > > > -
> > > >
> > > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_transactional_type.html
> > > > .
> > > > The pros and cons here are more complex - we get more consistent
> > > behaviour
> > > > by default, but there can be perf/scalability consequences.
> > > >
> > > > Any objections or thoughts on these?
> > > >
> > > > On Thu, Mar 19, 2020 at 4:44 PM Tim Armstrong <
> tarmstrong@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > I think ARM support can ship in whatever release it's reading in,
> > since
> > > > > it's not a breaking change.
> > > > >
> > > > > On Wed, Mar 18, 2020 at 9:43 PM 赵 仁海 <zh...@hotmail.com>
> wrote:
> > > > >
> > > > >> Thanks
> > > > >> I will work hard on this ^_^
> > > > >>
> > > > >> ________________________________
> > > > >> 发件人: Jim Apple <ap...@jbapple.com>
> > > > >> 发送时间: 2020年3月19日 10:21
> > > > >> 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > > >> 主题: Re: Impala 4.0 breaking changes
> > > > >>
> > > > >> I agree. I don’t know how far we are from having arm64 support,
> > > though,
> > > > >> and
> > > > >> we might not get there for a 4.0 release, I’d guess. But that
> > doesn’t
> > > > mean
> > > > >> it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or whatever.
> > > > >>
> > > > >> On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <
> > > > joemcdonnell@cloudera.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Patches to add support for arm64 are definitely welcome in any
> > > > release.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Joe
> > > > >> >
> > > > >> > On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com>
> > > wrote:
> > > > >> >
> > > > >> > > Hi
> > > > >> > >
> > > > >> > > Could we  add support for arm64?
> > > > >> > >
> > > > >> > > Thanks
> > > > >> > > Zhao Renhai
> > > > >> > >
> > > > >> > > ________________________________
> > > > >> > > 发件人: Joe McDonnell <jo...@cloudera.com>
> > > > >> > > 发送时间: 2020年3月17日 1:07
> > > > >> > > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > > >> > > 主题: Impala 4.0 breaking changes
> > > > >> > >
> > > > >> > > Now that Impala 3.4 is branched and master is Impala 4.0, we
> > need
> > > to
> > > > >> > decide
> > > > >> > > what breaking changes will happen in Impala 4.0. I have
> > provided a
> > > > >> series
> > > > >> > > of proposals below. I welcome feedback on them. Other
> proposals
> > > are
> > > > >> also
> > > > >> > > welcome.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Joe
> > > > >> > >
> > > > >> > > Proposal 0: Hadoop component versions
> > > > >> > >
> > > > >> > > Switch to CDP versions of components by default. This means
> that
> > > > >> Impala
> > > > >> > > will use Hive 3+ (which is already essentially Hive 4 and may
> > > change
> > > > >> > names
> > > > >> > > to being Hive 4).
> > > > >> > > Remove support for CDH versions of components.
> > > > >> > > This was already discussed in the original thread for Impala
> 4,
> > so
> > > > >> this
> > > > >> > is
> > > > >> > > not new.
> > > > >> > >
> > > > >> > > Proposal 1: OS support
> > > > >> > >
> > > > >> > > Drop support for Centos 6, Ubuntu 14, and Debian (all
> versions)
> > > > >> > > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> > > > >> > > Centos 7 development will be focused on newer Centos 7
> versions
> > > such
> > > > >> as
> > > > >> > 7.6
> > > > >> > > and 7.7.
> > > > >> > > Add support for Centos 8
> > > > >> > > Move main development from Ubuntu 16 to Ubuntu 18 over time.
> > > > >> > >
> > > > >> > > Proposal 2: Python support
> > > > >> > >
> > > > >> > > Drop support for Python 2.6
> > > > >> > > Add support for Python 3 over time.
> > > > >> > >
> > > > >> > > Proposal 3: Impala-lzo
> > > > >> > >
> > > > >> > > Drop support for Impala-lzo/hadoop-lzo
> > > > >> > >
> > > > >> > > Proposal 4: Clients
> > > > >> > >
> > > > >> > > Deprecate beeswax protocol. This means that it can be removed
> in
> > > the
> > > > >> next
> > > > >> > > major version number, but it would not be removed in Impala 4.
> > > > Current
> > > > >> > > users of beeswax would need to start migrating to HS2.
> > > > >> > >
> > > > >> > > Proposal 5: Sentry
> > > > >> > >
> > > > >> > > Drop support for Sentry in favor of Ranger.
> > > > >> > >
> > > > >> > > Proposal 6: Metadata
> > > > >> > >
> > > > >> > > Metadata V2 will become the default. Metadata V1 will be
> > > deprecated.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Joe
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > Sahil Takiar
> > > Software Engineer
> > > takiar.sahil@gmail.com | (510) 673-0309
> > >
> >
>

Re: Impala 4.0 breaking changes

Posted by Tim Armstrong <ta...@cloudera.com>.
That's a pretty good argument against defaulting to transactional tables.
You are right that it doesn't work out-of-the box with most other engines -
writing files into the base directory of the table/partition will not work
as intended afaik.

On Thu, May 7, 2020 at 1:10 PM Shant Hovsepian <sh...@cloudera.com> wrote:

> How compatible with other engines is the insert only transaction type.
>
> Very often data is loaded with spark, especially for cases with complex
> types where it's the only option. Will landing parquet files in the table
> path just work even if we don't get consistent inserts or does spark need
> to be aware of the table format in either case?
>
> -Shant
>
> On Thu, May 7, 2020 at 3:09 PM Sahil Takiar <ta...@gmail.com>
> wrote:
>
> > +1 on query results spooling, I've been thinking about enabling it by
> > default recently since it seems to be relatively stable.
> >
> > On Thu, May 7, 2020 at 11:41 AM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > I'm going to revive this thread. I thought of a few more defaults that
> we
> > > might want to change. These are default changes we (putting on Cloudera
> > hat
> > > temporarily) have made for some new production deployments and have
> been
> > > happy with.
> > >
> > > Query result spooling has a bunch of advantages for resource
> consumption
> > > and fetch speed. It uses a bounded amount of memory and scratch space,
> > but
> > > I think it's overall a better default. We've been using it in
> production
> > > for a while now and haven't had any issues.
> > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_spool_query_results.html
> > >
> > > I think we should also switch the default file format to parquet,
> because
> > > it's more correct (default text has some issues with escaping) and
> > because
> > > it's more performant.
> > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_file_format.html
> > >
> > > We could also consider creating insert_only transactional tables by
> > default
> > > -
> > >
> > >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_transactional_type.html
> > > .
> > > The pros and cons here are more complex - we get more consistent
> > behaviour
> > > by default, but there can be perf/scalability consequences.
> > >
> > > Any objections or thoughts on these?
> > >
> > > On Thu, Mar 19, 2020 at 4:44 PM Tim Armstrong <tarmstrong@cloudera.com
> >
> > > wrote:
> > >
> > > > I think ARM support can ship in whatever release it's reading in,
> since
> > > > it's not a breaking change.
> > > >
> > > > On Wed, Mar 18, 2020 at 9:43 PM 赵 仁海 <zh...@hotmail.com> wrote:
> > > >
> > > >> Thanks
> > > >> I will work hard on this ^_^
> > > >>
> > > >> ________________________________
> > > >> 发件人: Jim Apple <ap...@jbapple.com>
> > > >> 发送时间: 2020年3月19日 10:21
> > > >> 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > >> 主题: Re: Impala 4.0 breaking changes
> > > >>
> > > >> I agree. I don’t know how far we are from having arm64 support,
> > though,
> > > >> and
> > > >> we might not get there for a 4.0 release, I’d guess. But that
> doesn’t
> > > mean
> > > >> it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or whatever.
> > > >>
> > > >> On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <
> > > joemcdonnell@cloudera.com>
> > > >> wrote:
> > > >>
> > > >> > Patches to add support for arm64 are definitely welcome in any
> > > release.
> > > >> >
> > > >> > Thanks,
> > > >> > Joe
> > > >> >
> > > >> > On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com>
> > wrote:
> > > >> >
> > > >> > > Hi
> > > >> > >
> > > >> > > Could we  add support for arm64?
> > > >> > >
> > > >> > > Thanks
> > > >> > > Zhao Renhai
> > > >> > >
> > > >> > > ________________________________
> > > >> > > 发件人: Joe McDonnell <jo...@cloudera.com>
> > > >> > > 发送时间: 2020年3月17日 1:07
> > > >> > > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > >> > > 主题: Impala 4.0 breaking changes
> > > >> > >
> > > >> > > Now that Impala 3.4 is branched and master is Impala 4.0, we
> need
> > to
> > > >> > decide
> > > >> > > what breaking changes will happen in Impala 4.0. I have
> provided a
> > > >> series
> > > >> > > of proposals below. I welcome feedback on them. Other proposals
> > are
> > > >> also
> > > >> > > welcome.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Joe
> > > >> > >
> > > >> > > Proposal 0: Hadoop component versions
> > > >> > >
> > > >> > > Switch to CDP versions of components by default. This means that
> > > >> Impala
> > > >> > > will use Hive 3+ (which is already essentially Hive 4 and may
> > change
> > > >> > names
> > > >> > > to being Hive 4).
> > > >> > > Remove support for CDH versions of components.
> > > >> > > This was already discussed in the original thread for Impala 4,
> so
> > > >> this
> > > >> > is
> > > >> > > not new.
> > > >> > >
> > > >> > > Proposal 1: OS support
> > > >> > >
> > > >> > > Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> > > >> > > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> > > >> > > Centos 7 development will be focused on newer Centos 7 versions
> > such
> > > >> as
> > > >> > 7.6
> > > >> > > and 7.7.
> > > >> > > Add support for Centos 8
> > > >> > > Move main development from Ubuntu 16 to Ubuntu 18 over time.
> > > >> > >
> > > >> > > Proposal 2: Python support
> > > >> > >
> > > >> > > Drop support for Python 2.6
> > > >> > > Add support for Python 3 over time.
> > > >> > >
> > > >> > > Proposal 3: Impala-lzo
> > > >> > >
> > > >> > > Drop support for Impala-lzo/hadoop-lzo
> > > >> > >
> > > >> > > Proposal 4: Clients
> > > >> > >
> > > >> > > Deprecate beeswax protocol. This means that it can be removed in
> > the
> > > >> next
> > > >> > > major version number, but it would not be removed in Impala 4.
> > > Current
> > > >> > > users of beeswax would need to start migrating to HS2.
> > > >> > >
> > > >> > > Proposal 5: Sentry
> > > >> > >
> > > >> > > Drop support for Sentry in favor of Ranger.
> > > >> > >
> > > >> > > Proposal 6: Metadata
> > > >> > >
> > > >> > > Metadata V2 will become the default. Metadata V1 will be
> > deprecated.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Joe
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
> >
> > --
> > Sahil Takiar
> > Software Engineer
> > takiar.sahil@gmail.com | (510) 673-0309
> >
>

Re: Impala 4.0 breaking changes

Posted by Shant Hovsepian <sh...@cloudera.com>.
How compatible with other engines is the insert only transaction type.

Very often data is loaded with spark, especially for cases with complex
types where it's the only option. Will landing parquet files in the table
path just work even if we don't get consistent inserts or does spark need
to be aware of the table format in either case?

-Shant

On Thu, May 7, 2020 at 3:09 PM Sahil Takiar <ta...@gmail.com> wrote:

> +1 on query results spooling, I've been thinking about enabling it by
> default recently since it seems to be relatively stable.
>
> On Thu, May 7, 2020 at 11:41 AM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> > I'm going to revive this thread. I thought of a few more defaults that we
> > might want to change. These are default changes we (putting on Cloudera
> hat
> > temporarily) have made for some new production deployments and have been
> > happy with.
> >
> > Query result spooling has a bunch of advantages for resource consumption
> > and fetch speed. It uses a bounded amount of memory and scratch space,
> but
> > I think it's overall a better default. We've been using it in production
> > for a while now and haven't had any issues.
> >
> >
> https://impala.apache.org/docs/build/html/topics/impala_spool_query_results.html
> >
> > I think we should also switch the default file format to parquet, because
> > it's more correct (default text has some issues with escaping) and
> because
> > it's more performant.
> >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_file_format.html
> >
> > We could also consider creating insert_only transactional tables by
> default
> > -
> >
> >
> https://impala.apache.org/docs/build/html/topics/impala_default_transactional_type.html
> > .
> > The pros and cons here are more complex - we get more consistent
> behaviour
> > by default, but there can be perf/scalability consequences.
> >
> > Any objections or thoughts on these?
> >
> > On Thu, Mar 19, 2020 at 4:44 PM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > I think ARM support can ship in whatever release it's reading in, since
> > > it's not a breaking change.
> > >
> > > On Wed, Mar 18, 2020 at 9:43 PM 赵 仁海 <zh...@hotmail.com> wrote:
> > >
> > >> Thanks
> > >> I will work hard on this ^_^
> > >>
> > >> ________________________________
> > >> 发件人: Jim Apple <ap...@jbapple.com>
> > >> 发送时间: 2020年3月19日 10:21
> > >> 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > >> 主题: Re: Impala 4.0 breaking changes
> > >>
> > >> I agree. I don’t know how far we are from having arm64 support,
> though,
> > >> and
> > >> we might not get there for a 4.0 release, I’d guess. But that doesn’t
> > mean
> > >> it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or whatever.
> > >>
> > >> On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <
> > joemcdonnell@cloudera.com>
> > >> wrote:
> > >>
> > >> > Patches to add support for arm64 are definitely welcome in any
> > release.
> > >> >
> > >> > Thanks,
> > >> > Joe
> > >> >
> > >> > On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com>
> wrote:
> > >> >
> > >> > > Hi
> > >> > >
> > >> > > Could we  add support for arm64?
> > >> > >
> > >> > > Thanks
> > >> > > Zhao Renhai
> > >> > >
> > >> > > ________________________________
> > >> > > 发件人: Joe McDonnell <jo...@cloudera.com>
> > >> > > 发送时间: 2020年3月17日 1:07
> > >> > > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > >> > > 主题: Impala 4.0 breaking changes
> > >> > >
> > >> > > Now that Impala 3.4 is branched and master is Impala 4.0, we need
> to
> > >> > decide
> > >> > > what breaking changes will happen in Impala 4.0. I have provided a
> > >> series
> > >> > > of proposals below. I welcome feedback on them. Other proposals
> are
> > >> also
> > >> > > welcome.
> > >> > >
> > >> > > Thanks,
> > >> > > Joe
> > >> > >
> > >> > > Proposal 0: Hadoop component versions
> > >> > >
> > >> > > Switch to CDP versions of components by default. This means that
> > >> Impala
> > >> > > will use Hive 3+ (which is already essentially Hive 4 and may
> change
> > >> > names
> > >> > > to being Hive 4).
> > >> > > Remove support for CDH versions of components.
> > >> > > This was already discussed in the original thread for Impala 4, so
> > >> this
> > >> > is
> > >> > > not new.
> > >> > >
> > >> > > Proposal 1: OS support
> > >> > >
> > >> > > Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> > >> > > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> > >> > > Centos 7 development will be focused on newer Centos 7 versions
> such
> > >> as
> > >> > 7.6
> > >> > > and 7.7.
> > >> > > Add support for Centos 8
> > >> > > Move main development from Ubuntu 16 to Ubuntu 18 over time.
> > >> > >
> > >> > > Proposal 2: Python support
> > >> > >
> > >> > > Drop support for Python 2.6
> > >> > > Add support for Python 3 over time.
> > >> > >
> > >> > > Proposal 3: Impala-lzo
> > >> > >
> > >> > > Drop support for Impala-lzo/hadoop-lzo
> > >> > >
> > >> > > Proposal 4: Clients
> > >> > >
> > >> > > Deprecate beeswax protocol. This means that it can be removed in
> the
> > >> next
> > >> > > major version number, but it would not be removed in Impala 4.
> > Current
> > >> > > users of beeswax would need to start migrating to HS2.
> > >> > >
> > >> > > Proposal 5: Sentry
> > >> > >
> > >> > > Drop support for Sentry in favor of Ranger.
> > >> > >
> > >> > > Proposal 6: Metadata
> > >> > >
> > >> > > Metadata V2 will become the default. Metadata V1 will be
> deprecated.
> > >> > >
> > >> > > Thanks,
> > >> > > Joe
> > >> > >
> > >> >
> > >>
> > >
> >
>
>
> --
> Sahil Takiar
> Software Engineer
> takiar.sahil@gmail.com | (510) 673-0309
>

Re: Impala 4.0 breaking changes

Posted by Sahil Takiar <ta...@gmail.com>.
+1 on query results spooling, I've been thinking about enabling it by
default recently since it seems to be relatively stable.

On Thu, May 7, 2020 at 11:41 AM Tim Armstrong <ta...@cloudera.com>
wrote:

> I'm going to revive this thread. I thought of a few more defaults that we
> might want to change. These are default changes we (putting on Cloudera hat
> temporarily) have made for some new production deployments and have been
> happy with.
>
> Query result spooling has a bunch of advantages for resource consumption
> and fetch speed. It uses a bounded amount of memory and scratch space, but
> I think it's overall a better default. We've been using it in production
> for a while now and haven't had any issues.
>
> https://impala.apache.org/docs/build/html/topics/impala_spool_query_results.html
>
> I think we should also switch the default file format to parquet, because
> it's more correct (default text has some issues with escaping) and because
> it's more performant.
>
> https://impala.apache.org/docs/build/html/topics/impala_default_file_format.html
>
> We could also consider creating insert_only transactional tables by default
> -
>
> https://impala.apache.org/docs/build/html/topics/impala_default_transactional_type.html
> .
> The pros and cons here are more complex - we get more consistent behaviour
> by default, but there can be perf/scalability consequences.
>
> Any objections or thoughts on these?
>
> On Thu, Mar 19, 2020 at 4:44 PM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> > I think ARM support can ship in whatever release it's reading in, since
> > it's not a breaking change.
> >
> > On Wed, Mar 18, 2020 at 9:43 PM 赵 仁海 <zh...@hotmail.com> wrote:
> >
> >> Thanks
> >> I will work hard on this ^_^
> >>
> >> ________________________________
> >> 发件人: Jim Apple <ap...@jbapple.com>
> >> 发送时间: 2020年3月19日 10:21
> >> 收件人: dev@impala.apache.org <de...@impala.apache.org>
> >> 主题: Re: Impala 4.0 breaking changes
> >>
> >> I agree. I don’t know how far we are from having arm64 support, though,
> >> and
> >> we might not get there for a 4.0 release, I’d guess. But that doesn’t
> mean
> >> it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or whatever.
> >>
> >> On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <
> joemcdonnell@cloudera.com>
> >> wrote:
> >>
> >> > Patches to add support for arm64 are definitely welcome in any
> release.
> >> >
> >> > Thanks,
> >> > Joe
> >> >
> >> > On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com> wrote:
> >> >
> >> > > Hi
> >> > >
> >> > > Could we  add support for arm64?
> >> > >
> >> > > Thanks
> >> > > Zhao Renhai
> >> > >
> >> > > ________________________________
> >> > > 发件人: Joe McDonnell <jo...@cloudera.com>
> >> > > 发送时间: 2020年3月17日 1:07
> >> > > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> >> > > 主题: Impala 4.0 breaking changes
> >> > >
> >> > > Now that Impala 3.4 is branched and master is Impala 4.0, we need to
> >> > decide
> >> > > what breaking changes will happen in Impala 4.0. I have provided a
> >> series
> >> > > of proposals below. I welcome feedback on them. Other proposals are
> >> also
> >> > > welcome.
> >> > >
> >> > > Thanks,
> >> > > Joe
> >> > >
> >> > > Proposal 0: Hadoop component versions
> >> > >
> >> > > Switch to CDP versions of components by default. This means that
> >> Impala
> >> > > will use Hive 3+ (which is already essentially Hive 4 and may change
> >> > names
> >> > > to being Hive 4).
> >> > > Remove support for CDH versions of components.
> >> > > This was already discussed in the original thread for Impala 4, so
> >> this
> >> > is
> >> > > not new.
> >> > >
> >> > > Proposal 1: OS support
> >> > >
> >> > > Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> >> > > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> >> > > Centos 7 development will be focused on newer Centos 7 versions such
> >> as
> >> > 7.6
> >> > > and 7.7.
> >> > > Add support for Centos 8
> >> > > Move main development from Ubuntu 16 to Ubuntu 18 over time.
> >> > >
> >> > > Proposal 2: Python support
> >> > >
> >> > > Drop support for Python 2.6
> >> > > Add support for Python 3 over time.
> >> > >
> >> > > Proposal 3: Impala-lzo
> >> > >
> >> > > Drop support for Impala-lzo/hadoop-lzo
> >> > >
> >> > > Proposal 4: Clients
> >> > >
> >> > > Deprecate beeswax protocol. This means that it can be removed in the
> >> next
> >> > > major version number, but it would not be removed in Impala 4.
> Current
> >> > > users of beeswax would need to start migrating to HS2.
> >> > >
> >> > > Proposal 5: Sentry
> >> > >
> >> > > Drop support for Sentry in favor of Ranger.
> >> > >
> >> > > Proposal 6: Metadata
> >> > >
> >> > > Metadata V2 will become the default. Metadata V1 will be deprecated.
> >> > >
> >> > > Thanks,
> >> > > Joe
> >> > >
> >> >
> >>
> >
>


-- 
Sahil Takiar
Software Engineer
takiar.sahil@gmail.com | (510) 673-0309

Re: Impala 4.0 breaking changes

Posted by Tim Armstrong <ta...@cloudera.com>.
I'm going to revive this thread. I thought of a few more defaults that we
might want to change. These are default changes we (putting on Cloudera hat
temporarily) have made for some new production deployments and have been
happy with.

Query result spooling has a bunch of advantages for resource consumption
and fetch speed. It uses a bounded amount of memory and scratch space, but
I think it's overall a better default. We've been using it in production
for a while now and haven't had any issues.
https://impala.apache.org/docs/build/html/topics/impala_spool_query_results.html

I think we should also switch the default file format to parquet, because
it's more correct (default text has some issues with escaping) and because
it's more performant.
https://impala.apache.org/docs/build/html/topics/impala_default_file_format.html

We could also consider creating insert_only transactional tables by default
-
https://impala.apache.org/docs/build/html/topics/impala_default_transactional_type.html.
The pros and cons here are more complex - we get more consistent behaviour
by default, but there can be perf/scalability consequences.

Any objections or thoughts on these?

On Thu, Mar 19, 2020 at 4:44 PM Tim Armstrong <ta...@cloudera.com>
wrote:

> I think ARM support can ship in whatever release it's reading in, since
> it's not a breaking change.
>
> On Wed, Mar 18, 2020 at 9:43 PM 赵 仁海 <zh...@hotmail.com> wrote:
>
>> Thanks
>> I will work hard on this ^_^
>>
>> ________________________________
>> 发件人: Jim Apple <ap...@jbapple.com>
>> 发送时间: 2020年3月19日 10:21
>> 收件人: dev@impala.apache.org <de...@impala.apache.org>
>> 主题: Re: Impala 4.0 breaking changes
>>
>> I agree. I don’t know how far we are from having arm64 support, though,
>> and
>> we might not get there for a 4.0 release, I’d guess. But that doesn’t mean
>> it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or whatever.
>>
>> On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <jo...@cloudera.com>
>> wrote:
>>
>> > Patches to add support for arm64 are definitely welcome in any release.
>> >
>> > Thanks,
>> > Joe
>> >
>> > On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com> wrote:
>> >
>> > > Hi
>> > >
>> > > Could we  add support for arm64?
>> > >
>> > > Thanks
>> > > Zhao Renhai
>> > >
>> > > ________________________________
>> > > 发件人: Joe McDonnell <jo...@cloudera.com>
>> > > 发送时间: 2020年3月17日 1:07
>> > > 收件人: dev@impala.apache.org <de...@impala.apache.org>
>> > > 主题: Impala 4.0 breaking changes
>> > >
>> > > Now that Impala 3.4 is branched and master is Impala 4.0, we need to
>> > decide
>> > > what breaking changes will happen in Impala 4.0. I have provided a
>> series
>> > > of proposals below. I welcome feedback on them. Other proposals are
>> also
>> > > welcome.
>> > >
>> > > Thanks,
>> > > Joe
>> > >
>> > > Proposal 0: Hadoop component versions
>> > >
>> > > Switch to CDP versions of components by default. This means that
>> Impala
>> > > will use Hive 3+ (which is already essentially Hive 4 and may change
>> > names
>> > > to being Hive 4).
>> > > Remove support for CDH versions of components.
>> > > This was already discussed in the original thread for Impala 4, so
>> this
>> > is
>> > > not new.
>> > >
>> > > Proposal 1: OS support
>> > >
>> > > Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
>> > > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
>> > > Centos 7 development will be focused on newer Centos 7 versions such
>> as
>> > 7.6
>> > > and 7.7.
>> > > Add support for Centos 8
>> > > Move main development from Ubuntu 16 to Ubuntu 18 over time.
>> > >
>> > > Proposal 2: Python support
>> > >
>> > > Drop support for Python 2.6
>> > > Add support for Python 3 over time.
>> > >
>> > > Proposal 3: Impala-lzo
>> > >
>> > > Drop support for Impala-lzo/hadoop-lzo
>> > >
>> > > Proposal 4: Clients
>> > >
>> > > Deprecate beeswax protocol. This means that it can be removed in the
>> next
>> > > major version number, but it would not be removed in Impala 4. Current
>> > > users of beeswax would need to start migrating to HS2.
>> > >
>> > > Proposal 5: Sentry
>> > >
>> > > Drop support for Sentry in favor of Ranger.
>> > >
>> > > Proposal 6: Metadata
>> > >
>> > > Metadata V2 will become the default. Metadata V1 will be deprecated.
>> > >
>> > > Thanks,
>> > > Joe
>> > >
>> >
>>
>

Re: Impala 4.0 breaking changes

Posted by Tim Armstrong <ta...@cloudera.com>.
I think ARM support can ship in whatever release it's reading in, since
it's not a breaking change.

On Wed, Mar 18, 2020 at 9:43 PM 赵 仁海 <zh...@hotmail.com> wrote:

> Thanks
> I will work hard on this ^_^
>
> ________________________________
> 发件人: Jim Apple <ap...@jbapple.com>
> 发送时间: 2020年3月19日 10:21
> 收件人: dev@impala.apache.org <de...@impala.apache.org>
> 主题: Re: Impala 4.0 breaking changes
>
> I agree. I don’t know how far we are from having arm64 support, though, and
> we might not get there for a 4.0 release, I’d guess. But that doesn’t mean
> it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or whatever.
>
> On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <jo...@cloudera.com>
> wrote:
>
> > Patches to add support for arm64 are definitely welcome in any release.
> >
> > Thanks,
> > Joe
> >
> > On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com> wrote:
> >
> > > Hi
> > >
> > > Could we  add support for arm64?
> > >
> > > Thanks
> > > Zhao Renhai
> > >
> > > ________________________________
> > > 发件人: Joe McDonnell <jo...@cloudera.com>
> > > 发送时间: 2020年3月17日 1:07
> > > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > > 主题: Impala 4.0 breaking changes
> > >
> > > Now that Impala 3.4 is branched and master is Impala 4.0, we need to
> > decide
> > > what breaking changes will happen in Impala 4.0. I have provided a
> series
> > > of proposals below. I welcome feedback on them. Other proposals are
> also
> > > welcome.
> > >
> > > Thanks,
> > > Joe
> > >
> > > Proposal 0: Hadoop component versions
> > >
> > > Switch to CDP versions of components by default. This means that Impala
> > > will use Hive 3+ (which is already essentially Hive 4 and may change
> > names
> > > to being Hive 4).
> > > Remove support for CDH versions of components.
> > > This was already discussed in the original thread for Impala 4, so this
> > is
> > > not new.
> > >
> > > Proposal 1: OS support
> > >
> > > Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> > > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> > > Centos 7 development will be focused on newer Centos 7 versions such as
> > 7.6
> > > and 7.7.
> > > Add support for Centos 8
> > > Move main development from Ubuntu 16 to Ubuntu 18 over time.
> > >
> > > Proposal 2: Python support
> > >
> > > Drop support for Python 2.6
> > > Add support for Python 3 over time.
> > >
> > > Proposal 3: Impala-lzo
> > >
> > > Drop support for Impala-lzo/hadoop-lzo
> > >
> > > Proposal 4: Clients
> > >
> > > Deprecate beeswax protocol. This means that it can be removed in the
> next
> > > major version number, but it would not be removed in Impala 4. Current
> > > users of beeswax would need to start migrating to HS2.
> > >
> > > Proposal 5: Sentry
> > >
> > > Drop support for Sentry in favor of Ranger.
> > >
> > > Proposal 6: Metadata
> > >
> > > Metadata V2 will become the default. Metadata V1 will be deprecated.
> > >
> > > Thanks,
> > > Joe
> > >
> >
>

回复: Impala 4.0 breaking changes

Posted by 赵 仁海 <zh...@hotmail.com>.
Thanks
I will work hard on this ^_^

________________________________
发件人: Jim Apple <ap...@jbapple.com>
发送时间: 2020年3月19日 10:21
收件人: dev@impala.apache.org <de...@impala.apache.org>
主题: Re: Impala 4.0 breaking changes

I agree. I don’t know how far we are from having arm64 support, though, and
we might not get there for a 4.0 release, I’d guess. But that doesn’t mean
it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or whatever.

On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <jo...@cloudera.com>
wrote:

> Patches to add support for arm64 are definitely welcome in any release.
>
> Thanks,
> Joe
>
> On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com> wrote:
>
> > Hi
> >
> > Could we  add support for arm64?
> >
> > Thanks
> > Zhao Renhai
> >
> > ________________________________
> > 发件人: Joe McDonnell <jo...@cloudera.com>
> > 发送时间: 2020年3月17日 1:07
> > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > 主题: Impala 4.0 breaking changes
> >
> > Now that Impala 3.4 is branched and master is Impala 4.0, we need to
> decide
> > what breaking changes will happen in Impala 4.0. I have provided a series
> > of proposals below. I welcome feedback on them. Other proposals are also
> > welcome.
> >
> > Thanks,
> > Joe
> >
> > Proposal 0: Hadoop component versions
> >
> > Switch to CDP versions of components by default. This means that Impala
> > will use Hive 3+ (which is already essentially Hive 4 and may change
> names
> > to being Hive 4).
> > Remove support for CDH versions of components.
> > This was already discussed in the original thread for Impala 4, so this
> is
> > not new.
> >
> > Proposal 1: OS support
> >
> > Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> > Centos 7 development will be focused on newer Centos 7 versions such as
> 7.6
> > and 7.7.
> > Add support for Centos 8
> > Move main development from Ubuntu 16 to Ubuntu 18 over time.
> >
> > Proposal 2: Python support
> >
> > Drop support for Python 2.6
> > Add support for Python 3 over time.
> >
> > Proposal 3: Impala-lzo
> >
> > Drop support for Impala-lzo/hadoop-lzo
> >
> > Proposal 4: Clients
> >
> > Deprecate beeswax protocol. This means that it can be removed in the next
> > major version number, but it would not be removed in Impala 4. Current
> > users of beeswax would need to start migrating to HS2.
> >
> > Proposal 5: Sentry
> >
> > Drop support for Sentry in favor of Ranger.
> >
> > Proposal 6: Metadata
> >
> > Metadata V2 will become the default. Metadata V1 will be deprecated.
> >
> > Thanks,
> > Joe
> >
>

Re: Impala 4.0 breaking changes

Posted by Jim Apple <ap...@jbapple.com>.
I agree. I don’t know how far we are from having arm64 support, though, and
we might not get there for a 4.0 release, I’d guess. But that doesn’t mean
it couldn’t arrive by the time for 4.1 or 4.7 or 5.55 or whatever.

On Wed, Mar 18, 2020 at 6:32 PM Joe McDonnell <jo...@cloudera.com>
wrote:

> Patches to add support for arm64 are definitely welcome in any release.
>
> Thanks,
> Joe
>
> On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com> wrote:
>
> > Hi
> >
> > Could we  add support for arm64?
> >
> > Thanks
> > Zhao Renhai
> >
> > ________________________________
> > 发件人: Joe McDonnell <jo...@cloudera.com>
> > 发送时间: 2020年3月17日 1:07
> > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > 主题: Impala 4.0 breaking changes
> >
> > Now that Impala 3.4 is branched and master is Impala 4.0, we need to
> decide
> > what breaking changes will happen in Impala 4.0. I have provided a series
> > of proposals below. I welcome feedback on them. Other proposals are also
> > welcome.
> >
> > Thanks,
> > Joe
> >
> > Proposal 0: Hadoop component versions
> >
> > Switch to CDP versions of components by default. This means that Impala
> > will use Hive 3+ (which is already essentially Hive 4 and may change
> names
> > to being Hive 4).
> > Remove support for CDH versions of components.
> > This was already discussed in the original thread for Impala 4, so this
> is
> > not new.
> >
> > Proposal 1: OS support
> >
> > Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> > Centos 7 development will be focused on newer Centos 7 versions such as
> 7.6
> > and 7.7.
> > Add support for Centos 8
> > Move main development from Ubuntu 16 to Ubuntu 18 over time.
> >
> > Proposal 2: Python support
> >
> > Drop support for Python 2.6
> > Add support for Python 3 over time.
> >
> > Proposal 3: Impala-lzo
> >
> > Drop support for Impala-lzo/hadoop-lzo
> >
> > Proposal 4: Clients
> >
> > Deprecate beeswax protocol. This means that it can be removed in the next
> > major version number, but it would not be removed in Impala 4. Current
> > users of beeswax would need to start migrating to HS2.
> >
> > Proposal 5: Sentry
> >
> > Drop support for Sentry in favor of Ranger.
> >
> > Proposal 6: Metadata
> >
> > Metadata V2 will become the default. Metadata V1 will be deprecated.
> >
> > Thanks,
> > Joe
> >
>

Re: Impala 4.0 breaking changes

Posted by Joe McDonnell <jo...@cloudera.com>.
Patches to add support for arm64 are definitely welcome in any release.

Thanks,
Joe

On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com> wrote:

> Hi
>
> Could we  add support for arm64?
>
> Thanks
> Zhao Renhai
>
> ________________________________
> 发件人: Joe McDonnell <jo...@cloudera.com>
> 发送时间: 2020年3月17日 1:07
> 收件人: dev@impala.apache.org <de...@impala.apache.org>
> 主题: Impala 4.0 breaking changes
>
> Now that Impala 3.4 is branched and master is Impala 4.0, we need to decide
> what breaking changes will happen in Impala 4.0. I have provided a series
> of proposals below. I welcome feedback on them. Other proposals are also
> welcome.
>
> Thanks,
> Joe
>
> Proposal 0: Hadoop component versions
>
> Switch to CDP versions of components by default. This means that Impala
> will use Hive 3+ (which is already essentially Hive 4 and may change names
> to being Hive 4).
> Remove support for CDH versions of components.
> This was already discussed in the original thread for Impala 4, so this is
> not new.
>
> Proposal 1: OS support
>
> Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> Centos 7 development will be focused on newer Centos 7 versions such as 7.6
> and 7.7.
> Add support for Centos 8
> Move main development from Ubuntu 16 to Ubuntu 18 over time.
>
> Proposal 2: Python support
>
> Drop support for Python 2.6
> Add support for Python 3 over time.
>
> Proposal 3: Impala-lzo
>
> Drop support for Impala-lzo/hadoop-lzo
>
> Proposal 4: Clients
>
> Deprecate beeswax protocol. This means that it can be removed in the next
> major version number, but it would not be removed in Impala 4. Current
> users of beeswax would need to start migrating to HS2.
>
> Proposal 5: Sentry
>
> Drop support for Sentry in favor of Ranger.
>
> Proposal 6: Metadata
>
> Metadata V2 will become the default. Metadata V1 will be deprecated.
>
> Thanks,
> Joe
>

Re: Impala 4.0 breaking changes

Posted by Laszlo Gaal <la...@cloudera.com>.
Also, in parallel with Joe's item 1 (dropping support for Ubuntu 14.04,
Debian-all and CentOS 6)
I would also propose dropping these platforms from the native-toolchain
project.

  - LaszloG

On Tue, Mar 17, 2020 at 6:41 PM Sahil Takiar <ta...@gmail.com> wrote:

> A few additional suggestions for breaking changes:
>
>    - Do we still need the DECIMAL_V2 query option? Seems like this has been
>    true for a while. Maybe we can add it to the list of deprecated flags?
>    - Deprecate support for ADLS, since it has effectively been replaced by
>    ABFS
>    - Deprecate (or even remove) support for HDFS cacheing? Not sure how
>    extensively this is used, removing the code would be nice as it
> simplifies
>    part of the HDFS read path
>
>
> On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com> wrote:
>
> > Hi
> >
> > Could we  add support for arm64?
> >
> > Thanks
> > Zhao Renhai
> >
> > ________________________________
> > 发件人: Joe McDonnell <jo...@cloudera.com>
> > 发送时间: 2020年3月17日 1:07
> > 收件人: dev@impala.apache.org <de...@impala.apache.org>
> > 主题: Impala 4.0 breaking changes
> >
> > Now that Impala 3.4 is branched and master is Impala 4.0, we need to
> decide
> > what breaking changes will happen in Impala 4.0. I have provided a series
> > of proposals below. I welcome feedback on them. Other proposals are also
> > welcome.
> >
> > Thanks,
> > Joe
> >
> > Proposal 0: Hadoop component versions
> >
> > Switch to CDP versions of components by default. This means that Impala
> > will use Hive 3+ (which is already essentially Hive 4 and may change
> names
> > to being Hive 4).
> > Remove support for CDH versions of components.
> > This was already discussed in the original thread for Impala 4, so this
> is
> > not new.
> >
> > Proposal 1: OS support
> >
> > Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> > Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> > Centos 7 development will be focused on newer Centos 7 versions such as
> 7.6
> > and 7.7.
> > Add support for Centos 8
> > Move main development from Ubuntu 16 to Ubuntu 18 over time.
> >
> > Proposal 2: Python support
> >
> > Drop support for Python 2.6
> > Add support for Python 3 over time.
> >
> > Proposal 3: Impala-lzo
> >
> > Drop support for Impala-lzo/hadoop-lzo
> >
> > Proposal 4: Clients
> >
> > Deprecate beeswax protocol. This means that it can be removed in the next
> > major version number, but it would not be removed in Impala 4. Current
> > users of beeswax would need to start migrating to HS2.
> >
> > Proposal 5: Sentry
> >
> > Drop support for Sentry in favor of Ranger.
> >
> > Proposal 6: Metadata
> >
> > Metadata V2 will become the default. Metadata V1 will be deprecated.
> >
> > Thanks,
> > Joe
> >
>
>
> --
> Sahil Takiar
> Software Engineer
> takiar.sahil@gmail.com | (510) 673-0309
>

Re: Impala 4.0 breaking changes

Posted by Sahil Takiar <ta...@gmail.com>.
A few additional suggestions for breaking changes:

   - Do we still need the DECIMAL_V2 query option? Seems like this has been
   true for a while. Maybe we can add it to the list of deprecated flags?
   - Deprecate support for ADLS, since it has effectively been replaced by
   ABFS
   - Deprecate (or even remove) support for HDFS cacheing? Not sure how
   extensively this is used, removing the code would be nice as it simplifies
   part of the HDFS read path


On Mon, Mar 16, 2020 at 6:11 PM 赵 仁海 <zh...@hotmail.com> wrote:

> Hi
>
> Could we  add support for arm64?
>
> Thanks
> Zhao Renhai
>
> ________________________________
> 发件人: Joe McDonnell <jo...@cloudera.com>
> 发送时间: 2020年3月17日 1:07
> 收件人: dev@impala.apache.org <de...@impala.apache.org>
> 主题: Impala 4.0 breaking changes
>
> Now that Impala 3.4 is branched and master is Impala 4.0, we need to decide
> what breaking changes will happen in Impala 4.0. I have provided a series
> of proposals below. I welcome feedback on them. Other proposals are also
> welcome.
>
> Thanks,
> Joe
>
> Proposal 0: Hadoop component versions
>
> Switch to CDP versions of components by default. This means that Impala
> will use Hive 3+ (which is already essentially Hive 4 and may change names
> to being Hive 4).
> Remove support for CDH versions of components.
> This was already discussed in the original thread for Impala 4, so this is
> not new.
>
> Proposal 1: OS support
>
> Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> Centos 7 development will be focused on newer Centos 7 versions such as 7.6
> and 7.7.
> Add support for Centos 8
> Move main development from Ubuntu 16 to Ubuntu 18 over time.
>
> Proposal 2: Python support
>
> Drop support for Python 2.6
> Add support for Python 3 over time.
>
> Proposal 3: Impala-lzo
>
> Drop support for Impala-lzo/hadoop-lzo
>
> Proposal 4: Clients
>
> Deprecate beeswax protocol. This means that it can be removed in the next
> major version number, but it would not be removed in Impala 4. Current
> users of beeswax would need to start migrating to HS2.
>
> Proposal 5: Sentry
>
> Drop support for Sentry in favor of Ranger.
>
> Proposal 6: Metadata
>
> Metadata V2 will become the default. Metadata V1 will be deprecated.
>
> Thanks,
> Joe
>


-- 
Sahil Takiar
Software Engineer
takiar.sahil@gmail.com | (510) 673-0309

回复: Impala 4.0 breaking changes

Posted by 赵 仁海 <zh...@hotmail.com>.
Hi

Could we  add support for arm64?

Thanks
Zhao Renhai

________________________________
发件人: Joe McDonnell <jo...@cloudera.com>
发送时间: 2020年3月17日 1:07
收件人: dev@impala.apache.org <de...@impala.apache.org>
主题: Impala 4.0 breaking changes

Now that Impala 3.4 is branched and master is Impala 4.0, we need to decide
what breaking changes will happen in Impala 4.0. I have provided a series
of proposals below. I welcome feedback on them. Other proposals are also
welcome.

Thanks,
Joe

Proposal 0: Hadoop component versions

Switch to CDP versions of components by default. This means that Impala
will use Hive 3+ (which is already essentially Hive 4 and may change names
to being Hive 4).
Remove support for CDH versions of components.
This was already discussed in the original thread for Impala 4, so this is
not new.

Proposal 1: OS support

Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
Centos 7 development will be focused on newer Centos 7 versions such as 7.6
and 7.7.
Add support for Centos 8
Move main development from Ubuntu 16 to Ubuntu 18 over time.

Proposal 2: Python support

Drop support for Python 2.6
Add support for Python 3 over time.

Proposal 3: Impala-lzo

Drop support for Impala-lzo/hadoop-lzo

Proposal 4: Clients

Deprecate beeswax protocol. This means that it can be removed in the next
major version number, but it would not be removed in Impala 4. Current
users of beeswax would need to start migrating to HS2.

Proposal 5: Sentry

Drop support for Sentry in favor of Ranger.

Proposal 6: Metadata

Metadata V2 will become the default. Metadata V1 will be deprecated.

Thanks,
Joe

Re: Impala 4.0 breaking changes

Posted by Shant Hovsepian <sh...@cloudera.com>.
Looking forward to Proposal 3! Getting lzo out of the core tests would help
solve most of my random test data loading issues.

On Mon, Mar 16, 2020 at 1:07 PM Joe McDonnell <jo...@cloudera.com>
wrote:

> Now that Impala 3.4 is branched and master is Impala 4.0, we need to decide
> what breaking changes will happen in Impala 4.0. I have provided a series
> of proposals below. I welcome feedback on them. Other proposals are also
> welcome.
>
> Thanks,
> Joe
>
> Proposal 0: Hadoop component versions
>
> Switch to CDP versions of components by default. This means that Impala
> will use Hive 3+ (which is already essentially Hive 4 and may change names
> to being Hive 4).
> Remove support for CDH versions of components.
> This was already discussed in the original thread for Impala 4, so this is
> not new.
>
> Proposal 1: OS support
>
> Drop support for Centos 6, Ubuntu 14, and Debian (all versions)
> Retain support for Ubuntu 16, Ubuntu 18, Centos 7, and SLES 12
> Centos 7 development will be focused on newer Centos 7 versions such as 7.6
> and 7.7.
> Add support for Centos 8
> Move main development from Ubuntu 16 to Ubuntu 18 over time.
>
> Proposal 2: Python support
>
> Drop support for Python 2.6
> Add support for Python 3 over time.
>
> Proposal 3: Impala-lzo
>
> Drop support for Impala-lzo/hadoop-lzo
>
> Proposal 4: Clients
>
> Deprecate beeswax protocol. This means that it can be removed in the next
> major version number, but it would not be removed in Impala 4. Current
> users of beeswax would need to start migrating to HS2.
>
> Proposal 5: Sentry
>
> Drop support for Sentry in favor of Ranger.
>
> Proposal 6: Metadata
>
> Metadata V2 will become the default. Metadata V1 will be deprecated.
>
> Thanks,
> Joe
>