Query rewrite (reflections)

[Boris Tyukin <bo...@boristyukin.com>]

Hi guys,

I am curious if there are any discussions regarding the support of a query
rewrite feature in Impala. For example, Apache Dremio provides this cool
feature, which arguably can provide tremendous performance gains:
http://docs.dremio.com/acceleration/reflections.html

Basically, the idea is to have a few pre-aggregated tables and let Impala
pick the most efficient copy of the table during query time. I guess it is
a spin on an old "materialized view" idea, but the difference here that
users will query a single table without knowing about pre-aggregated
versions of that same table.

Re: Query rewrite (reflections)

[Boris Tyukin <bo...@boristyukin.com>]

thanks Andrew! no question it would be quite a project. We were looking at
AtScale and Dremio and I really liked the idea how these produces can
benefit from this feature, taking performance to a sub-second territory
even for very large tables.

On Wed, Oct 30, 2019 at 5:54 PM Andrew Sherman <as...@cloudera.com>
wrote:

> Hi Boris,
> I think materialized views are an area that everyone agrees that Impala
> needs to make progress on.
> There are no Jiras as yet for this.
> As you probably know, doing materialized view rewrites is a complex
> process during planning (similar to index selection) so this would require
> a significant amount of planner work
>
> -Andrew
>
> On Tue, Oct 15, 2019 at 7:47 AM Boris Tyukin <bo...@boristyukin.com>
> wrote:
>
>> Hi guys,
>>
>> I am curious if there are any discussions regarding the support of a
>> query rewrite feature in Impala. For example, Apache Dremio provides this
>> cool feature, which arguably can provide tremendous performance gains:
>> http://docs.dremio.com/acceleration/reflections.html
>>
>> Basically, the idea is to have a few pre-aggregated tables and let Impala
>> pick the most efficient copy of the table during query time. I guess it is
>> a spin on an old "materialized view" idea, but the difference here that
>> users will query a single table without knowing about pre-aggregated
>> versions of that same table.
>>
>>
>

Re: Query rewrite (reflections)

[Andrew Sherman <as...@cloudera.com>]

Hi Boris,
I think materialized views are an area that everyone agrees that Impala
needs to make progress on.
There are no Jiras as yet for this.
As you probably know, doing materialized view rewrites is a complex process
during planning (similar to index selection) so this would require a
significant amount of planner work

-Andrew

On Tue, Oct 15, 2019 at 7:47 AM Boris Tyukin <bo...@boristyukin.com> wrote:

> Hi guys,
>
> I am curious if there are any discussions regarding the support of a query
> rewrite feature in Impala. For example, Apache Dremio provides this cool
> feature, which arguably can provide tremendous performance gains:
> http://docs.dremio.com/acceleration/reflections.html
>
> Basically, the idea is to have a few pre-aggregated tables and let Impala
> pick the most efficient copy of the table during query time. I guess it is
> a spin on an old "materialized view" idea, but the difference here that
> users will query a single table without knowing about pre-aggregated
> versions of that same table.
>
>