You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Michael Mior <mm...@uwaterloo.ca> on 2016/02/24 22:40:04 UTC
Materialized view rewriting
Is there any documentation anywhere on how the current implementation of
query rewriting for materialized views work? Mostly I'm referring
to MaterializedViewSubstitutionVisitor. There's a lot of code to digest
with not a lot of documentation and it would be helpful to have a reference
to refer. Thanks!
Cheers,
--
Michael Mior
mmior@uwaterloo.ca
Re: Materialized view rewriting
Posted by Michael Mior <mm...@uwaterloo.ca>.
I expected as much re: code documentation. Totally understandable, but I
figured it couldn't hurt to check.
As for Cassandra and materialized views, that's exactly what I was thinking
and the reason I opened the issue about updates to support Cassandra 3.x :)
Calcite's automatic rewriting seems to work as expected when the
materializations are manually defined. I'd like to see if I can get the
adapter to automatically pull in materialized view definitions from
Cassandra. Since CQL and SQL are so similar, I'm hoping it won't be too
much of a challenge to have Calcite use the CQL definitions of the
materialized views.
--
Michael Mior
mmior@uwaterloo.ca
2016-02-24 17:22 GMT-05:00 Julian Hyde <jh...@apache.org>:
> By the way, interesting that you are interested in Cassandra and
> materialized views. Cassandra announced materialized view support
> recently[1] but they solved only half of a problem (not an
> insignificant half, I hasten to add), namely materialized view
> maintenance. They don't transparently substitute them into the query -
> you have to reference the materialized view explicitly in y our query
> - so in my book they've not delivered materialized view support. If
> you're planning to deliver REAL materialized view support to Cassandra
> that would be awesome.
>
> Julian
>
> [1]
> http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views
>
>
> On Wed, Feb 24, 2016 at 2:17 PM, Julian Hyde <jh...@apache.org> wrote:
> > As is typical for complex pieces of code like this, the documentation
> > is in the code (and the unit test). It's probably not what you wanted
> > to hear, but the code mutates quite fast and so if we'd written a
> > design doc a few months ago it would be partially inaccurate.
> >
> > I, Maryann Xue and Amogh Margoor are the main authors of this code.
> >
> > Suggest you find a relevant test case in MaterializationTest (or write
> > a new one) and run it with trace enabled and/or in a debugger. You
> > will see the process of matching an expression to a MV bottom up if
> > you watch each call to UnifyRule.unify.
> >
> > Julian
> >
> >
> > On Wed, Feb 24, 2016 at 1:40 PM, Michael Mior <mm...@uwaterloo.ca>
> wrote:
> >> Is there any documentation anywhere on how the current implementation of
> >> query rewriting for materialized views work? Mostly I'm referring
> >> to MaterializedViewSubstitutionVisitor. There's a lot of code to digest
> >> with not a lot of documentation and it would be helpful to have a
> reference
> >> to refer. Thanks!
> >>
> >> Cheers,
> >> --
> >> Michael Mior
> >> mmior@uwaterloo.ca
>
Re: Materialized view rewriting
Posted by Julian Hyde <jh...@apache.org>.
Rajat,
I’ll second what Maryann says. Please chime in on https://issues.apache.org/jira/browse/CALCITE-1101 <https://issues.apache.org/jira/browse/CALCITE-1101>. If it doesn’t make things easier for Quark we probably shouldn’t be doing it.
Julian
> On Feb 26, 2016, at 8:39 AM, Maryann Xue <ma...@gmail.com> wrote:
>
> Thank you for pointing out another way of defining materializations, Rajat!
>
> We had some discussion again about this interface, and Julian opened a JIRA
> https://issues.apache.org/jira/browse/CALCITE-1101.
>
> The main problems are:
>
> 1. The life cycle of the materializations. In Phoenix (we don't know about
> other projects, so welcome more comments), materializations are used to
> model secondary indices, which are in fact another type of Phoenix tables.
> So materializations for Phoenix should have exactly the same life cycle as
> its enclosed PhoenixSchema, which is a snapshot of all current table
> definitions as of the timestamp for a specific JDBC statement.
>
> 2. Right now for calling defineMaterialization method, we need to take a
> whole lot of trouble to get the CalciteSchema object which in fact should
> be internal to Calcite code.
>
> 3. The right point of time for defining/creating those materialized views.
> Whether for Quark or Phoenix, we need to make sure that we call
> defineMaterialization at the exact right point of time, which is after the
> schema is loaded and before the planner tries to collect and use them.
> Again this had better be something taken care of by Calcite instead of
> carefully maintained by the users.
>
>
> Let's follow up on that JIRA though.
>
>
>
> Thanks,
> Maryann
>
> On Fri, Feb 26, 2016 at 5:50 AM, Rajat Venkatesh <rv...@qubole.com>
> wrote:
>
>> In Quark, dont use hooks to define materializations. We use a tablefactory
>> [1] to defer until the schema is loaded[2].
>>
>> 1.
>>
>> https://github.com/qubole/quark/blob/master/optimizer/src/main/java/com/qubole/quark/planner/MetadataSchema.java#L85
>> 2.
>>
>> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/materialize/MaterializationService.java#L132
>>
>> Hopefully, I've understood your problem correctly.
>>
>>
>> On Thu, Feb 25, 2016 at 10:16 PM Maryann Xue <ma...@gmail.com>
>> wrote:
>>
>>> Hi Michael,
>>>
>>> We had a little difficulty defining our secondary indices as materialized
>>> views with our Schema SPI implementation too, and this made the code
>> pretty
>>> hacky. In order to call that defineMaterialization method, we hold the
>>> parent SchemaPlus object within our own Schema impl object so that we can
>>> get its own corresponding SchemaPlus object by calling "parentSchema
>>> .getSubSchema(this.name)" later on. We do this after the schema/table
>>> resolving phase (that is when the entire schema tree incl. your own
>> schema
>>> objects have been initiated) and call defineMaterialization for each
>>> secondary index under each subSchema. We add a hook in "Hook.TRIMMED" for
>>> this, which sounds pretty weird, but this is exactly a point after you
>> have
>>> the whole schema tree ready and before the materializations are asked for
>>> by the planner.
>>>
>>> Anyway, I do hope the interface can be modified to avoid all this
>> trouble.
>>>
>>>
>>> Thanks,
>>> Maryann
>>>
>>> On Thu, Feb 25, 2016 at 9:24 AM, Michael Mior <mm...@uwaterloo.ca>
>> wrote:
>>>
>>>> Any suggestions on the best place to hook in and add the materialized
>>>> views? It seems like doing so requires the SchemaPlus object
>>> corresponding
>>>> to the current schema. The current best approach I see is to save a
>>>> reference to the parent schema and then pull out the appropriate
>>> SchemaPlus
>>>> object in getTableMap. This seems like a bit of a hack though.
>>>>
>>>> --
>>>> Michael Mior
>>>> mmior@uwaterloo.ca
>>>>
>>>> 2016-02-24 17:22 GMT-05:00 Julian Hyde <jh...@apache.org>:
>>>>
>>>>> By the way, interesting that you are interested in Cassandra and
>>>>> materialized views. Cassandra announced materialized view support
>>>>> recently[1] but they solved only half of a problem (not an
>>>>> insignificant half, I hasten to add), namely materialized view
>>>>> maintenance. They don't transparently substitute them into the query
>> -
>>>>> you have to reference the materialized view explicitly in y our query
>>>>> - so in my book they've not delivered materialized view support. If
>>>>> you're planning to deliver REAL materialized view support to
>> Cassandra
>>>>> that would be awesome.
>>>>>
>>>>> Julian
>>>>>
>>>>> [1]
>>>>>
>>> http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views
>>>>>
>>>>>
>>>>> On Wed, Feb 24, 2016 at 2:17 PM, Julian Hyde <jh...@apache.org>
>> wrote:
>>>>>> As is typical for complex pieces of code like this, the
>> documentation
>>>>>> is in the code (and the unit test). It's probably not what you
>> wanted
>>>>>> to hear, but the code mutates quite fast and so if we'd written a
>>>>>> design doc a few months ago it would be partially inaccurate.
>>>>>>
>>>>>> I, Maryann Xue and Amogh Margoor are the main authors of this code.
>>>>>>
>>>>>> Suggest you find a relevant test case in MaterializationTest (or
>>> write
>>>>>> a new one) and run it with trace enabled and/or in a debugger. You
>>>>>> will see the process of matching an expression to a MV bottom up if
>>>>>> you watch each call to UnifyRule.unify.
>>>>>>
>>>>>> Julian
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 24, 2016 at 1:40 PM, Michael Mior <mm...@uwaterloo.ca>
>>>>> wrote:
>>>>>>> Is there any documentation anywhere on how the current
>>> implementation
>>>> of
>>>>>>> query rewriting for materialized views work? Mostly I'm referring
>>>>>>> to MaterializedViewSubstitutionVisitor. There's a lot of code to
>>>> digest
>>>>>>> with not a lot of documentation and it would be helpful to have a
>>>>> reference
>>>>>>> to refer. Thanks!
>>>>>>>
>>>>>>> Cheers,
>>>>>>> --
>>>>>>> Michael Mior
>>>>>>> mmior@uwaterloo.ca
>>>>>
>>>>
>>>
>>
Re: Materialized view rewriting
Posted by Maryann Xue <ma...@gmail.com>.
Thank you for pointing out another way of defining materializations, Rajat!
We had some discussion again about this interface, and Julian opened a JIRA
https://issues.apache.org/jira/browse/CALCITE-1101.
The main problems are:
1. The life cycle of the materializations. In Phoenix (we don't know about
other projects, so welcome more comments), materializations are used to
model secondary indices, which are in fact another type of Phoenix tables.
So materializations for Phoenix should have exactly the same life cycle as
its enclosed PhoenixSchema, which is a snapshot of all current table
definitions as of the timestamp for a specific JDBC statement.
2. Right now for calling defineMaterialization method, we need to take a
whole lot of trouble to get the CalciteSchema object which in fact should
be internal to Calcite code.
3. The right point of time for defining/creating those materialized views.
Whether for Quark or Phoenix, we need to make sure that we call
defineMaterialization at the exact right point of time, which is after the
schema is loaded and before the planner tries to collect and use them.
Again this had better be something taken care of by Calcite instead of
carefully maintained by the users.
Let's follow up on that JIRA though.
Thanks,
Maryann
On Fri, Feb 26, 2016 at 5:50 AM, Rajat Venkatesh <rv...@qubole.com>
wrote:
> In Quark, dont use hooks to define materializations. We use a tablefactory
> [1] to defer until the schema is loaded[2].
>
> 1.
>
> https://github.com/qubole/quark/blob/master/optimizer/src/main/java/com/qubole/quark/planner/MetadataSchema.java#L85
> 2.
>
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/materialize/MaterializationService.java#L132
>
> Hopefully, I've understood your problem correctly.
>
>
> On Thu, Feb 25, 2016 at 10:16 PM Maryann Xue <ma...@gmail.com>
> wrote:
>
> > Hi Michael,
> >
> > We had a little difficulty defining our secondary indices as materialized
> > views with our Schema SPI implementation too, and this made the code
> pretty
> > hacky. In order to call that defineMaterialization method, we hold the
> > parent SchemaPlus object within our own Schema impl object so that we can
> > get its own corresponding SchemaPlus object by calling "parentSchema
> > .getSubSchema(this.name)" later on. We do this after the schema/table
> > resolving phase (that is when the entire schema tree incl. your own
> schema
> > objects have been initiated) and call defineMaterialization for each
> > secondary index under each subSchema. We add a hook in "Hook.TRIMMED" for
> > this, which sounds pretty weird, but this is exactly a point after you
> have
> > the whole schema tree ready and before the materializations are asked for
> > by the planner.
> >
> > Anyway, I do hope the interface can be modified to avoid all this
> trouble.
> >
> >
> > Thanks,
> > Maryann
> >
> > On Thu, Feb 25, 2016 at 9:24 AM, Michael Mior <mm...@uwaterloo.ca>
> wrote:
> >
> > > Any suggestions on the best place to hook in and add the materialized
> > > views? It seems like doing so requires the SchemaPlus object
> > corresponding
> > > to the current schema. The current best approach I see is to save a
> > > reference to the parent schema and then pull out the appropriate
> > SchemaPlus
> > > object in getTableMap. This seems like a bit of a hack though.
> > >
> > > --
> > > Michael Mior
> > > mmior@uwaterloo.ca
> > >
> > > 2016-02-24 17:22 GMT-05:00 Julian Hyde <jh...@apache.org>:
> > >
> > > > By the way, interesting that you are interested in Cassandra and
> > > > materialized views. Cassandra announced materialized view support
> > > > recently[1] but they solved only half of a problem (not an
> > > > insignificant half, I hasten to add), namely materialized view
> > > > maintenance. They don't transparently substitute them into the query
> -
> > > > you have to reference the materialized view explicitly in y our query
> > > > - so in my book they've not delivered materialized view support. If
> > > > you're planning to deliver REAL materialized view support to
> Cassandra
> > > > that would be awesome.
> > > >
> > > > Julian
> > > >
> > > > [1]
> > > >
> > http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views
> > > >
> > > >
> > > > On Wed, Feb 24, 2016 at 2:17 PM, Julian Hyde <jh...@apache.org>
> wrote:
> > > > > As is typical for complex pieces of code like this, the
> documentation
> > > > > is in the code (and the unit test). It's probably not what you
> wanted
> > > > > to hear, but the code mutates quite fast and so if we'd written a
> > > > > design doc a few months ago it would be partially inaccurate.
> > > > >
> > > > > I, Maryann Xue and Amogh Margoor are the main authors of this code.
> > > > >
> > > > > Suggest you find a relevant test case in MaterializationTest (or
> > write
> > > > > a new one) and run it with trace enabled and/or in a debugger. You
> > > > > will see the process of matching an expression to a MV bottom up if
> > > > > you watch each call to UnifyRule.unify.
> > > > >
> > > > > Julian
> > > > >
> > > > >
> > > > > On Wed, Feb 24, 2016 at 1:40 PM, Michael Mior <mm...@uwaterloo.ca>
> > > > wrote:
> > > > >> Is there any documentation anywhere on how the current
> > implementation
> > > of
> > > > >> query rewriting for materialized views work? Mostly I'm referring
> > > > >> to MaterializedViewSubstitutionVisitor. There's a lot of code to
> > > digest
> > > > >> with not a lot of documentation and it would be helpful to have a
> > > > reference
> > > > >> to refer. Thanks!
> > > > >>
> > > > >> Cheers,
> > > > >> --
> > > > >> Michael Mior
> > > > >> mmior@uwaterloo.ca
> > > >
> > >
> >
>
Re: Materialized view rewriting
Posted by Rajat Venkatesh <rv...@qubole.com>.
In Quark, dont use hooks to define materializations. We use a tablefactory
[1] to defer until the schema is loaded[2].
1.
https://github.com/qubole/quark/blob/master/optimizer/src/main/java/com/qubole/quark/planner/MetadataSchema.java#L85
2.
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/materialize/MaterializationService.java#L132
Hopefully, I've understood your problem correctly.
On Thu, Feb 25, 2016 at 10:16 PM Maryann Xue <ma...@gmail.com> wrote:
> Hi Michael,
>
> We had a little difficulty defining our secondary indices as materialized
> views with our Schema SPI implementation too, and this made the code pretty
> hacky. In order to call that defineMaterialization method, we hold the
> parent SchemaPlus object within our own Schema impl object so that we can
> get its own corresponding SchemaPlus object by calling "parentSchema
> .getSubSchema(this.name)" later on. We do this after the schema/table
> resolving phase (that is when the entire schema tree incl. your own schema
> objects have been initiated) and call defineMaterialization for each
> secondary index under each subSchema. We add a hook in "Hook.TRIMMED" for
> this, which sounds pretty weird, but this is exactly a point after you have
> the whole schema tree ready and before the materializations are asked for
> by the planner.
>
> Anyway, I do hope the interface can be modified to avoid all this trouble.
>
>
> Thanks,
> Maryann
>
> On Thu, Feb 25, 2016 at 9:24 AM, Michael Mior <mm...@uwaterloo.ca> wrote:
>
> > Any suggestions on the best place to hook in and add the materialized
> > views? It seems like doing so requires the SchemaPlus object
> corresponding
> > to the current schema. The current best approach I see is to save a
> > reference to the parent schema and then pull out the appropriate
> SchemaPlus
> > object in getTableMap. This seems like a bit of a hack though.
> >
> > --
> > Michael Mior
> > mmior@uwaterloo.ca
> >
> > 2016-02-24 17:22 GMT-05:00 Julian Hyde <jh...@apache.org>:
> >
> > > By the way, interesting that you are interested in Cassandra and
> > > materialized views. Cassandra announced materialized view support
> > > recently[1] but they solved only half of a problem (not an
> > > insignificant half, I hasten to add), namely materialized view
> > > maintenance. They don't transparently substitute them into the query -
> > > you have to reference the materialized view explicitly in y our query
> > > - so in my book they've not delivered materialized view support. If
> > > you're planning to deliver REAL materialized view support to Cassandra
> > > that would be awesome.
> > >
> > > Julian
> > >
> > > [1]
> > >
> http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views
> > >
> > >
> > > On Wed, Feb 24, 2016 at 2:17 PM, Julian Hyde <jh...@apache.org> wrote:
> > > > As is typical for complex pieces of code like this, the documentation
> > > > is in the code (and the unit test). It's probably not what you wanted
> > > > to hear, but the code mutates quite fast and so if we'd written a
> > > > design doc a few months ago it would be partially inaccurate.
> > > >
> > > > I, Maryann Xue and Amogh Margoor are the main authors of this code.
> > > >
> > > > Suggest you find a relevant test case in MaterializationTest (or
> write
> > > > a new one) and run it with trace enabled and/or in a debugger. You
> > > > will see the process of matching an expression to a MV bottom up if
> > > > you watch each call to UnifyRule.unify.
> > > >
> > > > Julian
> > > >
> > > >
> > > > On Wed, Feb 24, 2016 at 1:40 PM, Michael Mior <mm...@uwaterloo.ca>
> > > wrote:
> > > >> Is there any documentation anywhere on how the current
> implementation
> > of
> > > >> query rewriting for materialized views work? Mostly I'm referring
> > > >> to MaterializedViewSubstitutionVisitor. There's a lot of code to
> > digest
> > > >> with not a lot of documentation and it would be helpful to have a
> > > reference
> > > >> to refer. Thanks!
> > > >>
> > > >> Cheers,
> > > >> --
> > > >> Michael Mior
> > > >> mmior@uwaterloo.ca
> > >
> >
>
Re: Materialized view rewriting
Posted by Michael Mior <mm...@uwaterloo.ca>.
Thanks for the suggestion Maryann! While it's obvious from this thread
that this isn't the ideal way to define materialized views, it did work in
this case. I'll be watching the JIRA for a better solution. I managed to
automated materialized view definitions in Cassandra in the case where
the original view has no WHERE clause defined. The somewhat tricky part
there is converting the clause written in CQL to SQL which Calcite can use.
Cheers,
--
Michael Mior
mmior@uwaterloo.ca
2016-02-25 11:46 GMT-05:00 Maryann Xue <ma...@gmail.com>:
> Hi Michael,
>
> We had a little difficulty defining our secondary indices as materialized
> views with our Schema SPI implementation too, and this made the code pretty
> hacky. In order to call that defineMaterialization method, we hold the
> parent SchemaPlus object within our own Schema impl object so that we can
> get its own corresponding SchemaPlus object by calling "parentSchema
> .getSubSchema(this.name)" later on. We do this after the schema/table
> resolving phase (that is when the entire schema tree incl. your own schema
> objects have been initiated) and call defineMaterialization for each
> secondary index under each subSchema. We add a hook in "Hook.TRIMMED" for
> this, which sounds pretty weird, but this is exactly a point after you have
> the whole schema tree ready and before the materializations are asked for
> by the planner.
>
> Anyway, I do hope the interface can be modified to avoid all this trouble.
>
>
> Thanks,
> Maryann
>
> On Thu, Feb 25, 2016 at 9:24 AM, Michael Mior <mm...@uwaterloo.ca> wrote:
>
> > Any suggestions on the best place to hook in and add the materialized
> > views? It seems like doing so requires the SchemaPlus object
> corresponding
> > to the current schema. The current best approach I see is to save a
> > reference to the parent schema and then pull out the appropriate
> SchemaPlus
> > object in getTableMap. This seems like a bit of a hack though.
> >
> > --
> > Michael Mior
> > mmior@uwaterloo.ca
> >
> > 2016-02-24 17:22 GMT-05:00 Julian Hyde <jh...@apache.org>:
> >
> > > By the way, interesting that you are interested in Cassandra and
> > > materialized views. Cassandra announced materialized view support
> > > recently[1] but they solved only half of a problem (not an
> > > insignificant half, I hasten to add), namely materialized view
> > > maintenance. They don't transparently substitute them into the query -
> > > you have to reference the materialized view explicitly in y our query
> > > - so in my book they've not delivered materialized view support. If
> > > you're planning to deliver REAL materialized view support to Cassandra
> > > that would be awesome.
> > >
> > > Julian
> > >
> > > [1]
> > >
> http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views
> > >
> > >
> > > On Wed, Feb 24, 2016 at 2:17 PM, Julian Hyde <jh...@apache.org> wrote:
> > > > As is typical for complex pieces of code like this, the documentation
> > > > is in the code (and the unit test). It's probably not what you wanted
> > > > to hear, but the code mutates quite fast and so if we'd written a
> > > > design doc a few months ago it would be partially inaccurate.
> > > >
> > > > I, Maryann Xue and Amogh Margoor are the main authors of this code.
> > > >
> > > > Suggest you find a relevant test case in MaterializationTest (or
> write
> > > > a new one) and run it with trace enabled and/or in a debugger. You
> > > > will see the process of matching an expression to a MV bottom up if
> > > > you watch each call to UnifyRule.unify.
> > > >
> > > > Julian
> > > >
> > > >
> > > > On Wed, Feb 24, 2016 at 1:40 PM, Michael Mior <mm...@uwaterloo.ca>
> > > wrote:
> > > >> Is there any documentation anywhere on how the current
> implementation
> > of
> > > >> query rewriting for materialized views work? Mostly I'm referring
> > > >> to MaterializedViewSubstitutionVisitor. There's a lot of code to
> > digest
> > > >> with not a lot of documentation and it would be helpful to have a
> > > reference
> > > >> to refer. Thanks!
> > > >>
> > > >> Cheers,
> > > >> --
> > > >> Michael Mior
> > > >> mmior@uwaterloo.ca
> > >
> >
>
Re: Materialized view rewriting
Posted by Maryann Xue <ma...@gmail.com>.
Hi Michael,
We had a little difficulty defining our secondary indices as materialized
views with our Schema SPI implementation too, and this made the code pretty
hacky. In order to call that defineMaterialization method, we hold the
parent SchemaPlus object within our own Schema impl object so that we can
get its own corresponding SchemaPlus object by calling "parentSchema
.getSubSchema(this.name)" later on. We do this after the schema/table
resolving phase (that is when the entire schema tree incl. your own schema
objects have been initiated) and call defineMaterialization for each
secondary index under each subSchema. We add a hook in "Hook.TRIMMED" for
this, which sounds pretty weird, but this is exactly a point after you have
the whole schema tree ready and before the materializations are asked for
by the planner.
Anyway, I do hope the interface can be modified to avoid all this trouble.
Thanks,
Maryann
On Thu, Feb 25, 2016 at 9:24 AM, Michael Mior <mm...@uwaterloo.ca> wrote:
> Any suggestions on the best place to hook in and add the materialized
> views? It seems like doing so requires the SchemaPlus object corresponding
> to the current schema. The current best approach I see is to save a
> reference to the parent schema and then pull out the appropriate SchemaPlus
> object in getTableMap. This seems like a bit of a hack though.
>
> --
> Michael Mior
> mmior@uwaterloo.ca
>
> 2016-02-24 17:22 GMT-05:00 Julian Hyde <jh...@apache.org>:
>
> > By the way, interesting that you are interested in Cassandra and
> > materialized views. Cassandra announced materialized view support
> > recently[1] but they solved only half of a problem (not an
> > insignificant half, I hasten to add), namely materialized view
> > maintenance. They don't transparently substitute them into the query -
> > you have to reference the materialized view explicitly in y our query
> > - so in my book they've not delivered materialized view support. If
> > you're planning to deliver REAL materialized view support to Cassandra
> > that would be awesome.
> >
> > Julian
> >
> > [1]
> > http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views
> >
> >
> > On Wed, Feb 24, 2016 at 2:17 PM, Julian Hyde <jh...@apache.org> wrote:
> > > As is typical for complex pieces of code like this, the documentation
> > > is in the code (and the unit test). It's probably not what you wanted
> > > to hear, but the code mutates quite fast and so if we'd written a
> > > design doc a few months ago it would be partially inaccurate.
> > >
> > > I, Maryann Xue and Amogh Margoor are the main authors of this code.
> > >
> > > Suggest you find a relevant test case in MaterializationTest (or write
> > > a new one) and run it with trace enabled and/or in a debugger. You
> > > will see the process of matching an expression to a MV bottom up if
> > > you watch each call to UnifyRule.unify.
> > >
> > > Julian
> > >
> > >
> > > On Wed, Feb 24, 2016 at 1:40 PM, Michael Mior <mm...@uwaterloo.ca>
> > wrote:
> > >> Is there any documentation anywhere on how the current implementation
> of
> > >> query rewriting for materialized views work? Mostly I'm referring
> > >> to MaterializedViewSubstitutionVisitor. There's a lot of code to
> digest
> > >> with not a lot of documentation and it would be helpful to have a
> > reference
> > >> to refer. Thanks!
> > >>
> > >> Cheers,
> > >> --
> > >> Michael Mior
> > >> mmior@uwaterloo.ca
> >
>
Re: Materialized view rewriting
Posted by Michael Mior <mm...@uwaterloo.ca>.
Any suggestions on the best place to hook in and add the materialized
views? It seems like doing so requires the SchemaPlus object corresponding
to the current schema. The current best approach I see is to save a
reference to the parent schema and then pull out the appropriate SchemaPlus
object in getTableMap. This seems like a bit of a hack though.
--
Michael Mior
mmior@uwaterloo.ca
2016-02-24 17:22 GMT-05:00 Julian Hyde <jh...@apache.org>:
> By the way, interesting that you are interested in Cassandra and
> materialized views. Cassandra announced materialized view support
> recently[1] but they solved only half of a problem (not an
> insignificant half, I hasten to add), namely materialized view
> maintenance. They don't transparently substitute them into the query -
> you have to reference the materialized view explicitly in y our query
> - so in my book they've not delivered materialized view support. If
> you're planning to deliver REAL materialized view support to Cassandra
> that would be awesome.
>
> Julian
>
> [1]
> http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views
>
>
> On Wed, Feb 24, 2016 at 2:17 PM, Julian Hyde <jh...@apache.org> wrote:
> > As is typical for complex pieces of code like this, the documentation
> > is in the code (and the unit test). It's probably not what you wanted
> > to hear, but the code mutates quite fast and so if we'd written a
> > design doc a few months ago it would be partially inaccurate.
> >
> > I, Maryann Xue and Amogh Margoor are the main authors of this code.
> >
> > Suggest you find a relevant test case in MaterializationTest (or write
> > a new one) and run it with trace enabled and/or in a debugger. You
> > will see the process of matching an expression to a MV bottom up if
> > you watch each call to UnifyRule.unify.
> >
> > Julian
> >
> >
> > On Wed, Feb 24, 2016 at 1:40 PM, Michael Mior <mm...@uwaterloo.ca>
> wrote:
> >> Is there any documentation anywhere on how the current implementation of
> >> query rewriting for materialized views work? Mostly I'm referring
> >> to MaterializedViewSubstitutionVisitor. There's a lot of code to digest
> >> with not a lot of documentation and it would be helpful to have a
> reference
> >> to refer. Thanks!
> >>
> >> Cheers,
> >> --
> >> Michael Mior
> >> mmior@uwaterloo.ca
>
Re: Materialized view rewriting
Posted by Julian Hyde <jh...@apache.org>.
By the way, interesting that you are interested in Cassandra and
materialized views. Cassandra announced materialized view support
recently[1] but they solved only half of a problem (not an
insignificant half, I hasten to add), namely materialized view
maintenance. They don't transparently substitute them into the query -
you have to reference the materialized view explicitly in y our query
- so in my book they've not delivered materialized view support. If
you're planning to deliver REAL materialized view support to Cassandra
that would be awesome.
Julian
[1] http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views
On Wed, Feb 24, 2016 at 2:17 PM, Julian Hyde <jh...@apache.org> wrote:
> As is typical for complex pieces of code like this, the documentation
> is in the code (and the unit test). It's probably not what you wanted
> to hear, but the code mutates quite fast and so if we'd written a
> design doc a few months ago it would be partially inaccurate.
>
> I, Maryann Xue and Amogh Margoor are the main authors of this code.
>
> Suggest you find a relevant test case in MaterializationTest (or write
> a new one) and run it with trace enabled and/or in a debugger. You
> will see the process of matching an expression to a MV bottom up if
> you watch each call to UnifyRule.unify.
>
> Julian
>
>
> On Wed, Feb 24, 2016 at 1:40 PM, Michael Mior <mm...@uwaterloo.ca> wrote:
>> Is there any documentation anywhere on how the current implementation of
>> query rewriting for materialized views work? Mostly I'm referring
>> to MaterializedViewSubstitutionVisitor. There's a lot of code to digest
>> with not a lot of documentation and it would be helpful to have a reference
>> to refer. Thanks!
>>
>> Cheers,
>> --
>> Michael Mior
>> mmior@uwaterloo.ca
Re: Materialized view rewriting
Posted by Julian Hyde <jh...@apache.org>.
As is typical for complex pieces of code like this, the documentation
is in the code (and the unit test). It's probably not what you wanted
to hear, but the code mutates quite fast and so if we'd written a
design doc a few months ago it would be partially inaccurate.
I, Maryann Xue and Amogh Margoor are the main authors of this code.
Suggest you find a relevant test case in MaterializationTest (or write
a new one) and run it with trace enabled and/or in a debugger. You
will see the process of matching an expression to a MV bottom up if
you watch each call to UnifyRule.unify.
Julian
On Wed, Feb 24, 2016 at 1:40 PM, Michael Mior <mm...@uwaterloo.ca> wrote:
> Is there any documentation anywhere on how the current implementation of
> query rewriting for materialized views work? Mostly I'm referring
> to MaterializedViewSubstitutionVisitor. There's a lot of code to digest
> with not a lot of documentation and it would be helpful to have a reference
> to refer. Thanks!
>
> Cheers,
> --
> Michael Mior
> mmior@uwaterloo.ca