Adding RelOptMaterializations to a planner

[Shubham Kumar <sh...@gmail.com>]

Hi contributors,

I am trying to retrieve materialized view query rewrite from Calcite.

In order to do that, I have created a planner and wanted to get the
optimized relNode after applying Materializations which is equivalent to
applying [1] present in Prepare class of Calcite. However, I am facing
difficulties while trying to do so. I have so far tried:

1) Doing exactly what's being done in [1] but it turned out some members of
<Prepare.Materializations> can't be accessed outside package, so I am
unable to obtain       RelOptMaterialization from it.

2) I figured I only need (tableRel, queryRel) relNodes for making a
RelOptMaterialization which can be made from Materialization.sql but when I
try to so with the a planner instance (i.e. planner.rel(validatedNode).rel)
: I am getting errors:

i) When using the different planner instance:
Relational expression LogicalFilter#24 belongs to a different planner than
is currently being used.
ii) When using the same planner instance:
java.lang.IllegalArgumentException: cannot move to STATE_2_READY from
STATE_5_CONVERTED

I figured out that maybe SqlToRelConverter could help out in this use case
but I don't know how to get instances of CatalogReader, Validator,
ViewExpander and SqlRexConvertletTable.

Can someone point out where I am going wrong and what's the correct way to
go about it.

[1]
https://github.com/apache/calcite/blob/73023148e7f37d494f6caf92b01b090f6dde13cd/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L320
-- 
Thanks & Regards

Shubham Kumar

Re: Adding RelOptMaterializations to a planner

[XING JIN <ji...@gmail.com>]

Hi Shubham,

In my understanding, same RelOptPlanner is the way to go, as one
VolcanoPlanner#registerMaterializations calls
RelOptMaterializations#useMaterializedViews and picks the best algebra
expression tree.
I'd suggest to create&register all the materializations into a
VolcanoPlanner and then "findBestExp".
Please note that there are two kinds of strategies for materialized view
optimization:
1. Substitution based materialized view optimization
2. Rewriting using plan structural information

For the first one, only RelOptMaterializations#useMaterializedViews is
enough, you can even don't rely on VolcanoPlanner, i.e. call
RelOptMaterializations#useMaterializedViews explicitly and pick the best
one by yourself. But for the second one, you need to to rely on
VolcanoPlanner and register corresponding rules from
AbstractMaterializedViewRule. The second one tend to be smarter but only
supports SPJA pattern.

In short, when you enable config of "materializationEnabled" for connection
property, both of the two strategies above are enabled.

In addition, I'd suggest to do canonicalization before materialized view
optimization, which helps a lot for materialized view matching.

I'm also doing some work for materialized view optimization. It would be
great to have more discussion on this :)
https://issues.apache.org/jira/browse/CALCITE-3334
https://docs.google.com/document/d/1JpwGNFE3hw3yXb7W3-95-jXKClZC5UFPKbuhgYDuEu4/edit#heading=h.bmvjxz1h5evc

Best,
Jin

Re: Adding RelOptMaterializations to a planner

[Stamatis Zampetakis <za...@gmail.com>]

Hi Shubham,

Regarding point 1) you can easily get around it by defining a few of your
own classes in a package with the same name as Prepare although I doubt is
the right approach.

Perhaps MaterializationTest [1] can help you get a few more ideas on how to
use the respective service.

Other than that if you want to see how to instantiate CatalogReader,
Validator, etc., maybe the code sample in [2] can help.

Best,
Stamatis

[1]
https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/MaterializationTest.java
[2]
https://github.com/zabetak/calcite/blob/livecodingdemo/core/src/test/java/org/apache/calcite/examples/foodmart/java/EndToEndExampleEnumerable.java

On Fri, Oct 4, 2019 at 12:22 PM Shubham Kumar <sh...@gmail.com>
wrote:

> Hi contributors,
>
> I am trying to retrieve materialized view query rewrite from Calcite.
>
> In order to do that, I have created a planner and wanted to get the
> optimized relNode after applying Materializations which is equivalent to
> applying [1] present in Prepare class of Calcite. However, I am facing
> difficulties while trying to do so. I have so far tried:
>
> 1) Doing exactly what's being done in [1] but it turned out some members of
> <Prepare.Materializations> can't be accessed outside package, so I am
> unable to obtain       RelOptMaterialization from it.
>
> 2) I figured I only need (tableRel, queryRel) relNodes for making a
> RelOptMaterialization which can be made from Materialization.sql but when I
> try to so with the a planner instance (i.e. planner.rel(validatedNode).rel)
> : I am getting errors:
>
> i) When using the different planner instance:
> Relational expression LogicalFilter#24 belongs to a different planner than
> is currently being used.
> ii) When using the same planner instance:
> java.lang.IllegalArgumentException: cannot move to STATE_2_READY from
> STATE_5_CONVERTED
>
> I figured out that maybe SqlToRelConverter could help out in this use case
> but I don't know how to get instances of CatalogReader, Validator,
> ViewExpander and SqlRexConvertletTable.
>
> Can someone point out where I am going wrong and what's the correct way to
> go about it.
>
> [1]
>
> https://github.com/apache/calcite/blob/73023148e7f37d494f6caf92b01b090f6dde13cd/core/src/main/java/org/apache/calcite/prepare/Prepare.java#L320
> --
> Thanks & Regards
>
> Shubham Kumar
>