You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Vladimir Ozerov <pp...@gmail.com> on 2019/10/28 21:24:15 UTC

Problem with converters and possibly rule matching

Hi colleagues,

We are building a Calcite-based optimizer for Hazelcast, and I have some
problems understanding Calcite's logic with respect to converters. Let me
briefly explain the problem.

We have an execution backend, so we do not need Bindable or Enumerable.
Instead, we would like to use Calcite to convert original SQL to a tree
with our own convention, then convert it to our internal representation,
and finally, execute.

We started with looking at other Calcite integrations and eventually came
to a classical two-phase optimization approach. We have two internal
conventions - LOGICAL and PHYSICAL. The goal is to optimize the tree as
follows:
1) NONE -> LOGICAL - heuristical optimizations
2) LOGICAL -> PHYSICAL - cost-based planning

Suppose that after the first phase I have the following tree of our own
operators:
HZLogicalRoot
-> HZLogicalProject
  -> HZLogicalScan

For this specific case, there is not much to optimize, so we only need to
transition to physical nodes and do some boilerplate with traits
propagation:
HZPhysicalRoot
-> HZPhysicalProject
  -> HZPhysicalScan

In order to achieve this, I define three rules, which just do a conversion
of relevant nodes. Volcano optimizer is used.

Now, the problem - somehow it works only when I override
Convention.Impl.canConvertConvention to true for our PHYSICAL convention,
but that blows the search space and the same rules are called many times. A
lot of time is spent on endless PHYSICAL -> LOGICAL conversions, which are
of no use.

If I change canConvertConvention to false, then rules are called a sensible
number of times, but cannot produce a complete PHYSICAL tree. Here is how
it works:
1) "Root" rule is invoked, which converts "HZLogicalRoot" to
"HZPhysicalRoot"
2) "Project" rule is invoked, but do not produce any transformations, since
it needs Scan distribution, which is not known yet. This desired behavior
at this point.
3) "Scan" rule is invoked, "HZLogicalScan" is converted to
"HZPhysicalScan". Distribution is resolved
4) At this point, we have [LogicalRoot, PhysicalRoot] -> [LogicalProject]
-> [LogicalScan, PhysicalScan] sets . I expect that since new scan was
installed, the "Project" rule will be fired again. This time we know the
distribution, so the transformation is possible. But the rule is not called
and we fail with an error.

So my questions are:
1) What is the real role of converters in this process? For some reason,
when unnecessary (from a logical standpoint) PHYSICAL -> LOGICAL conversion
is allowed, even complex plans could be built. And Drill does it for some
reason. But it costs multiple additional invocations of the same rules. Are
there any docs or presentations explaining the mechanics behind?
2) What are the minimum requirements, that will allow a rule on the parent
to be fired again after it's child node has changed?

I can provide any additional information, source code or even working
example of this problem if needed. I don't want to bother you with it at
the moment, because it feels like I miss something very simple.

Would appreciate your help.

Regards,
Vladimir.

Re: Problem with converters and possibly rule matching

Posted by Seliverstov Igor <gv...@gmail.com>.
Stamatis,

Small comment from me, for distributed systems is more efficient if
SINGLETON distribution appears only on root node, in other words as late as
possible, it allows to filter or cut rows (project) on source nodes which
significantly reduces network costs.

To do that (In case we use original volcano approach) we need to request
all possible distribution traits on project node (some of them we cannot
request, because we don't know whether a repartitioning and rehashing
happens in the middle) to provide a place for possible converted inputs
(possibly leaving empty subsets, which causes exceptions in current
implementation). Such approach significant slowdowns planning phase.

Regards,
Igor

пт, 1 нояб. 2019 г., 11:42 Stamatis Zampetakis <za...@gmail.com>:

> The discussion is interesting thanks for the nice examples.
> I am replying in this thread since it has all the examples and the history
> of the conversation.
>
> *Part I: Distribution physical or logical property*
>
> The Volcano paper writes the following regarding properties:
>
> "Logical properties can be derived from the logical algebra expression and
> include schema, expected size, etc., while physical properties depend on
> algorithms, e.g., sort order, partitioning, etc."
>
> We could say that partitioning is not only a property which depends on an
> algorithm but a property which depends on the schema. When you have a table
> you know in advance that the particular table is replicated.
>
> Basically, this means that the starting "logical" plan in your case could
> be the following:
> RootLogicalRel[convention=LOGICAL, distribution=ANY]
> -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
>   -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
>
> When you are about to create your physical projection it is not necessary
> to create three equivalent paths but the most promising by asking the input
> what can it offer as traits. Again this partially overlaps with the
> discussion about "On demand traitset" [1] where as I wrote there is already
> a mechanism to derive traits from children (collation, distribution, etc.).
>
> Note that I am not saying that there is nothing more to be done to improve
> the Calcite optimizer; I am just trying to provide a viable alternative
> given the current situation of the project.
>
> *Part II: Example with the original Volcano optimizer*
>
> Now let me try to show how the original (from the paper) Volcano optimizer
> could handle your query.
>
> The optimization starts and we request two properties convention and
> distribution.
>
> RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
> -> ProjectLogicalRel
>   -> MapScanLogicalRel
>
> Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but cannot
> satisfy the SINGLETON property so it propagates the demand
>
> RootPhysicalRel
> -> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>   -> MapScanLogicalRel
>
> Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the algorithm but
> cannot satisfy the SINGLETON property so it propagates the demand
>
> RootPhysicalRel
> -> ProjectPhysicalRel
>   -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>
> Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the algorithm and
> given that the table is replicated it satisfies the distribution property.
>
> RootPhysicalRel
> -> ProjectPhysicalRel
>   -> MapScanPhysicalRel
>
> So we end up with a complete plan that satisfies all properties and does
> not have redundant exchange operators.
> Moreover we didn't require all possible distributions when we applied Rule
> 2 since we just want to satisfy the SINGLETON property requested by the
> parent.
> The example above does not demonstrate the additional optimization paths
> which would be apply an enforcer (an Exchange operator to satisfy the
> SINGLETON property) at a higher level than the scan.
>
> Would this be an acceptable approach? Does it still suffer from a big
> search space?
>
> To make the analog with the actual Volcano optimizer in Calcite the thing
> that may be missing is to be able to know what trait was requested by the
> parent during the application of Rule 2.
>
> Best,
> Stamatis
>
> [1]
>
> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
>
> On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <pp...@gmail.com>
> wrote:
>
> > Hi Stamatis,
> >
> > The problem that the presented reproducer is a very simplified version of
> > what is actually needed. In reality, there are several distribution
> types -
> > PARTITIONED, REPLICATED, SINGLETON. To make things worse, PARTITIONED may
> > or may not have concrete distribution fields. In theory, I can create one
> > transformation per distribution type, but that would increase plan space
> > significantly. In my sample "Root <- Project <- Scan" plan, there is no
> > room for optimization at all, we only need to convert the nodes and
> > propagate traits. But if I follow the proposed approach, the planner
> would
> > create three equivalent paths. For complex queries, this may increase
> > optimization time significantly.
> >
> > What I need instead is to gradually convert and optimize nodes
> *bottom-up*,
> > instead of top-bottom. That is, create a physical scan first, then
> create a
> > physical project on top of it, etc. But at the same time, some rules
> still
> > require the top-bottom approach. So essentially I need the optimizer to
> do
> > both. Abstract converters help me establish bottom-up preparation but do
> > this at the cost of considering too many trait pairs, and as a result,
> also
> > pollute the search space.
> >
> > To the contrast, precise command "I transformed the node A to node B,
> > please re-trigger the rules for A's parents" would allow us to re-trigger
> > only required rules, without adding more nodes.
> >
> > Does it make sense?
> >
> > Regards,
> > Vladimir.
> >
> > ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <za...@gmail.com>:
> >
> > > I admit that I didn't thoroughly read the exchanges so far but forcing
> > the
> > > rules trigger does not seem the right approach.
> > >
> > > Other than that I would like to backtrack a bit to point 4.3 raised by
> > > Vladimir.
> > >
> > > "ProjectPhysicalRule [6] - transforms logical project to physical
> > > project *ONLY* if there is an underlying physical input with REPLICATED
> > or
> > > SINGLETON distribution"
> > >
> > > The rule could be modified to do the following two transformations:
> > > 1. Create a physical project and require the input to be REPLICATED.
> > > 2. Create a physical project and require
> > > the input to be SINGLETON.
> > >
> > > I would assume that afterwards when your scan rule fires it should go
> to
> > > the appropriate subset and give you back the desired plan. Is there a
> > > problem with this approach?
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <gv...@gmail.com>
> > > wrote:
> > >
> > > > Unfortunately it requires package-private API usage.
> > > >
> > > > Best solution there if Calcite provide it for end users.
> > > >
> > > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
> > > >
> > > > > “e pension” == “expand conversion” :)
> > > > >
> > > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <pp...@gmail.com>:
> > > > >
> > > > > > Yes, that may work. Even if e pension rule is used, for the most
> > > cases
> > > > it
> > > > > > will not trigger any real conversions, since we are moving from
> > > > abstract
> > > > > > convention to physical, and created converters will have the
> > opposite
> > > > > trait
> > > > > > direction (from physical to abstract).
> > > > > >
> > > > > > But again - ideally we only need to re-trigger the rules for a
> > > specific
> > > > > > node, no more than that. So API support like
> > > > > > “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
> > gvvinblade@gmail.com
> > > >:
> > > > > >
> > > > > >> I considered manual rules calling too, for now we use abstract
> > > > > converters
> > > > > >> +
> > > > > >> ExpandConversionRule for exchanges producing.
> > > > > >>
> > > > > >> You may create such converters manually (checking appropriate
> > > subset)
> > > > > this
> > > > > >> case you may reduce created converters count, also, a converter
> > is a
> > > > > quite
> > > > > >> special node, that does almost nothing (without corresponding
> > rule)
> > > it
> > > > > may
> > > > > >> be used just as a rule trigger.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Igor
> > > > > >>
> > > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <ppozerov@gmail.com
> >:
> > > > > >>
> > > > > >> > One funny hack which helped me is manual registration of a
> fake
> > > > > RelNode
> > > > > >> > with desired traits through VolcanoPlanner.register() method.
> > But
> > > > > again,
> > > > > >> > this leads to trashing. What could really help is a call to
> > > > > >> > VolcanoPlanner.fireRules() with desired rel. But this doesn't
> > work
> > > > out
> > > > > >> of
> > > > > >> > the box since some internals of the rule queue needs to be
> > > adjusted.
> > > > > >> >
> > > > > >> > What does the community think about adding a method which will
> > > > re-add
> > > > > >> rules
> > > > > >> > applicable to the specific RelNode to the rule queue?
> > > > > >> >
> > > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
> > ppozerov@gmail.com
> > > >:
> > > > > >> >
> > > > > >> > > Hi Igor,
> > > > > >> > >
> > > > > >> > > Yes, I came to the same conclusion, thank you. This is how
> it
> > > > > >> basically
> > > > > >> > > happens when converters are disabled:
> > > > > >> > > 1) We start with initial tree: [LogicalProject] <-
> > [LogicalScan]
> > > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is
> added
> > > to
> > > > > the
> > > > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > > > > >> > > 3) Finally, when it is time to fire a rule for PhysicalScan,
> > we
> > > > try
> > > > > to
> > > > > >> > get
> > > > > >> > > parents of that scan set with traits of the PhysicalScan.
> > Since
> > > > > there
> > > > > >> are
> > > > > >> > > no such parents (we skipped it intentionally), the rule is
> not
> > > > > queued.
> > > > > >> > >
> > > > > >> > > But when converters are enabled, a converter rel is created:
> > > > > >> > [LogicalProject]
> > > > > >> > > <- [LogicalScan, PhysicalScan,
> > ConverterFromPhysicalToLogical].
> > > No
> > > > > >> rules
> > > > > >> > > are fired for PhysicalScan again, but they are fired for
> > > converter
> > > > > >> since
> > > > > >> > > it has the necessary LOGICAL trait.
> > > > > >> > >
> > > > > >> > > It makes sense, that converters essentially allow forcing
> rule
> > > > > >> invocation
> > > > > >> > > on parents, even if the child was created with different
> > traits.
> > > > But
> > > > > >> it
> > > > > >> > > seems that this mechanism may be too heavy for complex
> queries
> > > > > >> because it
> > > > > >> > > literally creates hundreds of new converter rels and
> triggers
> > > > rules
> > > > > >> over
> > > > > >> > > and over again.
> > > > > >> > >
> > > > > >> > > We need some fine-grained alternative. Basically, what would
> > be
> > > > > enough
> > > > > >> > for
> > > > > >> > > me is to let the planner know somehow: "I created that rel,
> > and
> > > I
> > > > > want
> > > > > >> > you
> > > > > >> > > to execute parent rules not only using its trait but also on
> > > this
> > > > > and
> > > > > >> > those
> > > > > >> > > traits."
> > > > > >> > > Is there any API in Calcite which allows doing this without
> > > > > creating a
> > > > > >> > new
> > > > > >> > > rel node?
> > > > > >> > >
> > > > > >> > > Regards,
> > > > > >> > > Vladimir.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> > > > gvvinblade@gmail.com
> > > > > >:
> > > > > >> > >
> > > > > >> > >> Vladimir,
> > > > > >> > >>
> > > > > >> > >> Probably it'll help you:
> > > > > >> > >>
> > > > > >> > >> Seems the cause of issue in RelSubset.getParentRels()  The
> > > check
> > > > > used
> > > > > >> > when
> > > > > >> > >> the planner schedules newly matched rules after successful
> > > > > >> > transformation
> > > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the parent
> > rule
> > > > be
> > > > > >> > applied
> > > > > >> > >> once again (here your logical project with an input having
> > ANY
> > > > > >> > >> distribution
> > > > > >> > >> doesn't satisfy a transformed input traits).
> > > > > >> > >>
> > > > > >> > >> In our case we use another workaround, so there are also
> much
> > > > more
> > > > > >> > >> transformations than we wanted, so the desired rule is
> > > triggered.
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
> > ppozerov@gmail.com
> > > >:
> > > > > >> > >>
> > > > > >> > >> > Hi Vladimir,
> > > > > >> > >> >
> > > > > >> > >> > I am sorry. Pushed, it works now.
> > > > > >> > >> >
> > > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > > > > >> > >> > sitnikov.vladimir@gmail.com
> > > > > >> > >> > >:
> > > > > >> > >> >
> > > > > >> > >> > > > mvn clean test
> > > > > >> > >> > >
> > > > > >> > >> > > [ERROR] The goal you specified requires a project to
> > > execute
> > > > > but
> > > > > >> > >> there is
> > > > > >> > >> > > no POM in this directory
> > > > > >> > >> > >
> > > > > >> > >> > > Vladimir, please push missing files
> > > > > >> > >> > >
> > > > > >> > >> > > Vladimir
> > > > > >> > >> > >
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Re: Re: Problem with converters and possibly rule matching

Posted by Danny Chan <yu...@gmail.com>.
The 1 approach is what actually Apache Flink did and works great.

Vladimir Ozerov <pp...@gmail.com>于2019年11月20日 周三上午2:22写道:

> Hi Stamatis,
>
> This approach worth trying, but I am afraid that it may produce too many
> permutations. For example, this is the logical plan of one of TPC-H
> benchmarks, and I am not very keen expanding it much further :-)
>
> 901:ProjectLogicalRel(ps_partkey=[$0], val=[$1])
>   899:FilterLogicalRel(subset=[rel#900:Subset#66.LOGICAL.ANY.[]],
> condition=[>($1, $2)])
>     897:JoinLogicalRel(subset=[rel#898:Subset#65.LOGICAL.ANY.[]],
> condition=[true], joinType=[left])
>       882:AggregateLogicalRel(subset=[rel#883:Subset#57.LOGICAL.ANY.[]],
> group=[{0}], val=[SUM($1)])
>         880:ProjectLogicalRel(subset=[rel#881:Subset#56.LOGICAL.ANY.[]],
> ps_partkey=[$1], $f1=[*($2, $3)])
>           878:JoinLogicalRel(subset=[rel#879:Subset#55.LOGICAL.ANY.[]],
> condition=[=($5, $6)], joinType=[inner])
>             875:JoinLogicalRel(subset=[rel#876:Subset#53.LOGICAL.ANY.[]],
> condition=[=($0, $4)], joinType=[inner])
>
> 634:MapScanLogicalRel(subset=[rel#873:Subset#51.LOGICAL.ANY.[]],
> table=[[partsupp]], projects=[[0, 1, 2, 3]])
>
> 632:MapScanLogicalRel(subset=[rel#874:Subset#52.LOGICAL.ANY.[]],
> table=[[supplier]], projects=[[0, 1]])
>
> 630:MapScanLogicalRel(subset=[rel#877:Subset#54.LOGICAL.ANY.[]],
> table=[[nation]], projects=[[0, 1]], filter=[=($1, '[NATION]')])
>       895:AggregateLogicalRel(subset=[rel#896:Subset#64.LOGICAL.ANY.[]],
> group=[{}], agg#0=[SINGLE_VALUE($0)])
>         893:ProjectLogicalRel(subset=[rel#894:Subset#63.LOGICAL.ANY.[]],
> EXPR$0=[*($0, 0.9:DECIMAL(2, 1))])
>
> 891:AggregateLogicalRel(subset=[rel#892:Subset#62.LOGICAL.ANY.[]],
> group=[{}], agg#0=[SUM($0)])
>
> 889:ProjectLogicalRel(subset=[rel#890:Subset#61.LOGICAL.ANY.[]], $f0=[*($1,
> $2)])
>               887:JoinLogicalRel(subset=[rel#888:Subset#60.LOGICAL.ANY.[]],
> condition=[=($4, $5)], joinType=[inner])
>
> 885:JoinLogicalRel(subset=[rel#886:Subset#59.LOGICAL.ANY.[]],
> condition=[=($0, $3)], joinType=[inner])
>
> 680:MapScanLogicalRel(subset=[rel#884:Subset#58.LOGICAL.ANY.[]],
> table=[[partsupp]], projects=[[0, 2, 3]])
>
> 632:MapScanLogicalRel(subset=[rel#874:Subset#52.LOGICAL.ANY.[]],
> table=[[supplier]], projects=[[0, 1]])
>
> 630:MapScanLogicalRel(subset=[rel#877:Subset#54.LOGICAL.ANY.[]],
> table=[[nation]], projects=[[0, 1]], filter=[=($1, '[NATION]')])
>
> I have two working solutions at the moment, which do not require
> cross-convention converters:
> 1) Create separate rules for logical-physical node conversion, and for
> physical-physical optimization
> 2) A kind of hack: add fake AbstractConverter with logical traits after the
> physical node is created
>
> I hope one of them will work good enough for the production case. Most
> likely it would be p.1.
>
> Regards,
> Vladimir.
>
>
> пт, 15 нояб. 2019 г. в 11:04, Stamatis Zampetakis <za...@gmail.com>:
>
> > Hi again Vladimir,
> >
> > In my previous email, I mentioned three rules for performing logical to
> > physical conversions.
> > It's normal that if you add more operators we will end up with more
> rules.
> > Now that the example has a filter we have the following rule list:
> >
> > Rule 1: RootLogicalRel -> RootPhysicalRel
> > Rule 2: ProjectLogicalRel -> ProjectPhysicalRel
> > Rule 3: FilterLogicalRel -> FilterPhysicalRel
> > Rule 4: MapScanLogicalRel -> MapScanPhysicalRel
> >
> > If I understand well, your concern is that in order to get the optimal
> plan
> > you would have to introduce many additional rules for the Exchange
> operator
> > (e.g., ExchangeProjectTransposeRule, ExchangeFilterTransposeRue, etc.).
> > I was thinking that it is not necessary to introduce these kind of rules.
> > You could have only one rule, i.e., PhysicaExchangeRule, for enforcing a
> > distribution that would be applied when there is a requirement for
> > particular distribution.
> >
> > So for the initial plan
> > LogicalRoot [distribution=SINGLETON]
> > -> LogicalProject
> >  -> LogicalFilter
> >   -> LogicalScan
> >
> > The ruleset above should generate the following alternatives where the
> > PhysicalExchangeRule is applied 3 times.
> >
> > Alt1:
> > PhysicalRoot
> > -> SingletonExchange
> >  -> PhysicalProject
> >   -> PhysicalFilter
> >    -> PhysicalScan
> >
> > Alt2:
> > PhysicalRoot
> > -> PhysicalProject
> >  -> SingletonExchange
> >   -> PhysicalFilter
> >    -> PhysicalScan
> >
> > Alt3:
> > PhysicalRoot
> > -> PhysicalProject
> >  -> PhysicalFilter
> >   -> SingletonExchange
> >    -> PhysicalScan
> >
> > This is still in the same spirit of the previous example where the
> physical
> > property (distribution) is either satisfied immediately or passed down to
> > the next level.
> >
> > Best,
> > Stamatis
> >
> > On Thu, Nov 14, 2019 at 11:14 AM Vladimir Ozerov <pp...@gmail.com>
> > wrote:
> >
> > > search place* = search space
> > >
> > > чт, 14 нояб. 2019 г. в 13:10, Vladimir Ozerov <pp...@gmail.com>:
> > >
> > > > Hi Haisheng,
> > > >
> > > > I double-checked the code. My original version returned false for
> some
> > > > cases, but it didn't affect number of rules calls anyway, so I
> changed
> > it
> > > > to always return true. Please note that if I change the code as you
> > > > suggested, the test started failing, because bottom-up propagation of
> > > rule
> > > > calls no longer work: when the child is converted to physical form,
> the
> > > > parent logical node is not notified. This is the very problem I
> address
> > > > with that weird physical-to-logical conversions: they do not make
> > sense,
> > > > and converter expansion does not produce any new rels, but their
> > > existence
> > > > allow for logical rule re-trigger which ultimately allow the plan to
> > > > compile.
> > > >
> > > > Regarding two conventions - I agree that it may look strange, but I
> do
> > > not
> > > > see any problems from the correctness perspective. Separation of
> > logical
> > > > and physical planning helps me avoid excessive expansion of the
> search
> > > > place: I do not want my physical rules to produce new rels from not
> > > > optimized logical rels. Do you see any problems with that approach?
> > > >
> > > > Regards,
> > > > Vladimir
> > > >
> > > > ср, 6 нояб. 2019 г. в 03:52, Haisheng Yuan <h....@alibaba-inc.com>:
> > > >
> > > >> Hi Vladimir,
> > > >>
> > > >> The code in PHYSICAL convention L44 looks weird, I think it always
> > > >> returns true.
> > > >>
> > > >>
> > >
> >
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/HazelcastConventions.java#L44
> > > >>
> > > >> Try this:
> > > >>
> > > >> fromTraits.containsIfApplicable(Convention.PHYSICAL)
> > > >>     && toTraits.containsIfApplicable(Convention.PHYSICAL);
> > > >>
> > > >>
> > > >> Adding a AbstractConverter on logical operators is meaningless.
> > Calcite
> > > >> is mixing the concept of logical and physical together, which is
> sad.
> > > >>
> > > >> BTW, using 2 conventions is not appropriate and wrong.
> > > >>
> > > >> - Haisheng
> > > >>
> > > >> ------------------------------------------------------------------
> > > >> 发件人:Vladimir Ozerov<pp...@gmail.com>
> > > >> 日 期:2019年11月05日 18:02:15
> > > >> 收件人:Haisheng Yuan<h....@alibaba-inc.com>
> > > >> 抄 送:dev@calcite.apache.org (dev@calcite.apache.org)<
> > > >> dev@calcite.apache.org>
> > > >> 主 题:Re: Re: Problem with converters and possibly rule matching
> > > >>
> > > >> Hi Haisheng,
> > > >>
> > > >>  think I already tried something very similar to what you explained,
> > but
> > > >> it gave not an optimal plan. Please let me describe what I did. I
> > would
> > > >> appreciate your feedback.
> > > >>
> > > >> 1) We start with a simple operator tree Root <- Project <- Scan,
> where
> > > >> the root is a final aggregator in the distributed query engine:
> > > >> -> LogicalRoot
> > > >>  -> LogicalProject
> > > >>   -> LogicalScan
> > > >>
> > > >> 2) First, we convert the Root and enforce SINGLETON distribution on
> a
> > > >> child:
> > > >> *-> PhysicalRoot[SINGLETON]*
> > > >> * -> Enforcer#1[SINGLETON]*
> > > >>   -> LogicalProject
> > > >>    -> LogicalScan
> > > >>
> > > >> 3) Then the project's rule is invoked. It doesn't know the
> > distribution
> > > >> of the input, so it requests ANY distribution. Note that we have to
> > set
> > > ANY
> > > >> to the project as well since we do not know the distribution of the
> > > input:
> > > >> -> PhysicalRoot[SINGLETON]
> > > >>  -> Enforcer#1[SINGLETON]
> > > >> *  -> PhysicalProject[ANY]*
> > > >> *   -> Enforcer#2[ANY]*
> > > >>     -> LogicalScan
> > > >>
> > > >> 4) Finally, the physical scan is created and its distribution is
> > > >> resolved. Suppose that it is REPLICATED, i.e. the whole result set
> is
> > > >> located on all nodes.
> > > >> -> PhysicalRoot[SINGLETON]
> > > >>  -> Enforcer#1[SINGLETON]
> > > >>   -> PhysicalProject[ANY]
> > > >>    -> Enforcer#2[ANY]
> > > >> *    -> PhysicalScan[REPLICATED]*
> > > >>
> > > >> 5) Now as all logical nodes are converted, we start resolving
> > enforcers.
> > > >> The second one is no-op, since REPLICATED satisfies ANY:
> > > >> -> PhysicalRoot[SINGLETON]
> > > >>  -> Enforcer#1[SINGLETON]
> > > >>   -> PhysicalProject[ANY]
> > > >>    -> PhysicalScan[REPLICATED]
> > > >>
> > > >> 6) But the first enforcer now requires an Exchange, since ANY
> doesn't
> > > >> satisfy SINGLETON!
> > > >> -> PhysicalRoot[SINGLETON]
> > > >> * -> SingletonExchange[SINGLETON]*
> > > >>   -> PhysicalProject[ANY] // <= unresolved!
> > > >>    -> PhysicalScan[REPLICATED]
> > > >>
> > > >> The resulting plan requires data movement only because we didn't
> know
> > > >> precise distribution of the PhysicalProject when it was created. But
> > > should
> > > >> I enable Convention.Impl.canConvertConvention, bottom-up propagation
> > > >> kicks in, and the correct plan is produced because now
> LogicalProject
> > > >> has a chance to be converted to PhysicalProject with the concrete
> > > >> distribution. The optimized plan looks like this (since REPLICATED
> > > >> satisfies SINGLETON):
> > > >> -> PhysicalRoot[SINGLETON]
> > > >>  -> PhysicalProject[REPLICATED]
> > > >>   -> PhysicalScan[REPLICATED]
> > > >>
> > > >> You may see this in action in my reproducer:
> > > >> 1) Test producing "bad" plan:
> > > >>
> > >
> >
> https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L45
> > > >> 2) Root enforces SINGLETON on Project:
> > > >>
> > >
> >
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/RootPhysicalRule.java#L45
> > > >> 3) Project enforces default (ANY) distribution on Scan:
> > > >>
> > >
> >
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L49
> > > >>
> > > >> Please let me know if this flow is similar to what you meant.
> > > >>
> > > >> Regards,
> > > >> Vladimir.
> > > >>
> > > >> пн, 4 нояб. 2019 г. в 10:33, Haisheng Yuan <h.yuan@alibaba-inc.com
> >:
> > > >>
> > > >>> Hi Vladimir,
> > > >>>
> > > >>> This is still can be done through top-down request approach.
> > > >>>
> > > >>> PhysicalFilter operator should request ANY distribution from child
> > > >>> operator, unless there is outer reference in the filter condition,
> in
> > > which
> > > >>> case, PhysicalFilter should request SINGLETON or BROADCAST
> > > distribution. So
> > > >>> in your case, PhysicalFilter request ANY, its required distribution
> > > will be
> > > >>> enforced on filter's output.
> > > >>>
> > > >>> Regarding index usage, you should have a FIlterTableScan2IndexGet
> > > >>> logical transformation rule, and a IndexGet2IndexScan physical
> > > >>> implementation rule. Note that IndexGet is a logical operator and
> > > IndexScan
> > > >>> is a physical operator, which are also used by SQL Server.
> > > >>>
> > > >>> - Haisheng
> > > >>>
> > > >>> ------------------------------------------------------------------
> > > >>> 发件人:Vladimir Ozerov<pp...@gmail.com>
> > > >>> 日 期:2019年11月01日 17:30:26
> > > >>> 收件人:<de...@calcite.apache.org>
> > > >>> 主 题:Re: Problem with converters and possibly rule matching
> > > >>>
> > > >>> Hi Stamatis,
> > > >>>
> > > >>> Thank you for your reply. I also thought that we may set the
> > > distribution
> > > >>> trait during logical planning because it is known in advance. And
> the
> > > >>> example I gave will work! :-) But unfortunately, it will work only
> > > >>> because
> > > >>> the tree is very simple, and the Project is adjacent to the Scan.
> > This
> > > is
> > > >>> how my reproducer will work in that case:
> > > >>> 1) Root: enforce "SINGLETON" on Project
> > > >>> 2) Project: check the logical Scan, infer already resolved
> > > distribution,
> > > >>> then convert to [PhyiscalProject <- PhysicalScan]
> > > >>> 3) Resolve Root enforcer, adding and Exchange if needed.
> > > >>>
> > > >>> But this stops working as soon as a plan becomes more complex so
> that
> > > it
> > > >>> is
> > > >>> impossible to infer the distribution from the child immediately.
> > E.g.:
> > > >>> LogicalRoot [distribution=SINGLETON]
> > > >>> -> LogicalProject // We are here new and cannot produce the
> physical
> > > >>> project
> > > >>> -> LogicalFilter[distribution=?]
> > > >>> -> LogicalScan[distribution=REPLICATED]
> > > >>>
> > > >>> This is where your suggestion with cascading enforcement may kick
> in.
> > > But
> > > >>> now consider that instead of having REPLICATED distribution, which
> > > >>> satisfies SINGLETON, we have a PARTITIONED distribution, It doesn't
> > > >>> satisfy
> > > >>> SINGLETON, so we have to insert the exchange.
> > > >>>
> > > >>> Before:
> > > >>> PhysicalRoot [SINGLETON]
> > > >>> -> PhysicalProject [SINGLETON]
> > > >>> -> PhysicalFilter [SINGLETON]
> > > >>> -> LogicalScan [PARTITIONED] // SINGLETON is enforced here
> > > >>>
> > > >>> After:
> > > >>> PhysicalRoot [SINGLETON]
> > > >>> -> PhysicalProject [SINGLETON]
> > > >>> -> PhysicalFilter [SINGLETON]
> > > >>> -> SingletonExchange [SINGLETON]
> > > >>> -> PhysicalScan [PARTITIONED]
> > > >>>
> > > >>> But unfortunately, this plan is not optimal. Since we perform
> Project
> > > and
> > > >>> Filter after the exchange, we now have to reshuffle the whole
> table.
> > > The
> > > >>> optimal plan would be to pushdown Project and Filter past exchange:
> > > >>> PhysicalRoot [SINGLETON]
> > > >>> -> SingletonExchange [SINGLETON]
> > > >>> -> PhysicalProject [PARTITIONED]
> > > >>> -> PhysicalFilter [PARTITIONED]
> > > >>> -> PhysicalScan [PARTITIONED]
> > > >>>
> > > >>> But in order to achieve that, now I need to introduce new rules
> which
> > > >>> will
> > > >>> attempt to transpose dozens of different operators past exchange,
> so
> > > most
> > > >>> likely the planning will never finish :-)
> > > >>>
> > > >>> This is why it seems that the bottom-up approach seems to be more
> > > natural
> > > >>> for such cases. Another example is index support. A table may have
> a
> > > >>> number
> > > >>> of access methods in addition to plain scan, and every such method
> > may
> > > >>> produce a physical scan with different collations. It is not
> > practical
> > > to
> > > >>> try to enforce something from the top. Instead, we'd better produce
> > > >>> alternatives from the bottom.
> > > >>>
> > > >>> Basically, Drill tries to mix two approaches. First, it tries to
> > derive
> > > >>> traits from the child. If it fails and no transformations were
> > created,
> > > >>> it
> > > >>> just stops trait propagation. As a result, an exchange is created
> on
> > > top
> > > >>> of
> > > >>> the operator where we stopped, which again leads to not optimal
> > plans.
> > > >>> You
> > > >>> may observe this in action if you run my reproducer. "testPass"
> > > produces
> > > >>> an
> > > >>> optimal plan with help of abstract converters. "testDrill" produces
> > not
> > > >>> optimal plan with the unnecessary exchange.
> > > >>>
> > > >>> All in all, I cannot get how we can employ distribution metadata
> and
> > > >>> index
> > > >>> metadata for optimal planning without using a pure bottom-up
> > approach.
> > > >>>
> > > >>> Regards,
> > > >>> Vladimir.
> > > >>>
> > > >>> пт, 1 нояб. 2019 г. в 11:42, Stamatis Zampetakis <
> zabetak@gmail.com
> > >:
> > > >>>
> > > >>> > The discussion is interesting thanks for the nice examples.
> > > >>> > I am replying in this thread since it has all the examples and
> the
> > > >>> history
> > > >>> > of the conversation.
> > > >>> >
> > > >>> > *Part I: Distribution physical or logical property*
> > > >>> >
> > > >>> > The Volcano paper writes the following regarding properties:
> > > >>> >
> > > >>> > "Logical properties can be derived from the logical algebra
> > > expression
> > > >>> and
> > > >>> > include schema, expected size, etc., while physical properties
> > depend
> > > >>> on
> > > >>> > algorithms, e.g., sort order, partitioning, etc."
> > > >>> >
> > > >>> > We could say that partitioning is not only a property which
> depends
> > > on
> > > >>> an
> > > >>> > algorithm but a property which depends on the schema. When you
> > have a
> > > >>> table
> > > >>> > you know in advance that the particular table is replicated.
> > > >>> >
> > > >>> > Basically, this means that the starting "logical" plan in your
> case
> > > >>> could
> > > >>> > be the following:
> > > >>> > RootLogicalRel[convention=LOGICAL, distribution=ANY]
> > > >>> > -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
> > > >>> > -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
> > > >>> >
> > > >>> > When you are about to create your physical projection it is not
> > > >>> necessary
> > > >>> > to create three equivalent paths but the most promising by asking
> > the
> > > >>> input
> > > >>> > what can it offer as traits. Again this partially overlaps with
> the
> > > >>> > discussion about "On demand traitset" [1] where as I wrote there
> is
> > > >>> already
> > > >>> > a mechanism to derive traits from children (collation,
> > distribution,
> > > >>> etc.).
> > > >>> >
> > > >>> > Note that I am not saying that there is nothing more to be done
> to
> > > >>> improve
> > > >>> > the Calcite optimizer; I am just trying to provide a viable
> > > alternative
> > > >>> > given the current situation of the project.
> > > >>> >
> > > >>> > *Part II: Example with the original Volcano optimizer*
> > > >>> >
> > > >>> > Now let me try to show how the original (from the paper) Volcano
> > > >>> optimizer
> > > >>> > could handle your query.
> > > >>> >
> > > >>> > The optimization starts and we request two properties convention
> > and
> > > >>> > distribution.
> > > >>> >
> > > >>> > RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
> > > >>> > -> ProjectLogicalRel
> > > >>> > -> MapScanLogicalRel
> > > >>> >
> > > >>> > Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm
> but
> > > >>> cannot
> > > >>> > satisfy the SINGLETON property so it propagates the demand
> > > >>> >
> > > >>> > RootPhysicalRel
> > > >>> > -> ProjectLogicalRel [convention=PHYSICAL,
> distribution=SINGLETON]
> > > >>> > -> MapScanLogicalRel
> > > >>> >
> > > >>> > Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the
> > algorithm
> > > >>> but
> > > >>> > cannot satisfy the SINGLETON property so it propagates the demand
> > > >>> >
> > > >>> > RootPhysicalRel
> > > >>> > -> ProjectPhysicalRel
> > > >>> > -> MapScanLogicalRel [convention=PHYSICAL,
> distribution=SINGLETON]
> > > >>> >
> > > >>> > Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the
> > algorithm
> > > >>> and
> > > >>> > given that the table is replicated it satisfies the distribution
> > > >>> property.
> > > >>> >
> > > >>> > RootPhysicalRel
> > > >>> > -> ProjectPhysicalRel
> > > >>> > -> MapScanPhysicalRel
> > > >>> >
> > > >>> > So we end up with a complete plan that satisfies all properties
> and
> > > >>> does
> > > >>> > not have redundant exchange operators.
> > > >>> > Moreover we didn't require all possible distributions when we
> > applied
> > > >>> Rule
> > > >>> > 2 since we just want to satisfy the SINGLETON property requested
> by
> > > the
> > > >>> > parent.
> > > >>> > The example above does not demonstrate the additional
> optimization
> > > >>> paths
> > > >>> > which would be apply an enforcer (an Exchange operator to satisfy
> > the
> > > >>> > SINGLETON property) at a higher level than the scan.
> > > >>> >
> > > >>> > Would this be an acceptable approach? Does it still suffer from a
> > big
> > > >>> > search space?
> > > >>> >
> > > >>> > To make the analog with the actual Volcano optimizer in Calcite
> the
> > > >>> thing
> > > >>> > that may be missing is to be able to know what trait was
> requested
> > by
> > > >>> the
> > > >>> > parent during the application of Rule 2.
> > > >>> >
> > > >>> > Best,
> > > >>> > Stamatis
> > > >>> >
> > > >>> > [1]
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
> > > >>> >
> > > >>> > On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <
> > ppozerov@gmail.com>
> > > >>> > wrote:
> > > >>> >
> > > >>> > > Hi Stamatis,
> > > >>> > >
> > > >>> > > The problem that the presented reproducer is a very simplified
> > > >>> version of
> > > >>> > > what is actually needed. In reality, there are several
> > distribution
> > > >>> > types -
> > > >>> > > PARTITIONED, REPLICATED, SINGLETON. To make things worse,
> > > >>> PARTITIONED may
> > > >>> > > or may not have concrete distribution fields. In theory, I can
> > > >>> create one
> > > >>> > > transformation per distribution type, but that would increase
> > plan
> > > >>> space
> > > >>> > > significantly. In my sample "Root <- Project <- Scan" plan,
> there
> > > is
> > > >>> no
> > > >>> > > room for optimization at all, we only need to convert the nodes
> > and
> > > >>> > > propagate traits. But if I follow the proposed approach, the
> > > planner
> > > >>> > would
> > > >>> > > create three equivalent paths. For complex queries, this may
> > > increase
> > > >>> > > optimization time significantly.
> > > >>> > >
> > > >>> > > What I need instead is to gradually convert and optimize nodes
> > > >>> > *bottom-up*,
> > > >>> > > instead of top-bottom. That is, create a physical scan first,
> > then
> > > >>> > create a
> > > >>> > > physical project on top of it, etc. But at the same time, some
> > > rules
> > > >>> > still
> > > >>> > > require the top-bottom approach. So essentially I need the
> > > optimizer
> > > >>> to
> > > >>> > do
> > > >>> > > both. Abstract converters help me establish bottom-up
> preparation
> > > >>> but do
> > > >>> > > this at the cost of considering too many trait pairs, and as a
> > > >>> result,
> > > >>> > also
> > > >>> > > pollute the search space.
> > > >>> > >
> > > >>> > > To the contrast, precise command "I transformed the node A to
> > node
> > > B,
> > > >>> > > please re-trigger the rules for A's parents" would allow us to
> > > >>> re-trigger
> > > >>> > > only required rules, without adding more nodes.
> > > >>> > >
> > > >>> > > Does it make sense?
> > > >>> > >
> > > >>> > > Regards,
> > > >>> > > Vladimir.
> > > >>> > >
> > > >>> > > ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <
> > > zabetak@gmail.com
> > > >>> >:
> > > >>> > >
> > > >>> > > > I admit that I didn't thoroughly read the exchanges so far
> but
> > > >>> forcing
> > > >>> > > the
> > > >>> > > > rules trigger does not seem the right approach.
> > > >>> > > >
> > > >>> > > > Other than that I would like to backtrack a bit to point 4.3
> > > >>> raised by
> > > >>> > > > Vladimir.
> > > >>> > > >
> > > >>> > > > "ProjectPhysicalRule [6] - transforms logical project to
> > physical
> > > >>> > > > project *ONLY* if there is an underlying physical input with
> > > >>> REPLICATED
> > > >>> > > or
> > > >>> > > > SINGLETON distribution"
> > > >>> > > >
> > > >>> > > > The rule could be modified to do the following two
> > > transformations:
> > > >>> > > > 1. Create a physical project and require the input to be
> > > >>> REPLICATED.
> > > >>> > > > 2. Create a physical project and require
> > > >>> > > > the input to be SINGLETON.
> > > >>> > > >
> > > >>> > > > I would assume that afterwards when your scan rule fires it
> > > should
> > > >>> go
> > > >>> > to
> > > >>> > > > the appropriate subset and give you back the desired plan. Is
> > > >>> there a
> > > >>> > > > problem with this approach?
> > > >>> > > >
> > > >>> > > > Best,
> > > >>> > > > Stamatis
> > > >>> > > >
> > > >>> > > > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <
> > > >>> gvvinblade@gmail.com>
> > > >>> > > > wrote:
> > > >>> > > >
> > > >>> > > > > Unfortunately it requires package-private API usage.
> > > >>> > > > >
> > > >>> > > > > Best solution there if Calcite provide it for end users.
> > > >>> > > > >
> > > >>> > > > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <
> > ppozerov@gmail.com
> > > >:
> > > >>> > > > >
> > > >>> > > > > > “e pension” == “expand conversion” :)
> > > >>> > > > > >
> > > >>> > > > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <
> > > >>> ppozerov@gmail.com>:
> > > >>> > > > > >
> > > >>> > > > > > > Yes, that may work. Even if e pension rule is used, for
> > the
> > > >>> most
> > > >>> > > > cases
> > > >>> > > > > it
> > > >>> > > > > > > will not trigger any real conversions, since we are
> > moving
> > > >>> from
> > > >>> > > > > abstract
> > > >>> > > > > > > convention to physical, and created converters will
> have
> > > the
> > > >>> > > opposite
> > > >>> > > > > > trait
> > > >>> > > > > > > direction (from physical to abstract).
> > > >>> > > > > > >
> > > >>> > > > > > > But again - ideally we only need to re-trigger the
> rules
> > > for
> > > >>> a
> > > >>> > > > specific
> > > >>> > > > > > > node, no more than that. So API support like
> > > >>> > > > > > > “VolcanoPlanner.forceRules(RelNode)” would be very
> > > >>> convenient.
> > > >>> > > > > > >
> > > >>> > > > > > > What do you think?
> > > >>> > > > > > >
> > > >>> > > > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
> > > >>> > > gvvinblade@gmail.com
> > > >>> > > > >:
> > > >>> > > > > > >
> > > >>> > > > > > >> I considered manual rules calling too, for now we use
> > > >>> abstract
> > > >>> > > > > > converters
> > > >>> > > > > > >> +
> > > >>> > > > > > >> ExpandConversionRule for exchanges producing.
> > > >>> > > > > > >>
> > > >>> > > > > > >> You may create such converters manually (checking
> > > >>> appropriate
> > > >>> > > > subset)
> > > >>> > > > > > this
> > > >>> > > > > > >> case you may reduce created converters count, also, a
> > > >>> converter
> > > >>> > > is a
> > > >>> > > > > > quite
> > > >>> > > > > > >> special node, that does almost nothing (without
> > > >>> corresponding
> > > >>> > > rule)
> > > >>> > > > it
> > > >>> > > > > > may
> > > >>> > > > > > >> be used just as a rule trigger.
> > > >>> > > > > > >>
> > > >>> > > > > > >> Regards,
> > > >>> > > > > > >> Igor
> > > >>> > > > > > >>
> > > >>> > > > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <
> > > >>> ppozerov@gmail.com
> > > >>> > >:
> > > >>> > > > > > >>
> > > >>> > > > > > >> > One funny hack which helped me is manual
> registration
> > > of a
> > > >>> > fake
> > > >>> > > > > > RelNode
> > > >>> > > > > > >> > with desired traits through
> VolcanoPlanner.register()
> > > >>> method.
> > > >>> > > But
> > > >>> > > > > > again,
> > > >>> > > > > > >> > this leads to trashing. What could really help is a
> > call
> > > >>> to
> > > >>> > > > > > >> > VolcanoPlanner.fireRules() with desired rel. But
> this
> > > >>> doesn't
> > > >>> > > work
> > > >>> > > > > out
> > > >>> > > > > > >> of
> > > >>> > > > > > >> > the box since some internals of the rule queue needs
> > to
> > > be
> > > >>> > > > adjusted.
> > > >>> > > > > > >> >
> > > >>> > > > > > >> > What does the community think about adding a method
> > > which
> > > >>> will
> > > >>> > > > > re-add
> > > >>> > > > > > >> rules
> > > >>> > > > > > >> > applicable to the specific RelNode to the rule
> queue?
> > > >>> > > > > > >> >
> > > >>> > > > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
> > > >>> > > ppozerov@gmail.com
> > > >>> > > > >:
> > > >>> > > > > > >> >
> > > >>> > > > > > >> > > Hi Igor,
> > > >>> > > > > > >> > >
> > > >>> > > > > > >> > > Yes, I came to the same conclusion, thank you.
> This
> > is
> > > >>> how
> > > >>> > it
> > > >>> > > > > > >> basically
> > > >>> > > > > > >> > > happens when converters are disabled:
> > > >>> > > > > > >> > > 1) We start with initial tree: [LogicalProject] <-
> > > >>> > > [LogicalScan]
> > > >>> > > > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so
> > it
> > > is
> > > >>> > added
> > > >>> > > > to
> > > >>> > > > > > the
> > > >>> > > > > > >> > > set: [LogicalProject] <- [LogicalScan,
> PhysicalScan]
> > > >>> > > > > > >> > > 3) Finally, when it is time to fire a rule for
> > > >>> PhysicalScan,
> > > >>> > > we
> > > >>> > > > > try
> > > >>> > > > > > to
> > > >>> > > > > > >> > get
> > > >>> > > > > > >> > > parents of that scan set with traits of the
> > > >>> PhysicalScan.
> > > >>> > > Since
> > > >>> > > > > > there
> > > >>> > > > > > >> are
> > > >>> > > > > > >> > > no such parents (we skipped it intentionally), the
> > > rule
> > > >>> is
> > > >>> > not
> > > >>> > > > > > queued.
> > > >>> > > > > > >> > >
> > > >>> > > > > > >> > > But when converters are enabled, a converter rel
> is
> > > >>> created:
> > > >>> > > > > > >> > [LogicalProject]
> > > >>> > > > > > >> > > <- [LogicalScan, PhysicalScan,
> > > >>> > > ConverterFromPhysicalToLogical].
> > > >>> > > > No
> > > >>> > > > > > >> rules
> > > >>> > > > > > >> > > are fired for PhysicalScan again, but they are
> fired
> > > for
> > > >>> > > > converter
> > > >>> > > > > > >> since
> > > >>> > > > > > >> > > it has the necessary LOGICAL trait.
> > > >>> > > > > > >> > >
> > > >>> > > > > > >> > > It makes sense, that converters essentially allow
> > > >>> forcing
> > > >>> > rule
> > > >>> > > > > > >> invocation
> > > >>> > > > > > >> > > on parents, even if the child was created with
> > > different
> > > >>> > > traits.
> > > >>> > > > > But
> > > >>> > > > > > >> it
> > > >>> > > > > > >> > > seems that this mechanism may be too heavy for
> > complex
> > > >>> > queries
> > > >>> > > > > > >> because it
> > > >>> > > > > > >> > > literally creates hundreds of new converter rels
> and
> > > >>> > triggers
> > > >>> > > > > rules
> > > >>> > > > > > >> over
> > > >>> > > > > > >> > > and over again.
> > > >>> > > > > > >> > >
> > > >>> > > > > > >> > > We need some fine-grained alternative. Basically,
> > what
> > > >>> would
> > > >>> > > be
> > > >>> > > > > > enough
> > > >>> > > > > > >> > for
> > > >>> > > > > > >> > > me is to let the planner know somehow: "I created
> > that
> > > >>> rel,
> > > >>> > > and
> > > >>> > > > I
> > > >>> > > > > > want
> > > >>> > > > > > >> > you
> > > >>> > > > > > >> > > to execute parent rules not only using its trait
> but
> > > >>> also on
> > > >>> > > > this
> > > >>> > > > > > and
> > > >>> > > > > > >> > those
> > > >>> > > > > > >> > > traits."
> > > >>> > > > > > >> > > Is there any API in Calcite which allows doing
> this
> > > >>> without
> > > >>> > > > > > creating a
> > > >>> > > > > > >> > new
> > > >>> > > > > > >> > > rel node?
> > > >>> > > > > > >> > >
> > > >>> > > > > > >> > > Regards,
> > > >>> > > > > > >> > > Vladimir.
> > > >>> > > > > > >> > >
> > > >>> > > > > > >> > >
> > > >>> > > > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> > > >>> > > > > gvvinblade@gmail.com
> > > >>> > > > > > >:
> > > >>> > > > > > >> > >
> > > >>> > > > > > >> > >> Vladimir,
> > > >>> > > > > > >> > >>
> > > >>> > > > > > >> > >> Probably it'll help you:
> > > >>> > > > > > >> > >>
> > > >>> > > > > > >> > >> Seems the cause of issue in
> > RelSubset.getParentRels()
> > > >>> The
> > > >>> > > > check
> > > >>> > > > > > used
> > > >>> > > > > > >> > when
> > > >>> > > > > > >> > >> the planner schedules newly matched rules after
> > > >>> successful
> > > >>> > > > > > >> > transformation
> > > >>> > > > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents
> the
> > > >>> parent
> > > >>> > > rule
> > > >>> > > > > be
> > > >>> > > > > > >> > applied
> > > >>> > > > > > >> > >> once again (here your logical project with an
> input
> > > >>> having
> > > >>> > > ANY
> > > >>> > > > > > >> > >> distribution
> > > >>> > > > > > >> > >> doesn't satisfy a transformed input traits).
> > > >>> > > > > > >> > >>
> > > >>> > > > > > >> > >> In our case we use another workaround, so there
> are
> > > >>> also
> > > >>> > much
> > > >>> > > > > more
> > > >>> > > > > > >> > >> transformations than we wanted, so the desired
> rule
> > > is
> > > >>> > > > triggered.
> > > >>> > > > > > >> > >>
> > > >>> > > > > > >> > >>
> > > >>> > > > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
> > > >>> > > ppozerov@gmail.com
> > > >>> > > > >:
> > > >>> > > > > > >> > >>
> > > >>> > > > > > >> > >> > Hi Vladimir,
> > > >>> > > > > > >> > >> >
> > > >>> > > > > > >> > >> > I am sorry. Pushed, it works now.
> > > >>> > > > > > >> > >> >
> > > >>> > > > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov
> <
> > > >>> > > > > > >> > >> > sitnikov.vladimir@gmail.com
> > > >>> > > > > > >> > >> > >:
> > > >>> > > > > > >> > >> >
> > > >>> > > > > > >> > >> > > > mvn clean test
> > > >>> > > > > > >> > >> > >
> > > >>> > > > > > >> > >> > > [ERROR] The goal you specified requires a
> > project
> > > >>> to
> > > >>> > > > execute
> > > >>> > > > > > but
> > > >>> > > > > > >> > >> there is
> > > >>> > > > > > >> > >> > > no POM in this directory
> > > >>> > > > > > >> > >> > >
> > > >>> > > > > > >> > >> > > Vladimir, please push missing files
> > > >>> > > > > > >> > >> > >
> > > >>> > > > > > >> > >> > > Vladimir
> > > >>> > > > > > >> > >> > >
> > > >>> > > > > > >> > >> >
> > > >>> > > > > > >> > >>
> > > >>> > > > > > >> > >
> > > >>> > > > > > >> >
> > > >>> > > > > > >>
> > > >>> > > > > > >
> > > >>> > > > > >
> > > >>> > > > >
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>>
> > > >>
> > >
> >
>

Re: Re: Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
Hi Stamatis,

This approach worth trying, but I am afraid that it may produce too many
permutations. For example, this is the logical plan of one of TPC-H
benchmarks, and I am not very keen expanding it much further :-)

901:ProjectLogicalRel(ps_partkey=[$0], val=[$1])
  899:FilterLogicalRel(subset=[rel#900:Subset#66.LOGICAL.ANY.[]],
condition=[>($1, $2)])
    897:JoinLogicalRel(subset=[rel#898:Subset#65.LOGICAL.ANY.[]],
condition=[true], joinType=[left])
      882:AggregateLogicalRel(subset=[rel#883:Subset#57.LOGICAL.ANY.[]],
group=[{0}], val=[SUM($1)])
        880:ProjectLogicalRel(subset=[rel#881:Subset#56.LOGICAL.ANY.[]],
ps_partkey=[$1], $f1=[*($2, $3)])
          878:JoinLogicalRel(subset=[rel#879:Subset#55.LOGICAL.ANY.[]],
condition=[=($5, $6)], joinType=[inner])
            875:JoinLogicalRel(subset=[rel#876:Subset#53.LOGICAL.ANY.[]],
condition=[=($0, $4)], joinType=[inner])

634:MapScanLogicalRel(subset=[rel#873:Subset#51.LOGICAL.ANY.[]],
table=[[partsupp]], projects=[[0, 1, 2, 3]])

632:MapScanLogicalRel(subset=[rel#874:Subset#52.LOGICAL.ANY.[]],
table=[[supplier]], projects=[[0, 1]])

630:MapScanLogicalRel(subset=[rel#877:Subset#54.LOGICAL.ANY.[]],
table=[[nation]], projects=[[0, 1]], filter=[=($1, '[NATION]')])
      895:AggregateLogicalRel(subset=[rel#896:Subset#64.LOGICAL.ANY.[]],
group=[{}], agg#0=[SINGLE_VALUE($0)])
        893:ProjectLogicalRel(subset=[rel#894:Subset#63.LOGICAL.ANY.[]],
EXPR$0=[*($0, 0.9:DECIMAL(2, 1))])

891:AggregateLogicalRel(subset=[rel#892:Subset#62.LOGICAL.ANY.[]],
group=[{}], agg#0=[SUM($0)])

889:ProjectLogicalRel(subset=[rel#890:Subset#61.LOGICAL.ANY.[]], $f0=[*($1,
$2)])
              887:JoinLogicalRel(subset=[rel#888:Subset#60.LOGICAL.ANY.[]],
condition=[=($4, $5)], joinType=[inner])

885:JoinLogicalRel(subset=[rel#886:Subset#59.LOGICAL.ANY.[]],
condition=[=($0, $3)], joinType=[inner])

680:MapScanLogicalRel(subset=[rel#884:Subset#58.LOGICAL.ANY.[]],
table=[[partsupp]], projects=[[0, 2, 3]])

632:MapScanLogicalRel(subset=[rel#874:Subset#52.LOGICAL.ANY.[]],
table=[[supplier]], projects=[[0, 1]])

630:MapScanLogicalRel(subset=[rel#877:Subset#54.LOGICAL.ANY.[]],
table=[[nation]], projects=[[0, 1]], filter=[=($1, '[NATION]')])

I have two working solutions at the moment, which do not require
cross-convention converters:
1) Create separate rules for logical-physical node conversion, and for
physical-physical optimization
2) A kind of hack: add fake AbstractConverter with logical traits after the
physical node is created

I hope one of them will work good enough for the production case. Most
likely it would be p.1.

Regards,
Vladimir.


пт, 15 нояб. 2019 г. в 11:04, Stamatis Zampetakis <za...@gmail.com>:

> Hi again Vladimir,
>
> In my previous email, I mentioned three rules for performing logical to
> physical conversions.
> It's normal that if you add more operators we will end up with more rules.
> Now that the example has a filter we have the following rule list:
>
> Rule 1: RootLogicalRel -> RootPhysicalRel
> Rule 2: ProjectLogicalRel -> ProjectPhysicalRel
> Rule 3: FilterLogicalRel -> FilterPhysicalRel
> Rule 4: MapScanLogicalRel -> MapScanPhysicalRel
>
> If I understand well, your concern is that in order to get the optimal plan
> you would have to introduce many additional rules for the Exchange operator
> (e.g., ExchangeProjectTransposeRule, ExchangeFilterTransposeRue, etc.).
> I was thinking that it is not necessary to introduce these kind of rules.
> You could have only one rule, i.e., PhysicaExchangeRule, for enforcing a
> distribution that would be applied when there is a requirement for
> particular distribution.
>
> So for the initial plan
> LogicalRoot [distribution=SINGLETON]
> -> LogicalProject
>  -> LogicalFilter
>   -> LogicalScan
>
> The ruleset above should generate the following alternatives where the
> PhysicalExchangeRule is applied 3 times.
>
> Alt1:
> PhysicalRoot
> -> SingletonExchange
>  -> PhysicalProject
>   -> PhysicalFilter
>    -> PhysicalScan
>
> Alt2:
> PhysicalRoot
> -> PhysicalProject
>  -> SingletonExchange
>   -> PhysicalFilter
>    -> PhysicalScan
>
> Alt3:
> PhysicalRoot
> -> PhysicalProject
>  -> PhysicalFilter
>   -> SingletonExchange
>    -> PhysicalScan
>
> This is still in the same spirit of the previous example where the physical
> property (distribution) is either satisfied immediately or passed down to
> the next level.
>
> Best,
> Stamatis
>
> On Thu, Nov 14, 2019 at 11:14 AM Vladimir Ozerov <pp...@gmail.com>
> wrote:
>
> > search place* = search space
> >
> > чт, 14 нояб. 2019 г. в 13:10, Vladimir Ozerov <pp...@gmail.com>:
> >
> > > Hi Haisheng,
> > >
> > > I double-checked the code. My original version returned false for some
> > > cases, but it didn't affect number of rules calls anyway, so I changed
> it
> > > to always return true. Please note that if I change the code as you
> > > suggested, the test started failing, because bottom-up propagation of
> > rule
> > > calls no longer work: when the child is converted to physical form, the
> > > parent logical node is not notified. This is the very problem I address
> > > with that weird physical-to-logical conversions: they do not make
> sense,
> > > and converter expansion does not produce any new rels, but their
> > existence
> > > allow for logical rule re-trigger which ultimately allow the plan to
> > > compile.
> > >
> > > Regarding two conventions - I agree that it may look strange, but I do
> > not
> > > see any problems from the correctness perspective. Separation of
> logical
> > > and physical planning helps me avoid excessive expansion of the search
> > > place: I do not want my physical rules to produce new rels from not
> > > optimized logical rels. Do you see any problems with that approach?
> > >
> > > Regards,
> > > Vladimir
> > >
> > > ср, 6 нояб. 2019 г. в 03:52, Haisheng Yuan <h....@alibaba-inc.com>:
> > >
> > >> Hi Vladimir,
> > >>
> > >> The code in PHYSICAL convention L44 looks weird, I think it always
> > >> returns true.
> > >>
> > >>
> >
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/HazelcastConventions.java#L44
> > >>
> > >> Try this:
> > >>
> > >> fromTraits.containsIfApplicable(Convention.PHYSICAL)
> > >>     && toTraits.containsIfApplicable(Convention.PHYSICAL);
> > >>
> > >>
> > >> Adding a AbstractConverter on logical operators is meaningless.
> Calcite
> > >> is mixing the concept of logical and physical together, which is sad.
> > >>
> > >> BTW, using 2 conventions is not appropriate and wrong.
> > >>
> > >> - Haisheng
> > >>
> > >> ------------------------------------------------------------------
> > >> 发件人:Vladimir Ozerov<pp...@gmail.com>
> > >> 日 期:2019年11月05日 18:02:15
> > >> 收件人:Haisheng Yuan<h....@alibaba-inc.com>
> > >> 抄 送:dev@calcite.apache.org (dev@calcite.apache.org)<
> > >> dev@calcite.apache.org>
> > >> 主 题:Re: Re: Problem with converters and possibly rule matching
> > >>
> > >> Hi Haisheng,
> > >>
> > >>  think I already tried something very similar to what you explained,
> but
> > >> it gave not an optimal plan. Please let me describe what I did. I
> would
> > >> appreciate your feedback.
> > >>
> > >> 1) We start with a simple operator tree Root <- Project <- Scan, where
> > >> the root is a final aggregator in the distributed query engine:
> > >> -> LogicalRoot
> > >>  -> LogicalProject
> > >>   -> LogicalScan
> > >>
> > >> 2) First, we convert the Root and enforce SINGLETON distribution on a
> > >> child:
> > >> *-> PhysicalRoot[SINGLETON]*
> > >> * -> Enforcer#1[SINGLETON]*
> > >>   -> LogicalProject
> > >>    -> LogicalScan
> > >>
> > >> 3) Then the project's rule is invoked. It doesn't know the
> distribution
> > >> of the input, so it requests ANY distribution. Note that we have to
> set
> > ANY
> > >> to the project as well since we do not know the distribution of the
> > input:
> > >> -> PhysicalRoot[SINGLETON]
> > >>  -> Enforcer#1[SINGLETON]
> > >> *  -> PhysicalProject[ANY]*
> > >> *   -> Enforcer#2[ANY]*
> > >>     -> LogicalScan
> > >>
> > >> 4) Finally, the physical scan is created and its distribution is
> > >> resolved. Suppose that it is REPLICATED, i.e. the whole result set is
> > >> located on all nodes.
> > >> -> PhysicalRoot[SINGLETON]
> > >>  -> Enforcer#1[SINGLETON]
> > >>   -> PhysicalProject[ANY]
> > >>    -> Enforcer#2[ANY]
> > >> *    -> PhysicalScan[REPLICATED]*
> > >>
> > >> 5) Now as all logical nodes are converted, we start resolving
> enforcers.
> > >> The second one is no-op, since REPLICATED satisfies ANY:
> > >> -> PhysicalRoot[SINGLETON]
> > >>  -> Enforcer#1[SINGLETON]
> > >>   -> PhysicalProject[ANY]
> > >>    -> PhysicalScan[REPLICATED]
> > >>
> > >> 6) But the first enforcer now requires an Exchange, since ANY doesn't
> > >> satisfy SINGLETON!
> > >> -> PhysicalRoot[SINGLETON]
> > >> * -> SingletonExchange[SINGLETON]*
> > >>   -> PhysicalProject[ANY] // <= unresolved!
> > >>    -> PhysicalScan[REPLICATED]
> > >>
> > >> The resulting plan requires data movement only because we didn't know
> > >> precise distribution of the PhysicalProject when it was created. But
> > should
> > >> I enable Convention.Impl.canConvertConvention, bottom-up propagation
> > >> kicks in, and the correct plan is produced because now LogicalProject
> > >> has a chance to be converted to PhysicalProject with the concrete
> > >> distribution. The optimized plan looks like this (since REPLICATED
> > >> satisfies SINGLETON):
> > >> -> PhysicalRoot[SINGLETON]
> > >>  -> PhysicalProject[REPLICATED]
> > >>   -> PhysicalScan[REPLICATED]
> > >>
> > >> You may see this in action in my reproducer:
> > >> 1) Test producing "bad" plan:
> > >>
> >
> https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L45
> > >> 2) Root enforces SINGLETON on Project:
> > >>
> >
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/RootPhysicalRule.java#L45
> > >> 3) Project enforces default (ANY) distribution on Scan:
> > >>
> >
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L49
> > >>
> > >> Please let me know if this flow is similar to what you meant.
> > >>
> > >> Regards,
> > >> Vladimir.
> > >>
> > >> пн, 4 нояб. 2019 г. в 10:33, Haisheng Yuan <h....@alibaba-inc.com>:
> > >>
> > >>> Hi Vladimir,
> > >>>
> > >>> This is still can be done through top-down request approach.
> > >>>
> > >>> PhysicalFilter operator should request ANY distribution from child
> > >>> operator, unless there is outer reference in the filter condition, in
> > which
> > >>> case, PhysicalFilter should request SINGLETON or BROADCAST
> > distribution. So
> > >>> in your case, PhysicalFilter request ANY, its required distribution
> > will be
> > >>> enforced on filter's output.
> > >>>
> > >>> Regarding index usage, you should have a FIlterTableScan2IndexGet
> > >>> logical transformation rule, and a IndexGet2IndexScan physical
> > >>> implementation rule. Note that IndexGet is a logical operator and
> > IndexScan
> > >>> is a physical operator, which are also used by SQL Server.
> > >>>
> > >>> - Haisheng
> > >>>
> > >>> ------------------------------------------------------------------
> > >>> 发件人:Vladimir Ozerov<pp...@gmail.com>
> > >>> 日 期:2019年11月01日 17:30:26
> > >>> 收件人:<de...@calcite.apache.org>
> > >>> 主 题:Re: Problem with converters and possibly rule matching
> > >>>
> > >>> Hi Stamatis,
> > >>>
> > >>> Thank you for your reply. I also thought that we may set the
> > distribution
> > >>> trait during logical planning because it is known in advance. And the
> > >>> example I gave will work! :-) But unfortunately, it will work only
> > >>> because
> > >>> the tree is very simple, and the Project is adjacent to the Scan.
> This
> > is
> > >>> how my reproducer will work in that case:
> > >>> 1) Root: enforce "SINGLETON" on Project
> > >>> 2) Project: check the logical Scan, infer already resolved
> > distribution,
> > >>> then convert to [PhyiscalProject <- PhysicalScan]
> > >>> 3) Resolve Root enforcer, adding and Exchange if needed.
> > >>>
> > >>> But this stops working as soon as a plan becomes more complex so that
> > it
> > >>> is
> > >>> impossible to infer the distribution from the child immediately.
> E.g.:
> > >>> LogicalRoot [distribution=SINGLETON]
> > >>> -> LogicalProject // We are here new and cannot produce the physical
> > >>> project
> > >>> -> LogicalFilter[distribution=?]
> > >>> -> LogicalScan[distribution=REPLICATED]
> > >>>
> > >>> This is where your suggestion with cascading enforcement may kick in.
> > But
> > >>> now consider that instead of having REPLICATED distribution, which
> > >>> satisfies SINGLETON, we have a PARTITIONED distribution, It doesn't
> > >>> satisfy
> > >>> SINGLETON, so we have to insert the exchange.
> > >>>
> > >>> Before:
> > >>> PhysicalRoot [SINGLETON]
> > >>> -> PhysicalProject [SINGLETON]
> > >>> -> PhysicalFilter [SINGLETON]
> > >>> -> LogicalScan [PARTITIONED] // SINGLETON is enforced here
> > >>>
> > >>> After:
> > >>> PhysicalRoot [SINGLETON]
> > >>> -> PhysicalProject [SINGLETON]
> > >>> -> PhysicalFilter [SINGLETON]
> > >>> -> SingletonExchange [SINGLETON]
> > >>> -> PhysicalScan [PARTITIONED]
> > >>>
> > >>> But unfortunately, this plan is not optimal. Since we perform Project
> > and
> > >>> Filter after the exchange, we now have to reshuffle the whole table.
> > The
> > >>> optimal plan would be to pushdown Project and Filter past exchange:
> > >>> PhysicalRoot [SINGLETON]
> > >>> -> SingletonExchange [SINGLETON]
> > >>> -> PhysicalProject [PARTITIONED]
> > >>> -> PhysicalFilter [PARTITIONED]
> > >>> -> PhysicalScan [PARTITIONED]
> > >>>
> > >>> But in order to achieve that, now I need to introduce new rules which
> > >>> will
> > >>> attempt to transpose dozens of different operators past exchange, so
> > most
> > >>> likely the planning will never finish :-)
> > >>>
> > >>> This is why it seems that the bottom-up approach seems to be more
> > natural
> > >>> for such cases. Another example is index support. A table may have a
> > >>> number
> > >>> of access methods in addition to plain scan, and every such method
> may
> > >>> produce a physical scan with different collations. It is not
> practical
> > to
> > >>> try to enforce something from the top. Instead, we'd better produce
> > >>> alternatives from the bottom.
> > >>>
> > >>> Basically, Drill tries to mix two approaches. First, it tries to
> derive
> > >>> traits from the child. If it fails and no transformations were
> created,
> > >>> it
> > >>> just stops trait propagation. As a result, an exchange is created on
> > top
> > >>> of
> > >>> the operator where we stopped, which again leads to not optimal
> plans.
> > >>> You
> > >>> may observe this in action if you run my reproducer. "testPass"
> > produces
> > >>> an
> > >>> optimal plan with help of abstract converters. "testDrill" produces
> not
> > >>> optimal plan with the unnecessary exchange.
> > >>>
> > >>> All in all, I cannot get how we can employ distribution metadata and
> > >>> index
> > >>> metadata for optimal planning without using a pure bottom-up
> approach.
> > >>>
> > >>> Regards,
> > >>> Vladimir.
> > >>>
> > >>> пт, 1 нояб. 2019 г. в 11:42, Stamatis Zampetakis <zabetak@gmail.com
> >:
> > >>>
> > >>> > The discussion is interesting thanks for the nice examples.
> > >>> > I am replying in this thread since it has all the examples and the
> > >>> history
> > >>> > of the conversation.
> > >>> >
> > >>> > *Part I: Distribution physical or logical property*
> > >>> >
> > >>> > The Volcano paper writes the following regarding properties:
> > >>> >
> > >>> > "Logical properties can be derived from the logical algebra
> > expression
> > >>> and
> > >>> > include schema, expected size, etc., while physical properties
> depend
> > >>> on
> > >>> > algorithms, e.g., sort order, partitioning, etc."
> > >>> >
> > >>> > We could say that partitioning is not only a property which depends
> > on
> > >>> an
> > >>> > algorithm but a property which depends on the schema. When you
> have a
> > >>> table
> > >>> > you know in advance that the particular table is replicated.
> > >>> >
> > >>> > Basically, this means that the starting "logical" plan in your case
> > >>> could
> > >>> > be the following:
> > >>> > RootLogicalRel[convention=LOGICAL, distribution=ANY]
> > >>> > -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
> > >>> > -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
> > >>> >
> > >>> > When you are about to create your physical projection it is not
> > >>> necessary
> > >>> > to create three equivalent paths but the most promising by asking
> the
> > >>> input
> > >>> > what can it offer as traits. Again this partially overlaps with the
> > >>> > discussion about "On demand traitset" [1] where as I wrote there is
> > >>> already
> > >>> > a mechanism to derive traits from children (collation,
> distribution,
> > >>> etc.).
> > >>> >
> > >>> > Note that I am not saying that there is nothing more to be done to
> > >>> improve
> > >>> > the Calcite optimizer; I am just trying to provide a viable
> > alternative
> > >>> > given the current situation of the project.
> > >>> >
> > >>> > *Part II: Example with the original Volcano optimizer*
> > >>> >
> > >>> > Now let me try to show how the original (from the paper) Volcano
> > >>> optimizer
> > >>> > could handle your query.
> > >>> >
> > >>> > The optimization starts and we request two properties convention
> and
> > >>> > distribution.
> > >>> >
> > >>> > RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
> > >>> > -> ProjectLogicalRel
> > >>> > -> MapScanLogicalRel
> > >>> >
> > >>> > Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but
> > >>> cannot
> > >>> > satisfy the SINGLETON property so it propagates the demand
> > >>> >
> > >>> > RootPhysicalRel
> > >>> > -> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
> > >>> > -> MapScanLogicalRel
> > >>> >
> > >>> > Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the
> algorithm
> > >>> but
> > >>> > cannot satisfy the SINGLETON property so it propagates the demand
> > >>> >
> > >>> > RootPhysicalRel
> > >>> > -> ProjectPhysicalRel
> > >>> > -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
> > >>> >
> > >>> > Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the
> algorithm
> > >>> and
> > >>> > given that the table is replicated it satisfies the distribution
> > >>> property.
> > >>> >
> > >>> > RootPhysicalRel
> > >>> > -> ProjectPhysicalRel
> > >>> > -> MapScanPhysicalRel
> > >>> >
> > >>> > So we end up with a complete plan that satisfies all properties and
> > >>> does
> > >>> > not have redundant exchange operators.
> > >>> > Moreover we didn't require all possible distributions when we
> applied
> > >>> Rule
> > >>> > 2 since we just want to satisfy the SINGLETON property requested by
> > the
> > >>> > parent.
> > >>> > The example above does not demonstrate the additional optimization
> > >>> paths
> > >>> > which would be apply an enforcer (an Exchange operator to satisfy
> the
> > >>> > SINGLETON property) at a higher level than the scan.
> > >>> >
> > >>> > Would this be an acceptable approach? Does it still suffer from a
> big
> > >>> > search space?
> > >>> >
> > >>> > To make the analog with the actual Volcano optimizer in Calcite the
> > >>> thing
> > >>> > that may be missing is to be able to know what trait was requested
> by
> > >>> the
> > >>> > parent during the application of Rule 2.
> > >>> >
> > >>> > Best,
> > >>> > Stamatis
> > >>> >
> > >>> > [1]
> > >>> >
> > >>> >
> > >>>
> >
> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
> > >>> >
> > >>> > On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <
> ppozerov@gmail.com>
> > >>> > wrote:
> > >>> >
> > >>> > > Hi Stamatis,
> > >>> > >
> > >>> > > The problem that the presented reproducer is a very simplified
> > >>> version of
> > >>> > > what is actually needed. In reality, there are several
> distribution
> > >>> > types -
> > >>> > > PARTITIONED, REPLICATED, SINGLETON. To make things worse,
> > >>> PARTITIONED may
> > >>> > > or may not have concrete distribution fields. In theory, I can
> > >>> create one
> > >>> > > transformation per distribution type, but that would increase
> plan
> > >>> space
> > >>> > > significantly. In my sample "Root <- Project <- Scan" plan, there
> > is
> > >>> no
> > >>> > > room for optimization at all, we only need to convert the nodes
> and
> > >>> > > propagate traits. But if I follow the proposed approach, the
> > planner
> > >>> > would
> > >>> > > create three equivalent paths. For complex queries, this may
> > increase
> > >>> > > optimization time significantly.
> > >>> > >
> > >>> > > What I need instead is to gradually convert and optimize nodes
> > >>> > *bottom-up*,
> > >>> > > instead of top-bottom. That is, create a physical scan first,
> then
> > >>> > create a
> > >>> > > physical project on top of it, etc. But at the same time, some
> > rules
> > >>> > still
> > >>> > > require the top-bottom approach. So essentially I need the
> > optimizer
> > >>> to
> > >>> > do
> > >>> > > both. Abstract converters help me establish bottom-up preparation
> > >>> but do
> > >>> > > this at the cost of considering too many trait pairs, and as a
> > >>> result,
> > >>> > also
> > >>> > > pollute the search space.
> > >>> > >
> > >>> > > To the contrast, precise command "I transformed the node A to
> node
> > B,
> > >>> > > please re-trigger the rules for A's parents" would allow us to
> > >>> re-trigger
> > >>> > > only required rules, without adding more nodes.
> > >>> > >
> > >>> > > Does it make sense?
> > >>> > >
> > >>> > > Regards,
> > >>> > > Vladimir.
> > >>> > >
> > >>> > > ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <
> > zabetak@gmail.com
> > >>> >:
> > >>> > >
> > >>> > > > I admit that I didn't thoroughly read the exchanges so far but
> > >>> forcing
> > >>> > > the
> > >>> > > > rules trigger does not seem the right approach.
> > >>> > > >
> > >>> > > > Other than that I would like to backtrack a bit to point 4.3
> > >>> raised by
> > >>> > > > Vladimir.
> > >>> > > >
> > >>> > > > "ProjectPhysicalRule [6] - transforms logical project to
> physical
> > >>> > > > project *ONLY* if there is an underlying physical input with
> > >>> REPLICATED
> > >>> > > or
> > >>> > > > SINGLETON distribution"
> > >>> > > >
> > >>> > > > The rule could be modified to do the following two
> > transformations:
> > >>> > > > 1. Create a physical project and require the input to be
> > >>> REPLICATED.
> > >>> > > > 2. Create a physical project and require
> > >>> > > > the input to be SINGLETON.
> > >>> > > >
> > >>> > > > I would assume that afterwards when your scan rule fires it
> > should
> > >>> go
> > >>> > to
> > >>> > > > the appropriate subset and give you back the desired plan. Is
> > >>> there a
> > >>> > > > problem with this approach?
> > >>> > > >
> > >>> > > > Best,
> > >>> > > > Stamatis
> > >>> > > >
> > >>> > > > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <
> > >>> gvvinblade@gmail.com>
> > >>> > > > wrote:
> > >>> > > >
> > >>> > > > > Unfortunately it requires package-private API usage.
> > >>> > > > >
> > >>> > > > > Best solution there if Calcite provide it for end users.
> > >>> > > > >
> > >>> > > > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <
> ppozerov@gmail.com
> > >:
> > >>> > > > >
> > >>> > > > > > “e pension” == “expand conversion” :)
> > >>> > > > > >
> > >>> > > > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <
> > >>> ppozerov@gmail.com>:
> > >>> > > > > >
> > >>> > > > > > > Yes, that may work. Even if e pension rule is used, for
> the
> > >>> most
> > >>> > > > cases
> > >>> > > > > it
> > >>> > > > > > > will not trigger any real conversions, since we are
> moving
> > >>> from
> > >>> > > > > abstract
> > >>> > > > > > > convention to physical, and created converters will have
> > the
> > >>> > > opposite
> > >>> > > > > > trait
> > >>> > > > > > > direction (from physical to abstract).
> > >>> > > > > > >
> > >>> > > > > > > But again - ideally we only need to re-trigger the rules
> > for
> > >>> a
> > >>> > > > specific
> > >>> > > > > > > node, no more than that. So API support like
> > >>> > > > > > > “VolcanoPlanner.forceRules(RelNode)” would be very
> > >>> convenient.
> > >>> > > > > > >
> > >>> > > > > > > What do you think?
> > >>> > > > > > >
> > >>> > > > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
> > >>> > > gvvinblade@gmail.com
> > >>> > > > >:
> > >>> > > > > > >
> > >>> > > > > > >> I considered manual rules calling too, for now we use
> > >>> abstract
> > >>> > > > > > converters
> > >>> > > > > > >> +
> > >>> > > > > > >> ExpandConversionRule for exchanges producing.
> > >>> > > > > > >>
> > >>> > > > > > >> You may create such converters manually (checking
> > >>> appropriate
> > >>> > > > subset)
> > >>> > > > > > this
> > >>> > > > > > >> case you may reduce created converters count, also, a
> > >>> converter
> > >>> > > is a
> > >>> > > > > > quite
> > >>> > > > > > >> special node, that does almost nothing (without
> > >>> corresponding
> > >>> > > rule)
> > >>> > > > it
> > >>> > > > > > may
> > >>> > > > > > >> be used just as a rule trigger.
> > >>> > > > > > >>
> > >>> > > > > > >> Regards,
> > >>> > > > > > >> Igor
> > >>> > > > > > >>
> > >>> > > > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <
> > >>> ppozerov@gmail.com
> > >>> > >:
> > >>> > > > > > >>
> > >>> > > > > > >> > One funny hack which helped me is manual registration
> > of a
> > >>> > fake
> > >>> > > > > > RelNode
> > >>> > > > > > >> > with desired traits through VolcanoPlanner.register()
> > >>> method.
> > >>> > > But
> > >>> > > > > > again,
> > >>> > > > > > >> > this leads to trashing. What could really help is a
> call
> > >>> to
> > >>> > > > > > >> > VolcanoPlanner.fireRules() with desired rel. But this
> > >>> doesn't
> > >>> > > work
> > >>> > > > > out
> > >>> > > > > > >> of
> > >>> > > > > > >> > the box since some internals of the rule queue needs
> to
> > be
> > >>> > > > adjusted.
> > >>> > > > > > >> >
> > >>> > > > > > >> > What does the community think about adding a method
> > which
> > >>> will
> > >>> > > > > re-add
> > >>> > > > > > >> rules
> > >>> > > > > > >> > applicable to the specific RelNode to the rule queue?
> > >>> > > > > > >> >
> > >>> > > > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
> > >>> > > ppozerov@gmail.com
> > >>> > > > >:
> > >>> > > > > > >> >
> > >>> > > > > > >> > > Hi Igor,
> > >>> > > > > > >> > >
> > >>> > > > > > >> > > Yes, I came to the same conclusion, thank you. This
> is
> > >>> how
> > >>> > it
> > >>> > > > > > >> basically
> > >>> > > > > > >> > > happens when converters are disabled:
> > >>> > > > > > >> > > 1) We start with initial tree: [LogicalProject] <-
> > >>> > > [LogicalScan]
> > >>> > > > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so
> it
> > is
> > >>> > added
> > >>> > > > to
> > >>> > > > > > the
> > >>> > > > > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > >>> > > > > > >> > > 3) Finally, when it is time to fire a rule for
> > >>> PhysicalScan,
> > >>> > > we
> > >>> > > > > try
> > >>> > > > > > to
> > >>> > > > > > >> > get
> > >>> > > > > > >> > > parents of that scan set with traits of the
> > >>> PhysicalScan.
> > >>> > > Since
> > >>> > > > > > there
> > >>> > > > > > >> are
> > >>> > > > > > >> > > no such parents (we skipped it intentionally), the
> > rule
> > >>> is
> > >>> > not
> > >>> > > > > > queued.
> > >>> > > > > > >> > >
> > >>> > > > > > >> > > But when converters are enabled, a converter rel is
> > >>> created:
> > >>> > > > > > >> > [LogicalProject]
> > >>> > > > > > >> > > <- [LogicalScan, PhysicalScan,
> > >>> > > ConverterFromPhysicalToLogical].
> > >>> > > > No
> > >>> > > > > > >> rules
> > >>> > > > > > >> > > are fired for PhysicalScan again, but they are fired
> > for
> > >>> > > > converter
> > >>> > > > > > >> since
> > >>> > > > > > >> > > it has the necessary LOGICAL trait.
> > >>> > > > > > >> > >
> > >>> > > > > > >> > > It makes sense, that converters essentially allow
> > >>> forcing
> > >>> > rule
> > >>> > > > > > >> invocation
> > >>> > > > > > >> > > on parents, even if the child was created with
> > different
> > >>> > > traits.
> > >>> > > > > But
> > >>> > > > > > >> it
> > >>> > > > > > >> > > seems that this mechanism may be too heavy for
> complex
> > >>> > queries
> > >>> > > > > > >> because it
> > >>> > > > > > >> > > literally creates hundreds of new converter rels and
> > >>> > triggers
> > >>> > > > > rules
> > >>> > > > > > >> over
> > >>> > > > > > >> > > and over again.
> > >>> > > > > > >> > >
> > >>> > > > > > >> > > We need some fine-grained alternative. Basically,
> what
> > >>> would
> > >>> > > be
> > >>> > > > > > enough
> > >>> > > > > > >> > for
> > >>> > > > > > >> > > me is to let the planner know somehow: "I created
> that
> > >>> rel,
> > >>> > > and
> > >>> > > > I
> > >>> > > > > > want
> > >>> > > > > > >> > you
> > >>> > > > > > >> > > to execute parent rules not only using its trait but
> > >>> also on
> > >>> > > > this
> > >>> > > > > > and
> > >>> > > > > > >> > those
> > >>> > > > > > >> > > traits."
> > >>> > > > > > >> > > Is there any API in Calcite which allows doing this
> > >>> without
> > >>> > > > > > creating a
> > >>> > > > > > >> > new
> > >>> > > > > > >> > > rel node?
> > >>> > > > > > >> > >
> > >>> > > > > > >> > > Regards,
> > >>> > > > > > >> > > Vladimir.
> > >>> > > > > > >> > >
> > >>> > > > > > >> > >
> > >>> > > > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> > >>> > > > > gvvinblade@gmail.com
> > >>> > > > > > >:
> > >>> > > > > > >> > >
> > >>> > > > > > >> > >> Vladimir,
> > >>> > > > > > >> > >>
> > >>> > > > > > >> > >> Probably it'll help you:
> > >>> > > > > > >> > >>
> > >>> > > > > > >> > >> Seems the cause of issue in
> RelSubset.getParentRels()
> > >>> The
> > >>> > > > check
> > >>> > > > > > used
> > >>> > > > > > >> > when
> > >>> > > > > > >> > >> the planner schedules newly matched rules after
> > >>> successful
> > >>> > > > > > >> > transformation
> > >>> > > > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the
> > >>> parent
> > >>> > > rule
> > >>> > > > > be
> > >>> > > > > > >> > applied
> > >>> > > > > > >> > >> once again (here your logical project with an input
> > >>> having
> > >>> > > ANY
> > >>> > > > > > >> > >> distribution
> > >>> > > > > > >> > >> doesn't satisfy a transformed input traits).
> > >>> > > > > > >> > >>
> > >>> > > > > > >> > >> In our case we use another workaround, so there are
> > >>> also
> > >>> > much
> > >>> > > > > more
> > >>> > > > > > >> > >> transformations than we wanted, so the desired rule
> > is
> > >>> > > > triggered.
> > >>> > > > > > >> > >>
> > >>> > > > > > >> > >>
> > >>> > > > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
> > >>> > > ppozerov@gmail.com
> > >>> > > > >:
> > >>> > > > > > >> > >>
> > >>> > > > > > >> > >> > Hi Vladimir,
> > >>> > > > > > >> > >> >
> > >>> > > > > > >> > >> > I am sorry. Pushed, it works now.
> > >>> > > > > > >> > >> >
> > >>> > > > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > >>> > > > > > >> > >> > sitnikov.vladimir@gmail.com
> > >>> > > > > > >> > >> > >:
> > >>> > > > > > >> > >> >
> > >>> > > > > > >> > >> > > > mvn clean test
> > >>> > > > > > >> > >> > >
> > >>> > > > > > >> > >> > > [ERROR] The goal you specified requires a
> project
> > >>> to
> > >>> > > > execute
> > >>> > > > > > but
> > >>> > > > > > >> > >> there is
> > >>> > > > > > >> > >> > > no POM in this directory
> > >>> > > > > > >> > >> > >
> > >>> > > > > > >> > >> > > Vladimir, please push missing files
> > >>> > > > > > >> > >> > >
> > >>> > > > > > >> > >> > > Vladimir
> > >>> > > > > > >> > >> > >
> > >>> > > > > > >> > >> >
> > >>> > > > > > >> > >>
> > >>> > > > > > >> > >
> > >>> > > > > > >> >
> > >>> > > > > > >>
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>>
> > >>
> >
>

Re: Re: Re: Problem with converters and possibly rule matching

Posted by Stamatis Zampetakis <za...@gmail.com>.
Hi again Vladimir,

In my previous email, I mentioned three rules for performing logical to
physical conversions.
It's normal that if you add more operators we will end up with more rules.
Now that the example has a filter we have the following rule list:

Rule 1: RootLogicalRel -> RootPhysicalRel
Rule 2: ProjectLogicalRel -> ProjectPhysicalRel
Rule 3: FilterLogicalRel -> FilterPhysicalRel
Rule 4: MapScanLogicalRel -> MapScanPhysicalRel

If I understand well, your concern is that in order to get the optimal plan
you would have to introduce many additional rules for the Exchange operator
(e.g., ExchangeProjectTransposeRule, ExchangeFilterTransposeRue, etc.).
I was thinking that it is not necessary to introduce these kind of rules.
You could have only one rule, i.e., PhysicaExchangeRule, for enforcing a
distribution that would be applied when there is a requirement for
particular distribution.

So for the initial plan
LogicalRoot [distribution=SINGLETON]
-> LogicalProject
 -> LogicalFilter
  -> LogicalScan

The ruleset above should generate the following alternatives where the
PhysicalExchangeRule is applied 3 times.

Alt1:
PhysicalRoot
-> SingletonExchange
 -> PhysicalProject
  -> PhysicalFilter
   -> PhysicalScan

Alt2:
PhysicalRoot
-> PhysicalProject
 -> SingletonExchange
  -> PhysicalFilter
   -> PhysicalScan

Alt3:
PhysicalRoot
-> PhysicalProject
 -> PhysicalFilter
  -> SingletonExchange
   -> PhysicalScan

This is still in the same spirit of the previous example where the physical
property (distribution) is either satisfied immediately or passed down to
the next level.

Best,
Stamatis

On Thu, Nov 14, 2019 at 11:14 AM Vladimir Ozerov <pp...@gmail.com> wrote:

> search place* = search space
>
> чт, 14 нояб. 2019 г. в 13:10, Vladimir Ozerov <pp...@gmail.com>:
>
> > Hi Haisheng,
> >
> > I double-checked the code. My original version returned false for some
> > cases, but it didn't affect number of rules calls anyway, so I changed it
> > to always return true. Please note that if I change the code as you
> > suggested, the test started failing, because bottom-up propagation of
> rule
> > calls no longer work: when the child is converted to physical form, the
> > parent logical node is not notified. This is the very problem I address
> > with that weird physical-to-logical conversions: they do not make sense,
> > and converter expansion does not produce any new rels, but their
> existence
> > allow for logical rule re-trigger which ultimately allow the plan to
> > compile.
> >
> > Regarding two conventions - I agree that it may look strange, but I do
> not
> > see any problems from the correctness perspective. Separation of logical
> > and physical planning helps me avoid excessive expansion of the search
> > place: I do not want my physical rules to produce new rels from not
> > optimized logical rels. Do you see any problems with that approach?
> >
> > Regards,
> > Vladimir
> >
> > ср, 6 нояб. 2019 г. в 03:52, Haisheng Yuan <h....@alibaba-inc.com>:
> >
> >> Hi Vladimir,
> >>
> >> The code in PHYSICAL convention L44 looks weird, I think it always
> >> returns true.
> >>
> >>
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/HazelcastConventions.java#L44
> >>
> >> Try this:
> >>
> >> fromTraits.containsIfApplicable(Convention.PHYSICAL)
> >>     && toTraits.containsIfApplicable(Convention.PHYSICAL);
> >>
> >>
> >> Adding a AbstractConverter on logical operators is meaningless. Calcite
> >> is mixing the concept of logical and physical together, which is sad.
> >>
> >> BTW, using 2 conventions is not appropriate and wrong.
> >>
> >> - Haisheng
> >>
> >> ------------------------------------------------------------------
> >> 发件人:Vladimir Ozerov<pp...@gmail.com>
> >> 日 期:2019年11月05日 18:02:15
> >> 收件人:Haisheng Yuan<h....@alibaba-inc.com>
> >> 抄 送:dev@calcite.apache.org (dev@calcite.apache.org)<
> >> dev@calcite.apache.org>
> >> 主 题:Re: Re: Problem with converters and possibly rule matching
> >>
> >> Hi Haisheng,
> >>
> >>  think I already tried something very similar to what you explained, but
> >> it gave not an optimal plan. Please let me describe what I did. I would
> >> appreciate your feedback.
> >>
> >> 1) We start with a simple operator tree Root <- Project <- Scan, where
> >> the root is a final aggregator in the distributed query engine:
> >> -> LogicalRoot
> >>  -> LogicalProject
> >>   -> LogicalScan
> >>
> >> 2) First, we convert the Root and enforce SINGLETON distribution on a
> >> child:
> >> *-> PhysicalRoot[SINGLETON]*
> >> * -> Enforcer#1[SINGLETON]*
> >>   -> LogicalProject
> >>    -> LogicalScan
> >>
> >> 3) Then the project's rule is invoked. It doesn't know the distribution
> >> of the input, so it requests ANY distribution. Note that we have to set
> ANY
> >> to the project as well since we do not know the distribution of the
> input:
> >> -> PhysicalRoot[SINGLETON]
> >>  -> Enforcer#1[SINGLETON]
> >> *  -> PhysicalProject[ANY]*
> >> *   -> Enforcer#2[ANY]*
> >>     -> LogicalScan
> >>
> >> 4) Finally, the physical scan is created and its distribution is
> >> resolved. Suppose that it is REPLICATED, i.e. the whole result set is
> >> located on all nodes.
> >> -> PhysicalRoot[SINGLETON]
> >>  -> Enforcer#1[SINGLETON]
> >>   -> PhysicalProject[ANY]
> >>    -> Enforcer#2[ANY]
> >> *    -> PhysicalScan[REPLICATED]*
> >>
> >> 5) Now as all logical nodes are converted, we start resolving enforcers.
> >> The second one is no-op, since REPLICATED satisfies ANY:
> >> -> PhysicalRoot[SINGLETON]
> >>  -> Enforcer#1[SINGLETON]
> >>   -> PhysicalProject[ANY]
> >>    -> PhysicalScan[REPLICATED]
> >>
> >> 6) But the first enforcer now requires an Exchange, since ANY doesn't
> >> satisfy SINGLETON!
> >> -> PhysicalRoot[SINGLETON]
> >> * -> SingletonExchange[SINGLETON]*
> >>   -> PhysicalProject[ANY] // <= unresolved!
> >>    -> PhysicalScan[REPLICATED]
> >>
> >> The resulting plan requires data movement only because we didn't know
> >> precise distribution of the PhysicalProject when it was created. But
> should
> >> I enable Convention.Impl.canConvertConvention, bottom-up propagation
> >> kicks in, and the correct plan is produced because now LogicalProject
> >> has a chance to be converted to PhysicalProject with the concrete
> >> distribution. The optimized plan looks like this (since REPLICATED
> >> satisfies SINGLETON):
> >> -> PhysicalRoot[SINGLETON]
> >>  -> PhysicalProject[REPLICATED]
> >>   -> PhysicalScan[REPLICATED]
> >>
> >> You may see this in action in my reproducer:
> >> 1) Test producing "bad" plan:
> >>
> https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L45
> >> 2) Root enforces SINGLETON on Project:
> >>
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/RootPhysicalRule.java#L45
> >> 3) Project enforces default (ANY) distribution on Scan:
> >>
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L49
> >>
> >> Please let me know if this flow is similar to what you meant.
> >>
> >> Regards,
> >> Vladimir.
> >>
> >> пн, 4 нояб. 2019 г. в 10:33, Haisheng Yuan <h....@alibaba-inc.com>:
> >>
> >>> Hi Vladimir,
> >>>
> >>> This is still can be done through top-down request approach.
> >>>
> >>> PhysicalFilter operator should request ANY distribution from child
> >>> operator, unless there is outer reference in the filter condition, in
> which
> >>> case, PhysicalFilter should request SINGLETON or BROADCAST
> distribution. So
> >>> in your case, PhysicalFilter request ANY, its required distribution
> will be
> >>> enforced on filter's output.
> >>>
> >>> Regarding index usage, you should have a FIlterTableScan2IndexGet
> >>> logical transformation rule, and a IndexGet2IndexScan physical
> >>> implementation rule. Note that IndexGet is a logical operator and
> IndexScan
> >>> is a physical operator, which are also used by SQL Server.
> >>>
> >>> - Haisheng
> >>>
> >>> ------------------------------------------------------------------
> >>> 发件人:Vladimir Ozerov<pp...@gmail.com>
> >>> 日 期:2019年11月01日 17:30:26
> >>> 收件人:<de...@calcite.apache.org>
> >>> 主 题:Re: Problem with converters and possibly rule matching
> >>>
> >>> Hi Stamatis,
> >>>
> >>> Thank you for your reply. I also thought that we may set the
> distribution
> >>> trait during logical planning because it is known in advance. And the
> >>> example I gave will work! :-) But unfortunately, it will work only
> >>> because
> >>> the tree is very simple, and the Project is adjacent to the Scan. This
> is
> >>> how my reproducer will work in that case:
> >>> 1) Root: enforce "SINGLETON" on Project
> >>> 2) Project: check the logical Scan, infer already resolved
> distribution,
> >>> then convert to [PhyiscalProject <- PhysicalScan]
> >>> 3) Resolve Root enforcer, adding and Exchange if needed.
> >>>
> >>> But this stops working as soon as a plan becomes more complex so that
> it
> >>> is
> >>> impossible to infer the distribution from the child immediately. E.g.:
> >>> LogicalRoot [distribution=SINGLETON]
> >>> -> LogicalProject // We are here new and cannot produce the physical
> >>> project
> >>> -> LogicalFilter[distribution=?]
> >>> -> LogicalScan[distribution=REPLICATED]
> >>>
> >>> This is where your suggestion with cascading enforcement may kick in.
> But
> >>> now consider that instead of having REPLICATED distribution, which
> >>> satisfies SINGLETON, we have a PARTITIONED distribution, It doesn't
> >>> satisfy
> >>> SINGLETON, so we have to insert the exchange.
> >>>
> >>> Before:
> >>> PhysicalRoot [SINGLETON]
> >>> -> PhysicalProject [SINGLETON]
> >>> -> PhysicalFilter [SINGLETON]
> >>> -> LogicalScan [PARTITIONED] // SINGLETON is enforced here
> >>>
> >>> After:
> >>> PhysicalRoot [SINGLETON]
> >>> -> PhysicalProject [SINGLETON]
> >>> -> PhysicalFilter [SINGLETON]
> >>> -> SingletonExchange [SINGLETON]
> >>> -> PhysicalScan [PARTITIONED]
> >>>
> >>> But unfortunately, this plan is not optimal. Since we perform Project
> and
> >>> Filter after the exchange, we now have to reshuffle the whole table.
> The
> >>> optimal plan would be to pushdown Project and Filter past exchange:
> >>> PhysicalRoot [SINGLETON]
> >>> -> SingletonExchange [SINGLETON]
> >>> -> PhysicalProject [PARTITIONED]
> >>> -> PhysicalFilter [PARTITIONED]
> >>> -> PhysicalScan [PARTITIONED]
> >>>
> >>> But in order to achieve that, now I need to introduce new rules which
> >>> will
> >>> attempt to transpose dozens of different operators past exchange, so
> most
> >>> likely the planning will never finish :-)
> >>>
> >>> This is why it seems that the bottom-up approach seems to be more
> natural
> >>> for such cases. Another example is index support. A table may have a
> >>> number
> >>> of access methods in addition to plain scan, and every such method may
> >>> produce a physical scan with different collations. It is not practical
> to
> >>> try to enforce something from the top. Instead, we'd better produce
> >>> alternatives from the bottom.
> >>>
> >>> Basically, Drill tries to mix two approaches. First, it tries to derive
> >>> traits from the child. If it fails and no transformations were created,
> >>> it
> >>> just stops trait propagation. As a result, an exchange is created on
> top
> >>> of
> >>> the operator where we stopped, which again leads to not optimal plans.
> >>> You
> >>> may observe this in action if you run my reproducer. "testPass"
> produces
> >>> an
> >>> optimal plan with help of abstract converters. "testDrill" produces not
> >>> optimal plan with the unnecessary exchange.
> >>>
> >>> All in all, I cannot get how we can employ distribution metadata and
> >>> index
> >>> metadata for optimal planning without using a pure bottom-up approach.
> >>>
> >>> Regards,
> >>> Vladimir.
> >>>
> >>> пт, 1 нояб. 2019 г. в 11:42, Stamatis Zampetakis <za...@gmail.com>:
> >>>
> >>> > The discussion is interesting thanks for the nice examples.
> >>> > I am replying in this thread since it has all the examples and the
> >>> history
> >>> > of the conversation.
> >>> >
> >>> > *Part I: Distribution physical or logical property*
> >>> >
> >>> > The Volcano paper writes the following regarding properties:
> >>> >
> >>> > "Logical properties can be derived from the logical algebra
> expression
> >>> and
> >>> > include schema, expected size, etc., while physical properties depend
> >>> on
> >>> > algorithms, e.g., sort order, partitioning, etc."
> >>> >
> >>> > We could say that partitioning is not only a property which depends
> on
> >>> an
> >>> > algorithm but a property which depends on the schema. When you have a
> >>> table
> >>> > you know in advance that the particular table is replicated.
> >>> >
> >>> > Basically, this means that the starting "logical" plan in your case
> >>> could
> >>> > be the following:
> >>> > RootLogicalRel[convention=LOGICAL, distribution=ANY]
> >>> > -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
> >>> > -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
> >>> >
> >>> > When you are about to create your physical projection it is not
> >>> necessary
> >>> > to create three equivalent paths but the most promising by asking the
> >>> input
> >>> > what can it offer as traits. Again this partially overlaps with the
> >>> > discussion about "On demand traitset" [1] where as I wrote there is
> >>> already
> >>> > a mechanism to derive traits from children (collation, distribution,
> >>> etc.).
> >>> >
> >>> > Note that I am not saying that there is nothing more to be done to
> >>> improve
> >>> > the Calcite optimizer; I am just trying to provide a viable
> alternative
> >>> > given the current situation of the project.
> >>> >
> >>> > *Part II: Example with the original Volcano optimizer*
> >>> >
> >>> > Now let me try to show how the original (from the paper) Volcano
> >>> optimizer
> >>> > could handle your query.
> >>> >
> >>> > The optimization starts and we request two properties convention and
> >>> > distribution.
> >>> >
> >>> > RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
> >>> > -> ProjectLogicalRel
> >>> > -> MapScanLogicalRel
> >>> >
> >>> > Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but
> >>> cannot
> >>> > satisfy the SINGLETON property so it propagates the demand
> >>> >
> >>> > RootPhysicalRel
> >>> > -> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
> >>> > -> MapScanLogicalRel
> >>> >
> >>> > Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the algorithm
> >>> but
> >>> > cannot satisfy the SINGLETON property so it propagates the demand
> >>> >
> >>> > RootPhysicalRel
> >>> > -> ProjectPhysicalRel
> >>> > -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
> >>> >
> >>> > Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the algorithm
> >>> and
> >>> > given that the table is replicated it satisfies the distribution
> >>> property.
> >>> >
> >>> > RootPhysicalRel
> >>> > -> ProjectPhysicalRel
> >>> > -> MapScanPhysicalRel
> >>> >
> >>> > So we end up with a complete plan that satisfies all properties and
> >>> does
> >>> > not have redundant exchange operators.
> >>> > Moreover we didn't require all possible distributions when we applied
> >>> Rule
> >>> > 2 since we just want to satisfy the SINGLETON property requested by
> the
> >>> > parent.
> >>> > The example above does not demonstrate the additional optimization
> >>> paths
> >>> > which would be apply an enforcer (an Exchange operator to satisfy the
> >>> > SINGLETON property) at a higher level than the scan.
> >>> >
> >>> > Would this be an acceptable approach? Does it still suffer from a big
> >>> > search space?
> >>> >
> >>> > To make the analog with the actual Volcano optimizer in Calcite the
> >>> thing
> >>> > that may be missing is to be able to know what trait was requested by
> >>> the
> >>> > parent during the application of Rule 2.
> >>> >
> >>> > Best,
> >>> > Stamatis
> >>> >
> >>> > [1]
> >>> >
> >>> >
> >>>
> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
> >>> >
> >>> > On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <pp...@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hi Stamatis,
> >>> > >
> >>> > > The problem that the presented reproducer is a very simplified
> >>> version of
> >>> > > what is actually needed. In reality, there are several distribution
> >>> > types -
> >>> > > PARTITIONED, REPLICATED, SINGLETON. To make things worse,
> >>> PARTITIONED may
> >>> > > or may not have concrete distribution fields. In theory, I can
> >>> create one
> >>> > > transformation per distribution type, but that would increase plan
> >>> space
> >>> > > significantly. In my sample "Root <- Project <- Scan" plan, there
> is
> >>> no
> >>> > > room for optimization at all, we only need to convert the nodes and
> >>> > > propagate traits. But if I follow the proposed approach, the
> planner
> >>> > would
> >>> > > create three equivalent paths. For complex queries, this may
> increase
> >>> > > optimization time significantly.
> >>> > >
> >>> > > What I need instead is to gradually convert and optimize nodes
> >>> > *bottom-up*,
> >>> > > instead of top-bottom. That is, create a physical scan first, then
> >>> > create a
> >>> > > physical project on top of it, etc. But at the same time, some
> rules
> >>> > still
> >>> > > require the top-bottom approach. So essentially I need the
> optimizer
> >>> to
> >>> > do
> >>> > > both. Abstract converters help me establish bottom-up preparation
> >>> but do
> >>> > > this at the cost of considering too many trait pairs, and as a
> >>> result,
> >>> > also
> >>> > > pollute the search space.
> >>> > >
> >>> > > To the contrast, precise command "I transformed the node A to node
> B,
> >>> > > please re-trigger the rules for A's parents" would allow us to
> >>> re-trigger
> >>> > > only required rules, without adding more nodes.
> >>> > >
> >>> > > Does it make sense?
> >>> > >
> >>> > > Regards,
> >>> > > Vladimir.
> >>> > >
> >>> > > ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <
> zabetak@gmail.com
> >>> >:
> >>> > >
> >>> > > > I admit that I didn't thoroughly read the exchanges so far but
> >>> forcing
> >>> > > the
> >>> > > > rules trigger does not seem the right approach.
> >>> > > >
> >>> > > > Other than that I would like to backtrack a bit to point 4.3
> >>> raised by
> >>> > > > Vladimir.
> >>> > > >
> >>> > > > "ProjectPhysicalRule [6] - transforms logical project to physical
> >>> > > > project *ONLY* if there is an underlying physical input with
> >>> REPLICATED
> >>> > > or
> >>> > > > SINGLETON distribution"
> >>> > > >
> >>> > > > The rule could be modified to do the following two
> transformations:
> >>> > > > 1. Create a physical project and require the input to be
> >>> REPLICATED.
> >>> > > > 2. Create a physical project and require
> >>> > > > the input to be SINGLETON.
> >>> > > >
> >>> > > > I would assume that afterwards when your scan rule fires it
> should
> >>> go
> >>> > to
> >>> > > > the appropriate subset and give you back the desired plan. Is
> >>> there a
> >>> > > > problem with this approach?
> >>> > > >
> >>> > > > Best,
> >>> > > > Stamatis
> >>> > > >
> >>> > > > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <
> >>> gvvinblade@gmail.com>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Unfortunately it requires package-private API usage.
> >>> > > > >
> >>> > > > > Best solution there if Calcite provide it for end users.
> >>> > > > >
> >>> > > > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <ppozerov@gmail.com
> >:
> >>> > > > >
> >>> > > > > > “e pension” == “expand conversion” :)
> >>> > > > > >
> >>> > > > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <
> >>> ppozerov@gmail.com>:
> >>> > > > > >
> >>> > > > > > > Yes, that may work. Even if e pension rule is used, for the
> >>> most
> >>> > > > cases
> >>> > > > > it
> >>> > > > > > > will not trigger any real conversions, since we are moving
> >>> from
> >>> > > > > abstract
> >>> > > > > > > convention to physical, and created converters will have
> the
> >>> > > opposite
> >>> > > > > > trait
> >>> > > > > > > direction (from physical to abstract).
> >>> > > > > > >
> >>> > > > > > > But again - ideally we only need to re-trigger the rules
> for
> >>> a
> >>> > > > specific
> >>> > > > > > > node, no more than that. So API support like
> >>> > > > > > > “VolcanoPlanner.forceRules(RelNode)” would be very
> >>> convenient.
> >>> > > > > > >
> >>> > > > > > > What do you think?
> >>> > > > > > >
> >>> > > > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
> >>> > > gvvinblade@gmail.com
> >>> > > > >:
> >>> > > > > > >
> >>> > > > > > >> I considered manual rules calling too, for now we use
> >>> abstract
> >>> > > > > > converters
> >>> > > > > > >> +
> >>> > > > > > >> ExpandConversionRule for exchanges producing.
> >>> > > > > > >>
> >>> > > > > > >> You may create such converters manually (checking
> >>> appropriate
> >>> > > > subset)
> >>> > > > > > this
> >>> > > > > > >> case you may reduce created converters count, also, a
> >>> converter
> >>> > > is a
> >>> > > > > > quite
> >>> > > > > > >> special node, that does almost nothing (without
> >>> corresponding
> >>> > > rule)
> >>> > > > it
> >>> > > > > > may
> >>> > > > > > >> be used just as a rule trigger.
> >>> > > > > > >>
> >>> > > > > > >> Regards,
> >>> > > > > > >> Igor
> >>> > > > > > >>
> >>> > > > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <
> >>> ppozerov@gmail.com
> >>> > >:
> >>> > > > > > >>
> >>> > > > > > >> > One funny hack which helped me is manual registration
> of a
> >>> > fake
> >>> > > > > > RelNode
> >>> > > > > > >> > with desired traits through VolcanoPlanner.register()
> >>> method.
> >>> > > But
> >>> > > > > > again,
> >>> > > > > > >> > this leads to trashing. What could really help is a call
> >>> to
> >>> > > > > > >> > VolcanoPlanner.fireRules() with desired rel. But this
> >>> doesn't
> >>> > > work
> >>> > > > > out
> >>> > > > > > >> of
> >>> > > > > > >> > the box since some internals of the rule queue needs to
> be
> >>> > > > adjusted.
> >>> > > > > > >> >
> >>> > > > > > >> > What does the community think about adding a method
> which
> >>> will
> >>> > > > > re-add
> >>> > > > > > >> rules
> >>> > > > > > >> > applicable to the specific RelNode to the rule queue?
> >>> > > > > > >> >
> >>> > > > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
> >>> > > ppozerov@gmail.com
> >>> > > > >:
> >>> > > > > > >> >
> >>> > > > > > >> > > Hi Igor,
> >>> > > > > > >> > >
> >>> > > > > > >> > > Yes, I came to the same conclusion, thank you. This is
> >>> how
> >>> > it
> >>> > > > > > >> basically
> >>> > > > > > >> > > happens when converters are disabled:
> >>> > > > > > >> > > 1) We start with initial tree: [LogicalProject] <-
> >>> > > [LogicalScan]
> >>> > > > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it
> is
> >>> > added
> >>> > > > to
> >>> > > > > > the
> >>> > > > > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> >>> > > > > > >> > > 3) Finally, when it is time to fire a rule for
> >>> PhysicalScan,
> >>> > > we
> >>> > > > > try
> >>> > > > > > to
> >>> > > > > > >> > get
> >>> > > > > > >> > > parents of that scan set with traits of the
> >>> PhysicalScan.
> >>> > > Since
> >>> > > > > > there
> >>> > > > > > >> are
> >>> > > > > > >> > > no such parents (we skipped it intentionally), the
> rule
> >>> is
> >>> > not
> >>> > > > > > queued.
> >>> > > > > > >> > >
> >>> > > > > > >> > > But when converters are enabled, a converter rel is
> >>> created:
> >>> > > > > > >> > [LogicalProject]
> >>> > > > > > >> > > <- [LogicalScan, PhysicalScan,
> >>> > > ConverterFromPhysicalToLogical].
> >>> > > > No
> >>> > > > > > >> rules
> >>> > > > > > >> > > are fired for PhysicalScan again, but they are fired
> for
> >>> > > > converter
> >>> > > > > > >> since
> >>> > > > > > >> > > it has the necessary LOGICAL trait.
> >>> > > > > > >> > >
> >>> > > > > > >> > > It makes sense, that converters essentially allow
> >>> forcing
> >>> > rule
> >>> > > > > > >> invocation
> >>> > > > > > >> > > on parents, even if the child was created with
> different
> >>> > > traits.
> >>> > > > > But
> >>> > > > > > >> it
> >>> > > > > > >> > > seems that this mechanism may be too heavy for complex
> >>> > queries
> >>> > > > > > >> because it
> >>> > > > > > >> > > literally creates hundreds of new converter rels and
> >>> > triggers
> >>> > > > > rules
> >>> > > > > > >> over
> >>> > > > > > >> > > and over again.
> >>> > > > > > >> > >
> >>> > > > > > >> > > We need some fine-grained alternative. Basically, what
> >>> would
> >>> > > be
> >>> > > > > > enough
> >>> > > > > > >> > for
> >>> > > > > > >> > > me is to let the planner know somehow: "I created that
> >>> rel,
> >>> > > and
> >>> > > > I
> >>> > > > > > want
> >>> > > > > > >> > you
> >>> > > > > > >> > > to execute parent rules not only using its trait but
> >>> also on
> >>> > > > this
> >>> > > > > > and
> >>> > > > > > >> > those
> >>> > > > > > >> > > traits."
> >>> > > > > > >> > > Is there any API in Calcite which allows doing this
> >>> without
> >>> > > > > > creating a
> >>> > > > > > >> > new
> >>> > > > > > >> > > rel node?
> >>> > > > > > >> > >
> >>> > > > > > >> > > Regards,
> >>> > > > > > >> > > Vladimir.
> >>> > > > > > >> > >
> >>> > > > > > >> > >
> >>> > > > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> >>> > > > > gvvinblade@gmail.com
> >>> > > > > > >:
> >>> > > > > > >> > >
> >>> > > > > > >> > >> Vladimir,
> >>> > > > > > >> > >>
> >>> > > > > > >> > >> Probably it'll help you:
> >>> > > > > > >> > >>
> >>> > > > > > >> > >> Seems the cause of issue in RelSubset.getParentRels()
> >>> The
> >>> > > > check
> >>> > > > > > used
> >>> > > > > > >> > when
> >>> > > > > > >> > >> the planner schedules newly matched rules after
> >>> successful
> >>> > > > > > >> > transformation
> >>> > > > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the
> >>> parent
> >>> > > rule
> >>> > > > > be
> >>> > > > > > >> > applied
> >>> > > > > > >> > >> once again (here your logical project with an input
> >>> having
> >>> > > ANY
> >>> > > > > > >> > >> distribution
> >>> > > > > > >> > >> doesn't satisfy a transformed input traits).
> >>> > > > > > >> > >>
> >>> > > > > > >> > >> In our case we use another workaround, so there are
> >>> also
> >>> > much
> >>> > > > > more
> >>> > > > > > >> > >> transformations than we wanted, so the desired rule
> is
> >>> > > > triggered.
> >>> > > > > > >> > >>
> >>> > > > > > >> > >>
> >>> > > > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
> >>> > > ppozerov@gmail.com
> >>> > > > >:
> >>> > > > > > >> > >>
> >>> > > > > > >> > >> > Hi Vladimir,
> >>> > > > > > >> > >> >
> >>> > > > > > >> > >> > I am sorry. Pushed, it works now.
> >>> > > > > > >> > >> >
> >>> > > > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> >>> > > > > > >> > >> > sitnikov.vladimir@gmail.com
> >>> > > > > > >> > >> > >:
> >>> > > > > > >> > >> >
> >>> > > > > > >> > >> > > > mvn clean test
> >>> > > > > > >> > >> > >
> >>> > > > > > >> > >> > > [ERROR] The goal you specified requires a project
> >>> to
> >>> > > > execute
> >>> > > > > > but
> >>> > > > > > >> > >> there is
> >>> > > > > > >> > >> > > no POM in this directory
> >>> > > > > > >> > >> > >
> >>> > > > > > >> > >> > > Vladimir, please push missing files
> >>> > > > > > >> > >> > >
> >>> > > > > > >> > >> > > Vladimir
> >>> > > > > > >> > >> > >
> >>> > > > > > >> > >> >
> >>> > > > > > >> > >>
> >>> > > > > > >> > >
> >>> > > > > > >> >
> >>> > > > > > >>
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>>
> >>
>

Re: Re: Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
search place* = search space

чт, 14 нояб. 2019 г. в 13:10, Vladimir Ozerov <pp...@gmail.com>:

> Hi Haisheng,
>
> I double-checked the code. My original version returned false for some
> cases, but it didn't affect number of rules calls anyway, so I changed it
> to always return true. Please note that if I change the code as you
> suggested, the test started failing, because bottom-up propagation of rule
> calls no longer work: when the child is converted to physical form, the
> parent logical node is not notified. This is the very problem I address
> with that weird physical-to-logical conversions: they do not make sense,
> and converter expansion does not produce any new rels, but their existence
> allow for logical rule re-trigger which ultimately allow the plan to
> compile.
>
> Regarding two conventions - I agree that it may look strange, but I do not
> see any problems from the correctness perspective. Separation of logical
> and physical planning helps me avoid excessive expansion of the search
> place: I do not want my physical rules to produce new rels from not
> optimized logical rels. Do you see any problems with that approach?
>
> Regards,
> Vladimir
>
> ср, 6 нояб. 2019 г. в 03:52, Haisheng Yuan <h....@alibaba-inc.com>:
>
>> Hi Vladimir,
>>
>> The code in PHYSICAL convention L44 looks weird, I think it always
>> returns true.
>>
>> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/HazelcastConventions.java#L44
>>
>> Try this:
>>
>> fromTraits.containsIfApplicable(Convention.PHYSICAL)
>>     && toTraits.containsIfApplicable(Convention.PHYSICAL);
>>
>>
>> Adding a AbstractConverter on logical operators is meaningless. Calcite
>> is mixing the concept of logical and physical together, which is sad.
>>
>> BTW, using 2 conventions is not appropriate and wrong.
>>
>> - Haisheng
>>
>> ------------------------------------------------------------------
>> 发件人:Vladimir Ozerov<pp...@gmail.com>
>> 日 期:2019年11月05日 18:02:15
>> 收件人:Haisheng Yuan<h....@alibaba-inc.com>
>> 抄 送:dev@calcite.apache.org (dev@calcite.apache.org)<
>> dev@calcite.apache.org>
>> 主 题:Re: Re: Problem with converters and possibly rule matching
>>
>> Hi Haisheng,
>>
>>  think I already tried something very similar to what you explained, but
>> it gave not an optimal plan. Please let me describe what I did. I would
>> appreciate your feedback.
>>
>> 1) We start with a simple operator tree Root <- Project <- Scan, where
>> the root is a final aggregator in the distributed query engine:
>> -> LogicalRoot
>>  -> LogicalProject
>>   -> LogicalScan
>>
>> 2) First, we convert the Root and enforce SINGLETON distribution on a
>> child:
>> *-> PhysicalRoot[SINGLETON]*
>> * -> Enforcer#1[SINGLETON]*
>>   -> LogicalProject
>>    -> LogicalScan
>>
>> 3) Then the project's rule is invoked. It doesn't know the distribution
>> of the input, so it requests ANY distribution. Note that we have to set ANY
>> to the project as well since we do not know the distribution of the input:
>> -> PhysicalRoot[SINGLETON]
>>  -> Enforcer#1[SINGLETON]
>> *  -> PhysicalProject[ANY]*
>> *   -> Enforcer#2[ANY]*
>>     -> LogicalScan
>>
>> 4) Finally, the physical scan is created and its distribution is
>> resolved. Suppose that it is REPLICATED, i.e. the whole result set is
>> located on all nodes.
>> -> PhysicalRoot[SINGLETON]
>>  -> Enforcer#1[SINGLETON]
>>   -> PhysicalProject[ANY]
>>    -> Enforcer#2[ANY]
>> *    -> PhysicalScan[REPLICATED]*
>>
>> 5) Now as all logical nodes are converted, we start resolving enforcers.
>> The second one is no-op, since REPLICATED satisfies ANY:
>> -> PhysicalRoot[SINGLETON]
>>  -> Enforcer#1[SINGLETON]
>>   -> PhysicalProject[ANY]
>>    -> PhysicalScan[REPLICATED]
>>
>> 6) But the first enforcer now requires an Exchange, since ANY doesn't
>> satisfy SINGLETON!
>> -> PhysicalRoot[SINGLETON]
>> * -> SingletonExchange[SINGLETON]*
>>   -> PhysicalProject[ANY] // <= unresolved!
>>    -> PhysicalScan[REPLICATED]
>>
>> The resulting plan requires data movement only because we didn't know
>> precise distribution of the PhysicalProject when it was created. But should
>> I enable Convention.Impl.canConvertConvention, bottom-up propagation
>> kicks in, and the correct plan is produced because now LogicalProject
>> has a chance to be converted to PhysicalProject with the concrete
>> distribution. The optimized plan looks like this (since REPLICATED
>> satisfies SINGLETON):
>> -> PhysicalRoot[SINGLETON]
>>  -> PhysicalProject[REPLICATED]
>>   -> PhysicalScan[REPLICATED]
>>
>> You may see this in action in my reproducer:
>> 1) Test producing "bad" plan:
>> https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L45
>> 2) Root enforces SINGLETON on Project:
>> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/RootPhysicalRule.java#L45
>> 3) Project enforces default (ANY) distribution on Scan:
>> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L49
>>
>> Please let me know if this flow is similar to what you meant.
>>
>> Regards,
>> Vladimir.
>>
>> пн, 4 нояб. 2019 г. в 10:33, Haisheng Yuan <h....@alibaba-inc.com>:
>>
>>> Hi Vladimir,
>>>
>>> This is still can be done through top-down request approach.
>>>
>>> PhysicalFilter operator should request ANY distribution from child
>>> operator, unless there is outer reference in the filter condition, in which
>>> case, PhysicalFilter should request SINGLETON or BROADCAST distribution. So
>>> in your case, PhysicalFilter request ANY, its required distribution will be
>>> enforced on filter's output.
>>>
>>> Regarding index usage, you should have a FIlterTableScan2IndexGet
>>> logical transformation rule, and a IndexGet2IndexScan physical
>>> implementation rule. Note that IndexGet is a logical operator and IndexScan
>>> is a physical operator, which are also used by SQL Server.
>>>
>>> - Haisheng
>>>
>>> ------------------------------------------------------------------
>>> 发件人:Vladimir Ozerov<pp...@gmail.com>
>>> 日 期:2019年11月01日 17:30:26
>>> 收件人:<de...@calcite.apache.org>
>>> 主 题:Re: Problem with converters and possibly rule matching
>>>
>>> Hi Stamatis,
>>>
>>> Thank you for your reply. I also thought that we may set the distribution
>>> trait during logical planning because it is known in advance. And the
>>> example I gave will work! :-) But unfortunately, it will work only
>>> because
>>> the tree is very simple, and the Project is adjacent to the Scan. This is
>>> how my reproducer will work in that case:
>>> 1) Root: enforce "SINGLETON" on Project
>>> 2) Project: check the logical Scan, infer already resolved distribution,
>>> then convert to [PhyiscalProject <- PhysicalScan]
>>> 3) Resolve Root enforcer, adding and Exchange if needed.
>>>
>>> But this stops working as soon as a plan becomes more complex so that it
>>> is
>>> impossible to infer the distribution from the child immediately. E.g.:
>>> LogicalRoot [distribution=SINGLETON]
>>> -> LogicalProject // We are here new and cannot produce the physical
>>> project
>>> -> LogicalFilter[distribution=?]
>>> -> LogicalScan[distribution=REPLICATED]
>>>
>>> This is where your suggestion with cascading enforcement may kick in. But
>>> now consider that instead of having REPLICATED distribution, which
>>> satisfies SINGLETON, we have a PARTITIONED distribution, It doesn't
>>> satisfy
>>> SINGLETON, so we have to insert the exchange.
>>>
>>> Before:
>>> PhysicalRoot [SINGLETON]
>>> -> PhysicalProject [SINGLETON]
>>> -> PhysicalFilter [SINGLETON]
>>> -> LogicalScan [PARTITIONED] // SINGLETON is enforced here
>>>
>>> After:
>>> PhysicalRoot [SINGLETON]
>>> -> PhysicalProject [SINGLETON]
>>> -> PhysicalFilter [SINGLETON]
>>> -> SingletonExchange [SINGLETON]
>>> -> PhysicalScan [PARTITIONED]
>>>
>>> But unfortunately, this plan is not optimal. Since we perform Project and
>>> Filter after the exchange, we now have to reshuffle the whole table. The
>>> optimal plan would be to pushdown Project and Filter past exchange:
>>> PhysicalRoot [SINGLETON]
>>> -> SingletonExchange [SINGLETON]
>>> -> PhysicalProject [PARTITIONED]
>>> -> PhysicalFilter [PARTITIONED]
>>> -> PhysicalScan [PARTITIONED]
>>>
>>> But in order to achieve that, now I need to introduce new rules which
>>> will
>>> attempt to transpose dozens of different operators past exchange, so most
>>> likely the planning will never finish :-)
>>>
>>> This is why it seems that the bottom-up approach seems to be more natural
>>> for such cases. Another example is index support. A table may have a
>>> number
>>> of access methods in addition to plain scan, and every such method may
>>> produce a physical scan with different collations. It is not practical to
>>> try to enforce something from the top. Instead, we'd better produce
>>> alternatives from the bottom.
>>>
>>> Basically, Drill tries to mix two approaches. First, it tries to derive
>>> traits from the child. If it fails and no transformations were created,
>>> it
>>> just stops trait propagation. As a result, an exchange is created on top
>>> of
>>> the operator where we stopped, which again leads to not optimal plans.
>>> You
>>> may observe this in action if you run my reproducer. "testPass" produces
>>> an
>>> optimal plan with help of abstract converters. "testDrill" produces not
>>> optimal plan with the unnecessary exchange.
>>>
>>> All in all, I cannot get how we can employ distribution metadata and
>>> index
>>> metadata for optimal planning without using a pure bottom-up approach.
>>>
>>> Regards,
>>> Vladimir.
>>>
>>> пт, 1 нояб. 2019 г. в 11:42, Stamatis Zampetakis <za...@gmail.com>:
>>>
>>> > The discussion is interesting thanks for the nice examples.
>>> > I am replying in this thread since it has all the examples and the
>>> history
>>> > of the conversation.
>>> >
>>> > *Part I: Distribution physical or logical property*
>>> >
>>> > The Volcano paper writes the following regarding properties:
>>> >
>>> > "Logical properties can be derived from the logical algebra expression
>>> and
>>> > include schema, expected size, etc., while physical properties depend
>>> on
>>> > algorithms, e.g., sort order, partitioning, etc."
>>> >
>>> > We could say that partitioning is not only a property which depends on
>>> an
>>> > algorithm but a property which depends on the schema. When you have a
>>> table
>>> > you know in advance that the particular table is replicated.
>>> >
>>> > Basically, this means that the starting "logical" plan in your case
>>> could
>>> > be the following:
>>> > RootLogicalRel[convention=LOGICAL, distribution=ANY]
>>> > -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
>>> > -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
>>> >
>>> > When you are about to create your physical projection it is not
>>> necessary
>>> > to create three equivalent paths but the most promising by asking the
>>> input
>>> > what can it offer as traits. Again this partially overlaps with the
>>> > discussion about "On demand traitset" [1] where as I wrote there is
>>> already
>>> > a mechanism to derive traits from children (collation, distribution,
>>> etc.).
>>> >
>>> > Note that I am not saying that there is nothing more to be done to
>>> improve
>>> > the Calcite optimizer; I am just trying to provide a viable alternative
>>> > given the current situation of the project.
>>> >
>>> > *Part II: Example with the original Volcano optimizer*
>>> >
>>> > Now let me try to show how the original (from the paper) Volcano
>>> optimizer
>>> > could handle your query.
>>> >
>>> > The optimization starts and we request two properties convention and
>>> > distribution.
>>> >
>>> > RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
>>> > -> ProjectLogicalRel
>>> > -> MapScanLogicalRel
>>> >
>>> > Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but
>>> cannot
>>> > satisfy the SINGLETON property so it propagates the demand
>>> >
>>> > RootPhysicalRel
>>> > -> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>>> > -> MapScanLogicalRel
>>> >
>>> > Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the algorithm
>>> but
>>> > cannot satisfy the SINGLETON property so it propagates the demand
>>> >
>>> > RootPhysicalRel
>>> > -> ProjectPhysicalRel
>>> > -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>>> >
>>> > Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the algorithm
>>> and
>>> > given that the table is replicated it satisfies the distribution
>>> property.
>>> >
>>> > RootPhysicalRel
>>> > -> ProjectPhysicalRel
>>> > -> MapScanPhysicalRel
>>> >
>>> > So we end up with a complete plan that satisfies all properties and
>>> does
>>> > not have redundant exchange operators.
>>> > Moreover we didn't require all possible distributions when we applied
>>> Rule
>>> > 2 since we just want to satisfy the SINGLETON property requested by the
>>> > parent.
>>> > The example above does not demonstrate the additional optimization
>>> paths
>>> > which would be apply an enforcer (an Exchange operator to satisfy the
>>> > SINGLETON property) at a higher level than the scan.
>>> >
>>> > Would this be an acceptable approach? Does it still suffer from a big
>>> > search space?
>>> >
>>> > To make the analog with the actual Volcano optimizer in Calcite the
>>> thing
>>> > that may be missing is to be able to know what trait was requested by
>>> the
>>> > parent during the application of Rule 2.
>>> >
>>> > Best,
>>> > Stamatis
>>> >
>>> > [1]
>>> >
>>> >
>>> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
>>> >
>>> > On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <pp...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Stamatis,
>>> > >
>>> > > The problem that the presented reproducer is a very simplified
>>> version of
>>> > > what is actually needed. In reality, there are several distribution
>>> > types -
>>> > > PARTITIONED, REPLICATED, SINGLETON. To make things worse,
>>> PARTITIONED may
>>> > > or may not have concrete distribution fields. In theory, I can
>>> create one
>>> > > transformation per distribution type, but that would increase plan
>>> space
>>> > > significantly. In my sample "Root <- Project <- Scan" plan, there is
>>> no
>>> > > room for optimization at all, we only need to convert the nodes and
>>> > > propagate traits. But if I follow the proposed approach, the planner
>>> > would
>>> > > create three equivalent paths. For complex queries, this may increase
>>> > > optimization time significantly.
>>> > >
>>> > > What I need instead is to gradually convert and optimize nodes
>>> > *bottom-up*,
>>> > > instead of top-bottom. That is, create a physical scan first, then
>>> > create a
>>> > > physical project on top of it, etc. But at the same time, some rules
>>> > still
>>> > > require the top-bottom approach. So essentially I need the optimizer
>>> to
>>> > do
>>> > > both. Abstract converters help me establish bottom-up preparation
>>> but do
>>> > > this at the cost of considering too many trait pairs, and as a
>>> result,
>>> > also
>>> > > pollute the search space.
>>> > >
>>> > > To the contrast, precise command "I transformed the node A to node B,
>>> > > please re-trigger the rules for A's parents" would allow us to
>>> re-trigger
>>> > > only required rules, without adding more nodes.
>>> > >
>>> > > Does it make sense?
>>> > >
>>> > > Regards,
>>> > > Vladimir.
>>> > >
>>> > > ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <zabetak@gmail.com
>>> >:
>>> > >
>>> > > > I admit that I didn't thoroughly read the exchanges so far but
>>> forcing
>>> > > the
>>> > > > rules trigger does not seem the right approach.
>>> > > >
>>> > > > Other than that I would like to backtrack a bit to point 4.3
>>> raised by
>>> > > > Vladimir.
>>> > > >
>>> > > > "ProjectPhysicalRule [6] - transforms logical project to physical
>>> > > > project *ONLY* if there is an underlying physical input with
>>> REPLICATED
>>> > > or
>>> > > > SINGLETON distribution"
>>> > > >
>>> > > > The rule could be modified to do the following two transformations:
>>> > > > 1. Create a physical project and require the input to be
>>> REPLICATED.
>>> > > > 2. Create a physical project and require
>>> > > > the input to be SINGLETON.
>>> > > >
>>> > > > I would assume that afterwards when your scan rule fires it should
>>> go
>>> > to
>>> > > > the appropriate subset and give you back the desired plan. Is
>>> there a
>>> > > > problem with this approach?
>>> > > >
>>> > > > Best,
>>> > > > Stamatis
>>> > > >
>>> > > > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <
>>> gvvinblade@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > Unfortunately it requires package-private API usage.
>>> > > > >
>>> > > > > Best solution there if Calcite provide it for end users.
>>> > > > >
>>> > > > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
>>> > > > >
>>> > > > > > “e pension” == “expand conversion” :)
>>> > > > > >
>>> > > > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <
>>> ppozerov@gmail.com>:
>>> > > > > >
>>> > > > > > > Yes, that may work. Even if e pension rule is used, for the
>>> most
>>> > > > cases
>>> > > > > it
>>> > > > > > > will not trigger any real conversions, since we are moving
>>> from
>>> > > > > abstract
>>> > > > > > > convention to physical, and created converters will have the
>>> > > opposite
>>> > > > > > trait
>>> > > > > > > direction (from physical to abstract).
>>> > > > > > >
>>> > > > > > > But again - ideally we only need to re-trigger the rules for
>>> a
>>> > > > specific
>>> > > > > > > node, no more than that. So API support like
>>> > > > > > > “VolcanoPlanner.forceRules(RelNode)” would be very
>>> convenient.
>>> > > > > > >
>>> > > > > > > What do you think?
>>> > > > > > >
>>> > > > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
>>> > > gvvinblade@gmail.com
>>> > > > >:
>>> > > > > > >
>>> > > > > > >> I considered manual rules calling too, for now we use
>>> abstract
>>> > > > > > converters
>>> > > > > > >> +
>>> > > > > > >> ExpandConversionRule for exchanges producing.
>>> > > > > > >>
>>> > > > > > >> You may create such converters manually (checking
>>> appropriate
>>> > > > subset)
>>> > > > > > this
>>> > > > > > >> case you may reduce created converters count, also, a
>>> converter
>>> > > is a
>>> > > > > > quite
>>> > > > > > >> special node, that does almost nothing (without
>>> corresponding
>>> > > rule)
>>> > > > it
>>> > > > > > may
>>> > > > > > >> be used just as a rule trigger.
>>> > > > > > >>
>>> > > > > > >> Regards,
>>> > > > > > >> Igor
>>> > > > > > >>
>>> > > > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <
>>> ppozerov@gmail.com
>>> > >:
>>> > > > > > >>
>>> > > > > > >> > One funny hack which helped me is manual registration of a
>>> > fake
>>> > > > > > RelNode
>>> > > > > > >> > with desired traits through VolcanoPlanner.register()
>>> method.
>>> > > But
>>> > > > > > again,
>>> > > > > > >> > this leads to trashing. What could really help is a call
>>> to
>>> > > > > > >> > VolcanoPlanner.fireRules() with desired rel. But this
>>> doesn't
>>> > > work
>>> > > > > out
>>> > > > > > >> of
>>> > > > > > >> > the box since some internals of the rule queue needs to be
>>> > > > adjusted.
>>> > > > > > >> >
>>> > > > > > >> > What does the community think about adding a method which
>>> will
>>> > > > > re-add
>>> > > > > > >> rules
>>> > > > > > >> > applicable to the specific RelNode to the rule queue?
>>> > > > > > >> >
>>> > > > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
>>> > > ppozerov@gmail.com
>>> > > > >:
>>> > > > > > >> >
>>> > > > > > >> > > Hi Igor,
>>> > > > > > >> > >
>>> > > > > > >> > > Yes, I came to the same conclusion, thank you. This is
>>> how
>>> > it
>>> > > > > > >> basically
>>> > > > > > >> > > happens when converters are disabled:
>>> > > > > > >> > > 1) We start with initial tree: [LogicalProject] <-
>>> > > [LogicalScan]
>>> > > > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is
>>> > added
>>> > > > to
>>> > > > > > the
>>> > > > > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
>>> > > > > > >> > > 3) Finally, when it is time to fire a rule for
>>> PhysicalScan,
>>> > > we
>>> > > > > try
>>> > > > > > to
>>> > > > > > >> > get
>>> > > > > > >> > > parents of that scan set with traits of the
>>> PhysicalScan.
>>> > > Since
>>> > > > > > there
>>> > > > > > >> are
>>> > > > > > >> > > no such parents (we skipped it intentionally), the rule
>>> is
>>> > not
>>> > > > > > queued.
>>> > > > > > >> > >
>>> > > > > > >> > > But when converters are enabled, a converter rel is
>>> created:
>>> > > > > > >> > [LogicalProject]
>>> > > > > > >> > > <- [LogicalScan, PhysicalScan,
>>> > > ConverterFromPhysicalToLogical].
>>> > > > No
>>> > > > > > >> rules
>>> > > > > > >> > > are fired for PhysicalScan again, but they are fired for
>>> > > > converter
>>> > > > > > >> since
>>> > > > > > >> > > it has the necessary LOGICAL trait.
>>> > > > > > >> > >
>>> > > > > > >> > > It makes sense, that converters essentially allow
>>> forcing
>>> > rule
>>> > > > > > >> invocation
>>> > > > > > >> > > on parents, even if the child was created with different
>>> > > traits.
>>> > > > > But
>>> > > > > > >> it
>>> > > > > > >> > > seems that this mechanism may be too heavy for complex
>>> > queries
>>> > > > > > >> because it
>>> > > > > > >> > > literally creates hundreds of new converter rels and
>>> > triggers
>>> > > > > rules
>>> > > > > > >> over
>>> > > > > > >> > > and over again.
>>> > > > > > >> > >
>>> > > > > > >> > > We need some fine-grained alternative. Basically, what
>>> would
>>> > > be
>>> > > > > > enough
>>> > > > > > >> > for
>>> > > > > > >> > > me is to let the planner know somehow: "I created that
>>> rel,
>>> > > and
>>> > > > I
>>> > > > > > want
>>> > > > > > >> > you
>>> > > > > > >> > > to execute parent rules not only using its trait but
>>> also on
>>> > > > this
>>> > > > > > and
>>> > > > > > >> > those
>>> > > > > > >> > > traits."
>>> > > > > > >> > > Is there any API in Calcite which allows doing this
>>> without
>>> > > > > > creating a
>>> > > > > > >> > new
>>> > > > > > >> > > rel node?
>>> > > > > > >> > >
>>> > > > > > >> > > Regards,
>>> > > > > > >> > > Vladimir.
>>> > > > > > >> > >
>>> > > > > > >> > >
>>> > > > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
>>> > > > > gvvinblade@gmail.com
>>> > > > > > >:
>>> > > > > > >> > >
>>> > > > > > >> > >> Vladimir,
>>> > > > > > >> > >>
>>> > > > > > >> > >> Probably it'll help you:
>>> > > > > > >> > >>
>>> > > > > > >> > >> Seems the cause of issue in RelSubset.getParentRels()
>>> The
>>> > > > check
>>> > > > > > used
>>> > > > > > >> > when
>>> > > > > > >> > >> the planner schedules newly matched rules after
>>> successful
>>> > > > > > >> > transformation
>>> > > > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the
>>> parent
>>> > > rule
>>> > > > > be
>>> > > > > > >> > applied
>>> > > > > > >> > >> once again (here your logical project with an input
>>> having
>>> > > ANY
>>> > > > > > >> > >> distribution
>>> > > > > > >> > >> doesn't satisfy a transformed input traits).
>>> > > > > > >> > >>
>>> > > > > > >> > >> In our case we use another workaround, so there are
>>> also
>>> > much
>>> > > > > more
>>> > > > > > >> > >> transformations than we wanted, so the desired rule is
>>> > > > triggered.
>>> > > > > > >> > >>
>>> > > > > > >> > >>
>>> > > > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
>>> > > ppozerov@gmail.com
>>> > > > >:
>>> > > > > > >> > >>
>>> > > > > > >> > >> > Hi Vladimir,
>>> > > > > > >> > >> >
>>> > > > > > >> > >> > I am sorry. Pushed, it works now.
>>> > > > > > >> > >> >
>>> > > > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
>>> > > > > > >> > >> > sitnikov.vladimir@gmail.com
>>> > > > > > >> > >> > >:
>>> > > > > > >> > >> >
>>> > > > > > >> > >> > > > mvn clean test
>>> > > > > > >> > >> > >
>>> > > > > > >> > >> > > [ERROR] The goal you specified requires a project
>>> to
>>> > > > execute
>>> > > > > > but
>>> > > > > > >> > >> there is
>>> > > > > > >> > >> > > no POM in this directory
>>> > > > > > >> > >> > >
>>> > > > > > >> > >> > > Vladimir, please push missing files
>>> > > > > > >> > >> > >
>>> > > > > > >> > >> > > Vladimir
>>> > > > > > >> > >> > >
>>> > > > > > >> > >> >
>>> > > > > > >> > >>
>>> > > > > > >> > >
>>> > > > > > >> >
>>> > > > > > >>
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>>
>>

Re: Re: Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
Hi Haisheng,

I double-checked the code. My original version returned false for some
cases, but it didn't affect number of rules calls anyway, so I changed it
to always return true. Please note that if I change the code as you
suggested, the test started failing, because bottom-up propagation of rule
calls no longer work: when the child is converted to physical form, the
parent logical node is not notified. This is the very problem I address
with that weird physical-to-logical conversions: they do not make sense,
and converter expansion does not produce any new rels, but their existence
allow for logical rule re-trigger which ultimately allow the plan to
compile.

Regarding two conventions - I agree that it may look strange, but I do not
see any problems from the correctness perspective. Separation of logical
and physical planning helps me avoid excessive expansion of the search
place: I do not want my physical rules to produce new rels from not
optimized logical rels. Do you see any problems with that approach?

Regards,
Vladimir

ср, 6 нояб. 2019 г. в 03:52, Haisheng Yuan <h....@alibaba-inc.com>:

> Hi Vladimir,
>
> The code in PHYSICAL convention L44 looks weird, I think it always returns
> true.
>
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/HazelcastConventions.java#L44
>
> Try this:
>
> fromTraits.containsIfApplicable(Convention.PHYSICAL)
>     && toTraits.containsIfApplicable(Convention.PHYSICAL);
>
>
> Adding a AbstractConverter on logical operators is meaningless. Calcite is
> mixing the concept of logical and physical together, which is sad.
>
> BTW, using 2 conventions is not appropriate and wrong.
>
> - Haisheng
>
> ------------------------------------------------------------------
> 发件人:Vladimir Ozerov<pp...@gmail.com>
> 日 期:2019年11月05日 18:02:15
> 收件人:Haisheng Yuan<h....@alibaba-inc.com>
> 抄 送:dev@calcite.apache.org (dev@calcite.apache.org)<dev@calcite.apache.org
> >
> 主 题:Re: Re: Problem with converters and possibly rule matching
>
> Hi Haisheng,
>
>  think I already tried something very similar to what you explained, but
> it gave not an optimal plan. Please let me describe what I did. I would
> appreciate your feedback.
>
> 1) We start with a simple operator tree Root <- Project <- Scan, where
> the root is a final aggregator in the distributed query engine:
> -> LogicalRoot
>  -> LogicalProject
>   -> LogicalScan
>
> 2) First, we convert the Root and enforce SINGLETON distribution on a
> child:
> *-> PhysicalRoot[SINGLETON]*
> * -> Enforcer#1[SINGLETON]*
>   -> LogicalProject
>    -> LogicalScan
>
> 3) Then the project's rule is invoked. It doesn't know the distribution of
> the input, so it requests ANY distribution. Note that we have to set ANY to
> the project as well since we do not know the distribution of the input:
> -> PhysicalRoot[SINGLETON]
>  -> Enforcer#1[SINGLETON]
> *  -> PhysicalProject[ANY]*
> *   -> Enforcer#2[ANY]*
>     -> LogicalScan
>
> 4) Finally, the physical scan is created and its distribution is resolved.
> Suppose that it is REPLICATED, i.e. the whole result set is located on all
> nodes.
> -> PhysicalRoot[SINGLETON]
>  -> Enforcer#1[SINGLETON]
>   -> PhysicalProject[ANY]
>    -> Enforcer#2[ANY]
> *    -> PhysicalScan[REPLICATED]*
>
> 5) Now as all logical nodes are converted, we start resolving enforcers.
> The second one is no-op, since REPLICATED satisfies ANY:
> -> PhysicalRoot[SINGLETON]
>  -> Enforcer#1[SINGLETON]
>   -> PhysicalProject[ANY]
>    -> PhysicalScan[REPLICATED]
>
> 6) But the first enforcer now requires an Exchange, since ANY doesn't
> satisfy SINGLETON!
> -> PhysicalRoot[SINGLETON]
> * -> SingletonExchange[SINGLETON]*
>   -> PhysicalProject[ANY] // <= unresolved!
>    -> PhysicalScan[REPLICATED]
>
> The resulting plan requires data movement only because we didn't know
> precise distribution of the PhysicalProject when it was created. But should
> I enable Convention.Impl.canConvertConvention, bottom-up propagation
> kicks in, and the correct plan is produced because now LogicalProject has
> a chance to be converted to PhysicalProject with the concrete
> distribution. The optimized plan looks like this (since REPLICATED
> satisfies SINGLETON):
> -> PhysicalRoot[SINGLETON]
>  -> PhysicalProject[REPLICATED]
>   -> PhysicalScan[REPLICATED]
>
> You may see this in action in my reproducer:
> 1) Test producing "bad" plan:
> https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L45
> 2) Root enforces SINGLETON on Project:
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/RootPhysicalRule.java#L45
> 3) Project enforces default (ANY) distribution on Scan:
> https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L49
>
> Please let me know if this flow is similar to what you meant.
>
> Regards,
> Vladimir.
>
> пн, 4 нояб. 2019 г. в 10:33, Haisheng Yuan <h....@alibaba-inc.com>:
>
>> Hi Vladimir,
>>
>> This is still can be done through top-down request approach.
>>
>> PhysicalFilter operator should request ANY distribution from child
>> operator, unless there is outer reference in the filter condition, in which
>> case, PhysicalFilter should request SINGLETON or BROADCAST distribution. So
>> in your case, PhysicalFilter request ANY, its required distribution will be
>> enforced on filter's output.
>>
>> Regarding index usage, you should have a FIlterTableScan2IndexGet logical
>> transformation rule, and a IndexGet2IndexScan physical implementation rule.
>> Note that IndexGet is a logical operator and IndexScan is a physical
>> operator, which are also used by SQL Server.
>>
>> - Haisheng
>>
>> ------------------------------------------------------------------
>> 发件人:Vladimir Ozerov<pp...@gmail.com>
>> 日 期:2019年11月01日 17:30:26
>> 收件人:<de...@calcite.apache.org>
>> 主 题:Re: Problem with converters and possibly rule matching
>>
>> Hi Stamatis,
>>
>> Thank you for your reply. I also thought that we may set the distribution
>> trait during logical planning because it is known in advance. And the
>> example I gave will work! :-) But unfortunately, it will work only because
>> the tree is very simple, and the Project is adjacent to the Scan. This is
>> how my reproducer will work in that case:
>> 1) Root: enforce "SINGLETON" on Project
>> 2) Project: check the logical Scan, infer already resolved distribution,
>> then convert to [PhyiscalProject <- PhysicalScan]
>> 3) Resolve Root enforcer, adding and Exchange if needed.
>>
>> But this stops working as soon as a plan becomes more complex so that it
>> is
>> impossible to infer the distribution from the child immediately. E.g.:
>> LogicalRoot [distribution=SINGLETON]
>> -> LogicalProject // We are here new and cannot produce the physical
>> project
>> -> LogicalFilter[distribution=?]
>> -> LogicalScan[distribution=REPLICATED]
>>
>> This is where your suggestion with cascading enforcement may kick in. But
>> now consider that instead of having REPLICATED distribution, which
>> satisfies SINGLETON, we have a PARTITIONED distribution, It doesn't
>> satisfy
>> SINGLETON, so we have to insert the exchange.
>>
>> Before:
>> PhysicalRoot [SINGLETON]
>> -> PhysicalProject [SINGLETON]
>> -> PhysicalFilter [SINGLETON]
>> -> LogicalScan [PARTITIONED] // SINGLETON is enforced here
>>
>> After:
>> PhysicalRoot [SINGLETON]
>> -> PhysicalProject [SINGLETON]
>> -> PhysicalFilter [SINGLETON]
>> -> SingletonExchange [SINGLETON]
>> -> PhysicalScan [PARTITIONED]
>>
>> But unfortunately, this plan is not optimal. Since we perform Project and
>> Filter after the exchange, we now have to reshuffle the whole table. The
>> optimal plan would be to pushdown Project and Filter past exchange:
>> PhysicalRoot [SINGLETON]
>> -> SingletonExchange [SINGLETON]
>> -> PhysicalProject [PARTITIONED]
>> -> PhysicalFilter [PARTITIONED]
>> -> PhysicalScan [PARTITIONED]
>>
>> But in order to achieve that, now I need to introduce new rules which will
>> attempt to transpose dozens of different operators past exchange, so most
>> likely the planning will never finish :-)
>>
>> This is why it seems that the bottom-up approach seems to be more natural
>> for such cases. Another example is index support. A table may have a
>> number
>> of access methods in addition to plain scan, and every such method may
>> produce a physical scan with different collations. It is not practical to
>> try to enforce something from the top. Instead, we'd better produce
>> alternatives from the bottom.
>>
>> Basically, Drill tries to mix two approaches. First, it tries to derive
>> traits from the child. If it fails and no transformations were created, it
>> just stops trait propagation. As a result, an exchange is created on top
>> of
>> the operator where we stopped, which again leads to not optimal plans. You
>> may observe this in action if you run my reproducer. "testPass" produces
>> an
>> optimal plan with help of abstract converters. "testDrill" produces not
>> optimal plan with the unnecessary exchange.
>>
>> All in all, I cannot get how we can employ distribution metadata and index
>> metadata for optimal planning without using a pure bottom-up approach.
>>
>> Regards,
>> Vladimir.
>>
>> пт, 1 нояб. 2019 г. в 11:42, Stamatis Zampetakis <za...@gmail.com>:
>>
>> > The discussion is interesting thanks for the nice examples.
>> > I am replying in this thread since it has all the examples and the
>> history
>> > of the conversation.
>> >
>> > *Part I: Distribution physical or logical property*
>> >
>> > The Volcano paper writes the following regarding properties:
>> >
>> > "Logical properties can be derived from the logical algebra expression
>> and
>> > include schema, expected size, etc., while physical properties depend on
>> > algorithms, e.g., sort order, partitioning, etc."
>> >
>> > We could say that partitioning is not only a property which depends on
>> an
>> > algorithm but a property which depends on the schema. When you have a
>> table
>> > you know in advance that the particular table is replicated.
>> >
>> > Basically, this means that the starting "logical" plan in your case
>> could
>> > be the following:
>> > RootLogicalRel[convention=LOGICAL, distribution=ANY]
>> > -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
>> > -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
>> >
>> > When you are about to create your physical projection it is not
>> necessary
>> > to create three equivalent paths but the most promising by asking the
>> input
>> > what can it offer as traits. Again this partially overlaps with the
>> > discussion about "On demand traitset" [1] where as I wrote there is
>> already
>> > a mechanism to derive traits from children (collation, distribution,
>> etc.).
>> >
>> > Note that I am not saying that there is nothing more to be done to
>> improve
>> > the Calcite optimizer; I am just trying to provide a viable alternative
>> > given the current situation of the project.
>> >
>> > *Part II: Example with the original Volcano optimizer*
>> >
>> > Now let me try to show how the original (from the paper) Volcano
>> optimizer
>> > could handle your query.
>> >
>> > The optimization starts and we request two properties convention and
>> > distribution.
>> >
>> > RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
>> > -> ProjectLogicalRel
>> > -> MapScanLogicalRel
>> >
>> > Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but
>> cannot
>> > satisfy the SINGLETON property so it propagates the demand
>> >
>> > RootPhysicalRel
>> > -> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>> > -> MapScanLogicalRel
>> >
>> > Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the algorithm
>> but
>> > cannot satisfy the SINGLETON property so it propagates the demand
>> >
>> > RootPhysicalRel
>> > -> ProjectPhysicalRel
>> > -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>> >
>> > Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the algorithm
>> and
>> > given that the table is replicated it satisfies the distribution
>> property.
>> >
>> > RootPhysicalRel
>> > -> ProjectPhysicalRel
>> > -> MapScanPhysicalRel
>> >
>> > So we end up with a complete plan that satisfies all properties and does
>> > not have redundant exchange operators.
>> > Moreover we didn't require all possible distributions when we applied
>> Rule
>> > 2 since we just want to satisfy the SINGLETON property requested by the
>> > parent.
>> > The example above does not demonstrate the additional optimization paths
>> > which would be apply an enforcer (an Exchange operator to satisfy the
>> > SINGLETON property) at a higher level than the scan.
>> >
>> > Would this be an acceptable approach? Does it still suffer from a big
>> > search space?
>> >
>> > To make the analog with the actual Volcano optimizer in Calcite the
>> thing
>> > that may be missing is to be able to know what trait was requested by
>> the
>> > parent during the application of Rule 2.
>> >
>> > Best,
>> > Stamatis
>> >
>> > [1]
>> >
>> >
>> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
>> >
>> > On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <pp...@gmail.com>
>> > wrote:
>> >
>> > > Hi Stamatis,
>> > >
>> > > The problem that the presented reproducer is a very simplified
>> version of
>> > > what is actually needed. In reality, there are several distribution
>> > types -
>> > > PARTITIONED, REPLICATED, SINGLETON. To make things worse, PARTITIONED
>> may
>> > > or may not have concrete distribution fields. In theory, I can create
>> one
>> > > transformation per distribution type, but that would increase plan
>> space
>> > > significantly. In my sample "Root <- Project <- Scan" plan, there is
>> no
>> > > room for optimization at all, we only need to convert the nodes and
>> > > propagate traits. But if I follow the proposed approach, the planner
>> > would
>> > > create three equivalent paths. For complex queries, this may increase
>> > > optimization time significantly.
>> > >
>> > > What I need instead is to gradually convert and optimize nodes
>> > *bottom-up*,
>> > > instead of top-bottom. That is, create a physical scan first, then
>> > create a
>> > > physical project on top of it, etc. But at the same time, some rules
>> > still
>> > > require the top-bottom approach. So essentially I need the optimizer
>> to
>> > do
>> > > both. Abstract converters help me establish bottom-up preparation but
>> do
>> > > this at the cost of considering too many trait pairs, and as a result,
>> > also
>> > > pollute the search space.
>> > >
>> > > To the contrast, precise command "I transformed the node A to node B,
>> > > please re-trigger the rules for A's parents" would allow us to
>> re-trigger
>> > > only required rules, without adding more nodes.
>> > >
>> > > Does it make sense?
>> > >
>> > > Regards,
>> > > Vladimir.
>> > >
>> > > ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <za...@gmail.com>:
>> > >
>> > > > I admit that I didn't thoroughly read the exchanges so far but
>> forcing
>> > > the
>> > > > rules trigger does not seem the right approach.
>> > > >
>> > > > Other than that I would like to backtrack a bit to point 4.3 raised
>> by
>> > > > Vladimir.
>> > > >
>> > > > "ProjectPhysicalRule [6] - transforms logical project to physical
>> > > > project *ONLY* if there is an underlying physical input with
>> REPLICATED
>> > > or
>> > > > SINGLETON distribution"
>> > > >
>> > > > The rule could be modified to do the following two transformations:
>> > > > 1. Create a physical project and require the input to be REPLICATED.
>> > > > 2. Create a physical project and require
>> > > > the input to be SINGLETON.
>> > > >
>> > > > I would assume that afterwards when your scan rule fires it should
>> go
>> > to
>> > > > the appropriate subset and give you back the desired plan. Is there
>> a
>> > > > problem with this approach?
>> > > >
>> > > > Best,
>> > > > Stamatis
>> > > >
>> > > > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <
>> gvvinblade@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Unfortunately it requires package-private API usage.
>> > > > >
>> > > > > Best solution there if Calcite provide it for end users.
>> > > > >
>> > > > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
>> > > > >
>> > > > > > “e pension” == “expand conversion” :)
>> > > > > >
>> > > > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <
>> ppozerov@gmail.com>:
>> > > > > >
>> > > > > > > Yes, that may work. Even if e pension rule is used, for the
>> most
>> > > > cases
>> > > > > it
>> > > > > > > will not trigger any real conversions, since we are moving
>> from
>> > > > > abstract
>> > > > > > > convention to physical, and created converters will have the
>> > > opposite
>> > > > > > trait
>> > > > > > > direction (from physical to abstract).
>> > > > > > >
>> > > > > > > But again - ideally we only need to re-trigger the rules for a
>> > > > specific
>> > > > > > > node, no more than that. So API support like
>> > > > > > > “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
>> > > > > > >
>> > > > > > > What do you think?
>> > > > > > >
>> > > > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
>> > > gvvinblade@gmail.com
>> > > > >:
>> > > > > > >
>> > > > > > >> I considered manual rules calling too, for now we use
>> abstract
>> > > > > > converters
>> > > > > > >> +
>> > > > > > >> ExpandConversionRule for exchanges producing.
>> > > > > > >>
>> > > > > > >> You may create such converters manually (checking appropriate
>> > > > subset)
>> > > > > > this
>> > > > > > >> case you may reduce created converters count, also, a
>> converter
>> > > is a
>> > > > > > quite
>> > > > > > >> special node, that does almost nothing (without corresponding
>> > > rule)
>> > > > it
>> > > > > > may
>> > > > > > >> be used just as a rule trigger.
>> > > > > > >>
>> > > > > > >> Regards,
>> > > > > > >> Igor
>> > > > > > >>
>> > > > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <
>> ppozerov@gmail.com
>> > >:
>> > > > > > >>
>> > > > > > >> > One funny hack which helped me is manual registration of a
>> > fake
>> > > > > > RelNode
>> > > > > > >> > with desired traits through VolcanoPlanner.register()
>> method.
>> > > But
>> > > > > > again,
>> > > > > > >> > this leads to trashing. What could really help is a call to
>> > > > > > >> > VolcanoPlanner.fireRules() with desired rel. But this
>> doesn't
>> > > work
>> > > > > out
>> > > > > > >> of
>> > > > > > >> > the box since some internals of the rule queue needs to be
>> > > > adjusted.
>> > > > > > >> >
>> > > > > > >> > What does the community think about adding a method which
>> will
>> > > > > re-add
>> > > > > > >> rules
>> > > > > > >> > applicable to the specific RelNode to the rule queue?
>> > > > > > >> >
>> > > > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
>> > > ppozerov@gmail.com
>> > > > >:
>> > > > > > >> >
>> > > > > > >> > > Hi Igor,
>> > > > > > >> > >
>> > > > > > >> > > Yes, I came to the same conclusion, thank you. This is
>> how
>> > it
>> > > > > > >> basically
>> > > > > > >> > > happens when converters are disabled:
>> > > > > > >> > > 1) We start with initial tree: [LogicalProject] <-
>> > > [LogicalScan]
>> > > > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is
>> > added
>> > > > to
>> > > > > > the
>> > > > > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
>> > > > > > >> > > 3) Finally, when it is time to fire a rule for
>> PhysicalScan,
>> > > we
>> > > > > try
>> > > > > > to
>> > > > > > >> > get
>> > > > > > >> > > parents of that scan set with traits of the PhysicalScan.
>> > > Since
>> > > > > > there
>> > > > > > >> are
>> > > > > > >> > > no such parents (we skipped it intentionally), the rule
>> is
>> > not
>> > > > > > queued.
>> > > > > > >> > >
>> > > > > > >> > > But when converters are enabled, a converter rel is
>> created:
>> > > > > > >> > [LogicalProject]
>> > > > > > >> > > <- [LogicalScan, PhysicalScan,
>> > > ConverterFromPhysicalToLogical].
>> > > > No
>> > > > > > >> rules
>> > > > > > >> > > are fired for PhysicalScan again, but they are fired for
>> > > > converter
>> > > > > > >> since
>> > > > > > >> > > it has the necessary LOGICAL trait.
>> > > > > > >> > >
>> > > > > > >> > > It makes sense, that converters essentially allow forcing
>> > rule
>> > > > > > >> invocation
>> > > > > > >> > > on parents, even if the child was created with different
>> > > traits.
>> > > > > But
>> > > > > > >> it
>> > > > > > >> > > seems that this mechanism may be too heavy for complex
>> > queries
>> > > > > > >> because it
>> > > > > > >> > > literally creates hundreds of new converter rels and
>> > triggers
>> > > > > rules
>> > > > > > >> over
>> > > > > > >> > > and over again.
>> > > > > > >> > >
>> > > > > > >> > > We need some fine-grained alternative. Basically, what
>> would
>> > > be
>> > > > > > enough
>> > > > > > >> > for
>> > > > > > >> > > me is to let the planner know somehow: "I created that
>> rel,
>> > > and
>> > > > I
>> > > > > > want
>> > > > > > >> > you
>> > > > > > >> > > to execute parent rules not only using its trait but
>> also on
>> > > > this
>> > > > > > and
>> > > > > > >> > those
>> > > > > > >> > > traits."
>> > > > > > >> > > Is there any API in Calcite which allows doing this
>> without
>> > > > > > creating a
>> > > > > > >> > new
>> > > > > > >> > > rel node?
>> > > > > > >> > >
>> > > > > > >> > > Regards,
>> > > > > > >> > > Vladimir.
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
>> > > > > gvvinblade@gmail.com
>> > > > > > >:
>> > > > > > >> > >
>> > > > > > >> > >> Vladimir,
>> > > > > > >> > >>
>> > > > > > >> > >> Probably it'll help you:
>> > > > > > >> > >>
>> > > > > > >> > >> Seems the cause of issue in RelSubset.getParentRels()
>> The
>> > > > check
>> > > > > > used
>> > > > > > >> > when
>> > > > > > >> > >> the planner schedules newly matched rules after
>> successful
>> > > > > > >> > transformation
>> > > > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the
>> parent
>> > > rule
>> > > > > be
>> > > > > > >> > applied
>> > > > > > >> > >> once again (here your logical project with an input
>> having
>> > > ANY
>> > > > > > >> > >> distribution
>> > > > > > >> > >> doesn't satisfy a transformed input traits).
>> > > > > > >> > >>
>> > > > > > >> > >> In our case we use another workaround, so there are also
>> > much
>> > > > > more
>> > > > > > >> > >> transformations than we wanted, so the desired rule is
>> > > > triggered.
>> > > > > > >> > >>
>> > > > > > >> > >>
>> > > > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
>> > > ppozerov@gmail.com
>> > > > >:
>> > > > > > >> > >>
>> > > > > > >> > >> > Hi Vladimir,
>> > > > > > >> > >> >
>> > > > > > >> > >> > I am sorry. Pushed, it works now.
>> > > > > > >> > >> >
>> > > > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
>> > > > > > >> > >> > sitnikov.vladimir@gmail.com
>> > > > > > >> > >> > >:
>> > > > > > >> > >> >
>> > > > > > >> > >> > > > mvn clean test
>> > > > > > >> > >> > >
>> > > > > > >> > >> > > [ERROR] The goal you specified requires a project to
>> > > > execute
>> > > > > > but
>> > > > > > >> > >> there is
>> > > > > > >> > >> > > no POM in this directory
>> > > > > > >> > >> > >
>> > > > > > >> > >> > > Vladimir, please push missing files
>> > > > > > >> > >> > >
>> > > > > > >> > >> > > Vladimir
>> > > > > > >> > >> > >
>> > > > > > >> > >> >
>> > > > > > >> > >>
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>

Re: Re: Re: Problem with converters and possibly rule matching

Posted by Haisheng Yuan <h....@alibaba-inc.com>.
Hi Vladimir,

The code in PHYSICAL convention L44 looks weird, I think it always returns true.
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/HazelcastConventions.java#L44

Try this:
fromTraits.containsIfApplicable(Convention.PHYSICAL)
    && toTraits.containsIfApplicable(Convention.PHYSICAL);

Adding a AbstractConverter on logical operators is meaningless. Calcite is mixing the concept of logical and physical together, which is sad.

BTW, using 2 conventions is not appropriate and wrong.

- Haisheng

------------------------------------------------------------------
发件人:Vladimir Ozerov<pp...@gmail.com>
日 期:2019年11月05日 18:02:15
收件人:Haisheng Yuan<h....@alibaba-inc.com>
抄 送:dev@calcite.apache.org (dev@calcite.apache.org)<de...@calcite.apache.org>
主 题:Re: Re: Problem with converters and possibly rule matching

Hi Haisheng,

 think I already tried something very similar to what you explained, but it gave not an optimal plan. Please let me describe what I did. I would appreciate your feedback. 

1) We start with a simple operator tree Root <- Project <- Scan, where the root is a final aggregator in the distributed query engine:
-> LogicalRoot
 -> LogicalProject
  -> LogicalScan

2) First, we convert the Root and enforce SINGLETON distribution on a child:
-> PhysicalRoot[SINGLETON]
 -> Enforcer#1[SINGLETON]
  -> LogicalProject
   -> LogicalScan

3) Then the project's rule is invoked. It doesn't know the distribution of the input, so it requests ANY distribution. Note that we have to set ANY to the project as well since we do not know the distribution of the input:
-> PhysicalRoot[SINGLETON]
 -> Enforcer#1[SINGLETON]
  -> PhysicalProject[ANY]
   -> Enforcer#2[ANY]
    -> LogicalScan

4) Finally, the physical scan is created and its distribution is resolved. Suppose that it is REPLICATED, i.e. the whole result set is located on all nodes.
-> PhysicalRoot[SINGLETON]
 -> Enforcer#1[SINGLETON]
  -> PhysicalProject[ANY]
   -> Enforcer#2[ANY]
    -> PhysicalScan[REPLICATED]

5) Now as all logical nodes are converted, we start resolving enforcers. The second one is no-op, since REPLICATED satisfies ANY:
-> PhysicalRoot[SINGLETON]
 -> Enforcer#1[SINGLETON]
  -> PhysicalProject[ANY]
   -> PhysicalScan[REPLICATED]

6) But the first enforcer now requires an Exchange, since ANY doesn't satisfy SINGLETON!
-> PhysicalRoot[SINGLETON]
 -> SingletonExchange[SINGLETON]
  -> PhysicalProject[ANY] // <= unresolved!
   -> PhysicalScan[REPLICATED]

The resulting plan requires data movement only because we didn't know precise distribution of the PhysicalProject when it was created. But should I enable Convention.Impl.canConvertConvention, bottom-up propagation kicks in, and the correct plan is produced because now LogicalProject has a chance to be converted to PhysicalProject with the concrete distribution. The optimized plan looks like this (since REPLICATED satisfies SINGLETON):
-> PhysicalRoot[SINGLETON]
 -> PhysicalProject[REPLICATED]
  -> PhysicalScan[REPLICATED]
You may see this in action in my reproducer:
1) Test producing "bad" plan: https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L45
2) Root enforces SINGLETON on Project: https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/RootPhysicalRule.java#L45
3) Project enforces default (ANY) distribution on Scan: https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L49

Please let me know if this flow is similar to what you meant.

Regards,
Vladimir.
пн, 4 нояб. 2019 г. в 10:33, Haisheng Yuan <h....@alibaba-inc.com>:

Hi Vladimir,

This is still can be done through top-down request approach.

PhysicalFilter operator should request ANY distribution from child operator, unless there is outer reference in the filter condition, in which case, PhysicalFilter should request SINGLETON or BROADCAST distribution. So in your case, PhysicalFilter request ANY, its required distribution will be enforced on filter's output.

Regarding index usage, you should have a FIlterTableScan2IndexGet logical transformation rule, and a IndexGet2IndexScan physical implementation rule. Note that IndexGet is a logical operator and IndexScan is a physical operator, which are also used by SQL Server.

- Haisheng

------------------------------------------------------------------
发件人:Vladimir Ozerov<pp...@gmail.com>
日 期:2019年11月01日 17:30:26
收件人:<de...@calcite.apache.org>
主 题:Re: Problem with converters and possibly rule matching

Hi Stamatis,

Thank you for your reply. I also thought that we may set the distribution
trait during logical planning because it is known in advance. And the
example I gave will work! :-) But unfortunately, it will work only because
the tree is very simple, and the Project is adjacent to the Scan. This is
how my reproducer will work in that case:
1) Root: enforce "SINGLETON" on Project
2) Project: check the logical Scan, infer already resolved distribution,
then convert to [PhyiscalProject <- PhysicalScan]
3) Resolve Root enforcer, adding and Exchange if needed.

But this stops working as soon as a plan becomes more complex so that it is
impossible to infer the distribution from the child immediately. E.g.:
LogicalRoot [distribution=SINGLETON]
-> LogicalProject // We are here new and cannot produce the physical project
 -> LogicalFilter[distribution=?]
 -> LogicalScan[distribution=REPLICATED]

This is where your suggestion with cascading enforcement may kick in. But
now consider that instead of having REPLICATED distribution, which
satisfies SINGLETON, we have a PARTITIONED distribution, It doesn't satisfy
SINGLETON, so we have to insert the exchange.

Before:
PhysicalRoot [SINGLETON]
-> PhysicalProject [SINGLETON]
 -> PhysicalFilter [SINGLETON]
 -> LogicalScan [PARTITIONED] // SINGLETON is enforced here

After:
PhysicalRoot [SINGLETON]
-> PhysicalProject [SINGLETON]
 -> PhysicalFilter [SINGLETON]
 -> SingletonExchange [SINGLETON]
 -> PhysicalScan [PARTITIONED]

But unfortunately, this plan is not optimal. Since we perform Project and
Filter after the exchange, we now have to reshuffle the whole table. The
optimal plan would be to pushdown Project and Filter past exchange:
PhysicalRoot [SINGLETON]
-> SingletonExchange [SINGLETON]
 -> PhysicalProject [PARTITIONED]
 -> PhysicalFilter [PARTITIONED]
 -> PhysicalScan [PARTITIONED]

But in order to achieve that, now I need to introduce new rules which will
attempt to transpose dozens of different operators past exchange, so most
likely the planning will never finish :-)

This is why it seems that the bottom-up approach seems to be more natural
for such cases. Another example is index support. A table may have a number
of access methods in addition to plain scan, and every such method may
produce a physical scan with different collations. It is not practical to
try to enforce something from the top. Instead, we'd better produce
alternatives from the bottom.

Basically, Drill tries to mix two approaches. First, it tries to derive
traits from the child. If it fails and no transformations were created, it
just stops trait propagation. As a result, an exchange is created on top of
the operator where we stopped, which again leads to not optimal plans. You
may observe this in action if you run my reproducer. "testPass" produces an
optimal plan with help of abstract converters. "testDrill" produces not
optimal plan with the unnecessary exchange.

All in all, I cannot get how we can employ distribution metadata and index
metadata for optimal planning without using a pure bottom-up approach.

Regards,
Vladimir.

пт, 1 нояб. 2019 г. в 11:42, Stamatis Zampetakis <za...@gmail.com>:

> The discussion is interesting thanks for the nice examples.
> I am replying in this thread since it has all the examples and the history
> of the conversation.
>
> *Part I: Distribution physical or logical property*
>
> The Volcano paper writes the following regarding properties:
>
> "Logical properties can be derived from the logical algebra expression and
> include schema, expected size, etc., while physical properties depend on
> algorithms, e.g., sort order, partitioning, etc."
>
> We could say that partitioning is not only a property which depends on an
> algorithm but a property which depends on the schema. When you have a table
> you know in advance that the particular table is replicated.
>
> Basically, this means that the starting "logical" plan in your case could
> be the following:
> RootLogicalRel[convention=LOGICAL, distribution=ANY]
> -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
> -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
>
> When you are about to create your physical projection it is not necessary
> to create three equivalent paths but the most promising by asking the input
> what can it offer as traits. Again this partially overlaps with the
> discussion about "On demand traitset" [1] where as I wrote there is already
> a mechanism to derive traits from children (collation, distribution, etc.).
>
> Note that I am not saying that there is nothing more to be done to improve
> the Calcite optimizer; I am just trying to provide a viable alternative
> given the current situation of the project.
>
> *Part II: Example with the original Volcano optimizer*
>
> Now let me try to show how the original (from the paper) Volcano optimizer
> could handle your query.
>
> The optimization starts and we request two properties convention and
> distribution.
>
> RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
> -> ProjectLogicalRel
> -> MapScanLogicalRel
>
> Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but cannot
> satisfy the SINGLETON property so it propagates the demand
>
> RootPhysicalRel
> -> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
> -> MapScanLogicalRel
>
> Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the algorithm but
> cannot satisfy the SINGLETON property so it propagates the demand
>
> RootPhysicalRel
> -> ProjectPhysicalRel
> -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>
> Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the algorithm and
> given that the table is replicated it satisfies the distribution property.
>
> RootPhysicalRel
> -> ProjectPhysicalRel
> -> MapScanPhysicalRel
>
> So we end up with a complete plan that satisfies all properties and does
> not have redundant exchange operators.
> Moreover we didn't require all possible distributions when we applied Rule
> 2 since we just want to satisfy the SINGLETON property requested by the
> parent.
> The example above does not demonstrate the additional optimization paths
> which would be apply an enforcer (an Exchange operator to satisfy the
> SINGLETON property) at a higher level than the scan.
>
> Would this be an acceptable approach? Does it still suffer from a big
> search space?
>
> To make the analog with the actual Volcano optimizer in Calcite the thing
> that may be missing is to be able to know what trait was requested by the
> parent during the application of Rule 2.
>
> Best,
> Stamatis
>
> [1]
>
> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
>
> On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <pp...@gmail.com>
> wrote:
>
> > Hi Stamatis,
> >
> > The problem that the presented reproducer is a very simplified version of
> > what is actually needed. In reality, there are several distribution
> types -
> > PARTITIONED, REPLICATED, SINGLETON. To make things worse, PARTITIONED may
> > or may not have concrete distribution fields. In theory, I can create one
> > transformation per distribution type, but that would increase plan space
> > significantly. In my sample "Root <- Project <- Scan" plan, there is no
> > room for optimization at all, we only need to convert the nodes and
> > propagate traits. But if I follow the proposed approach, the planner
> would
> > create three equivalent paths. For complex queries, this may increase
> > optimization time significantly.
> >
> > What I need instead is to gradually convert and optimize nodes
> *bottom-up*,
> > instead of top-bottom. That is, create a physical scan first, then
> create a
> > physical project on top of it, etc. But at the same time, some rules
> still
> > require the top-bottom approach. So essentially I need the optimizer to
> do
> > both. Abstract converters help me establish bottom-up preparation but do
> > this at the cost of considering too many trait pairs, and as a result,
> also
> > pollute the search space.
> >
> > To the contrast, precise command "I transformed the node A to node B,
> > please re-trigger the rules for A's parents" would allow us to re-trigger
> > only required rules, without adding more nodes.
> >
> > Does it make sense?
> >
> > Regards,
> > Vladimir.
> >
> > ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <za...@gmail.com>:
> >
> > > I admit that I didn't thoroughly read the exchanges so far but forcing
> > the
> > > rules trigger does not seem the right approach.
> > >
> > > Other than that I would like to backtrack a bit to point 4.3 raised by
> > > Vladimir.
> > >
> > > "ProjectPhysicalRule [6] - transforms logical project to physical
> > > project *ONLY* if there is an underlying physical input with REPLICATED
> > or
> > > SINGLETON distribution"
> > >
> > > The rule could be modified to do the following two transformations:
> > > 1. Create a physical project and require the input to be REPLICATED.
> > > 2. Create a physical project and require
> > > the input to be SINGLETON.
> > >
> > > I would assume that afterwards when your scan rule fires it should go
> to
> > > the appropriate subset and give you back the desired plan. Is there a
> > > problem with this approach?
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <gv...@gmail.com>
> > > wrote:
> > >
> > > > Unfortunately it requires package-private API usage.
> > > >
> > > > Best solution there if Calcite provide it for end users.
> > > >
> > > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
> > > >
> > > > > “e pension” == “expand conversion” :)
> > > > >
> > > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <pp...@gmail.com>:
> > > > >
> > > > > > Yes, that may work. Even if e pension rule is used, for the most
> > > cases
> > > > it
> > > > > > will not trigger any real conversions, since we are moving from
> > > > abstract
> > > > > > convention to physical, and created converters will have the
> > opposite
> > > > > trait
> > > > > > direction (from physical to abstract).
> > > > > >
> > > > > > But again - ideally we only need to re-trigger the rules for a
> > > specific
> > > > > > node, no more than that. So API support like
> > > > > > “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
> > gvvinblade@gmail.com
> > > >:
> > > > > >
> > > > > >> I considered manual rules calling too, for now we use abstract
> > > > > converters
> > > > > >> +
> > > > > >> ExpandConversionRule for exchanges producing.
> > > > > >>
> > > > > >> You may create such converters manually (checking appropriate
> > > subset)
> > > > > this
> > > > > >> case you may reduce created converters count, also, a converter
> > is a
> > > > > quite
> > > > > >> special node, that does almost nothing (without corresponding
> > rule)
> > > it
> > > > > may
> > > > > >> be used just as a rule trigger.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Igor
> > > > > >>
> > > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <ppozerov@gmail.com
> >:
> > > > > >>
> > > > > >> > One funny hack which helped me is manual registration of a
> fake
> > > > > RelNode
> > > > > >> > with desired traits through VolcanoPlanner.register() method.
> > But
> > > > > again,
> > > > > >> > this leads to trashing. What could really help is a call to
> > > > > >> > VolcanoPlanner.fireRules() with desired rel. But this doesn't
> > work
> > > > out
> > > > > >> of
> > > > > >> > the box since some internals of the rule queue needs to be
> > > adjusted.
> > > > > >> >
> > > > > >> > What does the community think about adding a method which will
> > > > re-add
> > > > > >> rules
> > > > > >> > applicable to the specific RelNode to the rule queue?
> > > > > >> >
> > > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
> > ppozerov@gmail.com
> > > >:
> > > > > >> >
> > > > > >> > > Hi Igor,
> > > > > >> > >
> > > > > >> > > Yes, I came to the same conclusion, thank you. This is how
> it
> > > > > >> basically
> > > > > >> > > happens when converters are disabled:
> > > > > >> > > 1) We start with initial tree: [LogicalProject] <-
> > [LogicalScan]
> > > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is
> added
> > > to
> > > > > the
> > > > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > > > > >> > > 3) Finally, when it is time to fire a rule for PhysicalScan,
> > we
> > > > try
> > > > > to
> > > > > >> > get
> > > > > >> > > parents of that scan set with traits of the PhysicalScan.
> > Since
> > > > > there
> > > > > >> are
> > > > > >> > > no such parents (we skipped it intentionally), the rule is
> not
> > > > > queued.
> > > > > >> > >
> > > > > >> > > But when converters are enabled, a converter rel is created:
> > > > > >> > [LogicalProject]
> > > > > >> > > <- [LogicalScan, PhysicalScan,
> > ConverterFromPhysicalToLogical].
> > > No
> > > > > >> rules
> > > > > >> > > are fired for PhysicalScan again, but they are fired for
> > > converter
> > > > > >> since
> > > > > >> > > it has the necessary LOGICAL trait.
> > > > > >> > >
> > > > > >> > > It makes sense, that converters essentially allow forcing
> rule
> > > > > >> invocation
> > > > > >> > > on parents, even if the child was created with different
> > traits.
> > > > But
> > > > > >> it
> > > > > >> > > seems that this mechanism may be too heavy for complex
> queries
> > > > > >> because it
> > > > > >> > > literally creates hundreds of new converter rels and
> triggers
> > > > rules
> > > > > >> over
> > > > > >> > > and over again.
> > > > > >> > >
> > > > > >> > > We need some fine-grained alternative. Basically, what would
> > be
> > > > > enough
> > > > > >> > for
> > > > > >> > > me is to let the planner know somehow: "I created that rel,
> > and
> > > I
> > > > > want
> > > > > >> > you
> > > > > >> > > to execute parent rules not only using its trait but also on
> > > this
> > > > > and
> > > > > >> > those
> > > > > >> > > traits."
> > > > > >> > > Is there any API in Calcite which allows doing this without
> > > > > creating a
> > > > > >> > new
> > > > > >> > > rel node?
> > > > > >> > >
> > > > > >> > > Regards,
> > > > > >> > > Vladimir.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> > > > gvvinblade@gmail.com
> > > > > >:
> > > > > >> > >
> > > > > >> > >> Vladimir,
> > > > > >> > >>
> > > > > >> > >> Probably it'll help you:
> > > > > >> > >>
> > > > > >> > >> Seems the cause of issue in RelSubset.getParentRels() The
> > > check
> > > > > used
> > > > > >> > when
> > > > > >> > >> the planner schedules newly matched rules after successful
> > > > > >> > transformation
> > > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the parent
> > rule
> > > > be
> > > > > >> > applied
> > > > > >> > >> once again (here your logical project with an input having
> > ANY
> > > > > >> > >> distribution
> > > > > >> > >> doesn't satisfy a transformed input traits).
> > > > > >> > >>
> > > > > >> > >> In our case we use another workaround, so there are also
> much
> > > > more
> > > > > >> > >> transformations than we wanted, so the desired rule is
> > > triggered.
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
> > ppozerov@gmail.com
> > > >:
> > > > > >> > >>
> > > > > >> > >> > Hi Vladimir,
> > > > > >> > >> >
> > > > > >> > >> > I am sorry. Pushed, it works now.
> > > > > >> > >> >
> > > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > > > > >> > >> > sitnikov.vladimir@gmail.com
> > > > > >> > >> > >:
> > > > > >> > >> >
> > > > > >> > >> > > > mvn clean test
> > > > > >> > >> > >
> > > > > >> > >> > > [ERROR] The goal you specified requires a project to
> > > execute
> > > > > but
> > > > > >> > >> there is
> > > > > >> > >> > > no POM in this directory
> > > > > >> > >> > >
> > > > > >> > >> > > Vladimir, please push missing files
> > > > > >> > >> > >
> > > > > >> > >> > > Vladimir
> > > > > >> > >> > >
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>



Re: Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
Hi Haisheng,

 think I already tried something very similar to what you explained, but it
gave not an optimal plan. Please let me describe what I did. I would
appreciate your feedback.

1) We start with a simple operator tree Root <- Project <- Scan, where the
root is a final aggregator in the distributed query engine:
-> LogicalRoot
 -> LogicalProject
  -> LogicalScan

2) First, we convert the Root and enforce SINGLETON distribution on a child:
*-> PhysicalRoot[SINGLETON]*
* -> Enforcer#1[SINGLETON]*
  -> LogicalProject
   -> LogicalScan

3) Then the project's rule is invoked. It doesn't know the distribution of
the input, so it requests ANY distribution. Note that we have to set ANY to
the project as well since we do not know the distribution of the input:
-> PhysicalRoot[SINGLETON]
 -> Enforcer#1[SINGLETON]
*  -> PhysicalProject[ANY]*
*   -> Enforcer#2[ANY]*
    -> LogicalScan

4) Finally, the physical scan is created and its distribution is resolved.
Suppose that it is REPLICATED, i.e. the whole result set is located on all
nodes.
-> PhysicalRoot[SINGLETON]
 -> Enforcer#1[SINGLETON]
  -> PhysicalProject[ANY]
   -> Enforcer#2[ANY]
*    -> PhysicalScan[REPLICATED]*

5) Now as all logical nodes are converted, we start resolving enforcers.
The second one is no-op, since REPLICATED satisfies ANY:
-> PhysicalRoot[SINGLETON]
 -> Enforcer#1[SINGLETON]
  -> PhysicalProject[ANY]
   -> PhysicalScan[REPLICATED]

6) But the first enforcer now requires an Exchange, since ANY doesn't
satisfy SINGLETON!
-> PhysicalRoot[SINGLETON]
* -> SingletonExchange[SINGLETON]*
  -> PhysicalProject[ANY] // <= unresolved!
   -> PhysicalScan[REPLICATED]

The resulting plan requires data movement only because we didn't know
precise distribution of the PhysicalProject when it was created. But should
I enable Convention.Impl.canConvertConvention, bottom-up propagation kicks
in, and the correct plan is produced because now LogicalProject has a
chance to be converted to PhysicalProject with the concrete distribution.
The optimized plan looks like this (since REPLICATED satisfies SINGLETON):
-> PhysicalRoot[SINGLETON]
 -> PhysicalProject[REPLICATED]
  -> PhysicalScan[REPLICATED]

You may see this in action in my reproducer:
1) Test producing "bad" plan:
https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java#L45
2) Root enforces SINGLETON on Project:
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/RootPhysicalRule.java#L45
3) Project enforces default (ANY) distribution on Scan:
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java#L49

Please let me know if this flow is similar to what you meant.

Regards,
Vladimir.

пн, 4 нояб. 2019 г. в 10:33, Haisheng Yuan <h....@alibaba-inc.com>:

> Hi Vladimir,
>
> This is still can be done through top-down request approach.
>
> PhysicalFilter operator should request ANY distribution from child
> operator, unless there is outer reference in the filter condition, in which
> case, PhysicalFilter should request SINGLETON or BROADCAST distribution. So
> in your case, PhysicalFilter request ANY, its required distribution will be
> enforced on filter's output.
>
> Regarding index usage, you should have a FIlterTableScan2IndexGet logical
> transformation rule, and a IndexGet2IndexScan physical implementation rule.
> Note that IndexGet is a logical operator and IndexScan is a physical
> operator, which are also used by SQL Server.
>
> - Haisheng
>
> ------------------------------------------------------------------
> 发件人:Vladimir Ozerov<pp...@gmail.com>
> 日 期:2019年11月01日 17:30:26
> 收件人:<de...@calcite.apache.org>
> 主 题:Re: Problem with converters and possibly rule matching
>
> Hi Stamatis,
>
> Thank you for your reply. I also thought that we may set the distribution
> trait during logical planning because it is known in advance. And the
> example I gave will work! :-) But unfortunately, it will work only because
> the tree is very simple, and the Project is adjacent to the Scan. This is
> how my reproducer will work in that case:
> 1) Root: enforce "SINGLETON" on Project
> 2) Project: check the logical Scan, infer already resolved distribution,
> then convert to [PhyiscalProject <- PhysicalScan]
> 3) Resolve Root enforcer, adding and Exchange if needed.
>
> But this stops working as soon as a plan becomes more complex so that it is
> impossible to infer the distribution from the child immediately. E.g.:
> LogicalRoot [distribution=SINGLETON]
> -> LogicalProject // We are here new and cannot produce the physical
> project
> -> LogicalFilter[distribution=?]
> -> LogicalScan[distribution=REPLICATED]
>
> This is where your suggestion with cascading enforcement may kick in. But
> now consider that instead of having REPLICATED distribution, which
> satisfies SINGLETON, we have a PARTITIONED distribution, It doesn't satisfy
> SINGLETON, so we have to insert the exchange.
>
> Before:
> PhysicalRoot [SINGLETON]
> -> PhysicalProject [SINGLETON]
> -> PhysicalFilter [SINGLETON]
> -> LogicalScan [PARTITIONED] // SINGLETON is enforced here
>
> After:
> PhysicalRoot [SINGLETON]
> -> PhysicalProject [SINGLETON]
> -> PhysicalFilter [SINGLETON]
> -> SingletonExchange [SINGLETON]
> -> PhysicalScan [PARTITIONED]
>
> But unfortunately, this plan is not optimal. Since we perform Project and
> Filter after the exchange, we now have to reshuffle the whole table. The
> optimal plan would be to pushdown Project and Filter past exchange:
> PhysicalRoot [SINGLETON]
> -> SingletonExchange [SINGLETON]
> -> PhysicalProject [PARTITIONED]
> -> PhysicalFilter [PARTITIONED]
> -> PhysicalScan [PARTITIONED]
>
> But in order to achieve that, now I need to introduce new rules which will
> attempt to transpose dozens of different operators past exchange, so most
> likely the planning will never finish :-)
>
> This is why it seems that the bottom-up approach seems to be more natural
> for such cases. Another example is index support. A table may have a number
> of access methods in addition to plain scan, and every such method may
> produce a physical scan with different collations. It is not practical to
> try to enforce something from the top. Instead, we'd better produce
> alternatives from the bottom.
>
> Basically, Drill tries to mix two approaches. First, it tries to derive
> traits from the child. If it fails and no transformations were created, it
> just stops trait propagation. As a result, an exchange is created on top of
> the operator where we stopped, which again leads to not optimal plans. You
> may observe this in action if you run my reproducer. "testPass" produces an
> optimal plan with help of abstract converters. "testDrill" produces not
> optimal plan with the unnecessary exchange.
>
> All in all, I cannot get how we can employ distribution metadata and index
> metadata for optimal planning without using a pure bottom-up approach.
>
> Regards,
> Vladimir.
>
> пт, 1 нояб. 2019 г. в 11:42, Stamatis Zampetakis <za...@gmail.com>:
>
> > The discussion is interesting thanks for the nice examples.
> > I am replying in this thread since it has all the examples and the
> history
> > of the conversation.
> >
> > *Part I: Distribution physical or logical property*
> >
> > The Volcano paper writes the following regarding properties:
> >
> > "Logical properties can be derived from the logical algebra expression
> and
> > include schema, expected size, etc., while physical properties depend on
> > algorithms, e.g., sort order, partitioning, etc."
> >
> > We could say that partitioning is not only a property which depends on an
> > algorithm but a property which depends on the schema. When you have a
> table
> > you know in advance that the particular table is replicated.
> >
> > Basically, this means that the starting "logical" plan in your case could
> > be the following:
> > RootLogicalRel[convention=LOGICAL, distribution=ANY]
> > -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
> > -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
> >
> > When you are about to create your physical projection it is not necessary
> > to create three equivalent paths but the most promising by asking the
> input
> > what can it offer as traits. Again this partially overlaps with the
> > discussion about "On demand traitset" [1] where as I wrote there is
> already
> > a mechanism to derive traits from children (collation, distribution,
> etc.).
> >
> > Note that I am not saying that there is nothing more to be done to
> improve
> > the Calcite optimizer; I am just trying to provide a viable alternative
> > given the current situation of the project.
> >
> > *Part II: Example with the original Volcano optimizer*
> >
> > Now let me try to show how the original (from the paper) Volcano
> optimizer
> > could handle your query.
> >
> > The optimization starts and we request two properties convention and
> > distribution.
> >
> > RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
> > -> ProjectLogicalRel
> > -> MapScanLogicalRel
> >
> > Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but
> cannot
> > satisfy the SINGLETON property so it propagates the demand
> >
> > RootPhysicalRel
> > -> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
> > -> MapScanLogicalRel
> >
> > Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the algorithm but
> > cannot satisfy the SINGLETON property so it propagates the demand
> >
> > RootPhysicalRel
> > -> ProjectPhysicalRel
> > -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
> >
> > Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the algorithm and
> > given that the table is replicated it satisfies the distribution
> property.
> >
> > RootPhysicalRel
> > -> ProjectPhysicalRel
> > -> MapScanPhysicalRel
> >
> > So we end up with a complete plan that satisfies all properties and does
> > not have redundant exchange operators.
> > Moreover we didn't require all possible distributions when we applied
> Rule
> > 2 since we just want to satisfy the SINGLETON property requested by the
> > parent.
> > The example above does not demonstrate the additional optimization paths
> > which would be apply an enforcer (an Exchange operator to satisfy the
> > SINGLETON property) at a higher level than the scan.
> >
> > Would this be an acceptable approach? Does it still suffer from a big
> > search space?
> >
> > To make the analog with the actual Volcano optimizer in Calcite the thing
> > that may be missing is to be able to know what trait was requested by the
> > parent during the application of Rule 2.
> >
> > Best,
> > Stamatis
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
> >
> > On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <pp...@gmail.com>
> > wrote:
> >
> > > Hi Stamatis,
> > >
> > > The problem that the presented reproducer is a very simplified version
> of
> > > what is actually needed. In reality, there are several distribution
> > types -
> > > PARTITIONED, REPLICATED, SINGLETON. To make things worse, PARTITIONED
> may
> > > or may not have concrete distribution fields. In theory, I can create
> one
> > > transformation per distribution type, but that would increase plan
> space
> > > significantly. In my sample "Root <- Project <- Scan" plan, there is no
> > > room for optimization at all, we only need to convert the nodes and
> > > propagate traits. But if I follow the proposed approach, the planner
> > would
> > > create three equivalent paths. For complex queries, this may increase
> > > optimization time significantly.
> > >
> > > What I need instead is to gradually convert and optimize nodes
> > *bottom-up*,
> > > instead of top-bottom. That is, create a physical scan first, then
> > create a
> > > physical project on top of it, etc. But at the same time, some rules
> > still
> > > require the top-bottom approach. So essentially I need the optimizer to
> > do
> > > both. Abstract converters help me establish bottom-up preparation but
> do
> > > this at the cost of considering too many trait pairs, and as a result,
> > also
> > > pollute the search space.
> > >
> > > To the contrast, precise command "I transformed the node A to node B,
> > > please re-trigger the rules for A's parents" would allow us to
> re-trigger
> > > only required rules, without adding more nodes.
> > >
> > > Does it make sense?
> > >
> > > Regards,
> > > Vladimir.
> > >
> > > ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <za...@gmail.com>:
> > >
> > > > I admit that I didn't thoroughly read the exchanges so far but
> forcing
> > > the
> > > > rules trigger does not seem the right approach.
> > > >
> > > > Other than that I would like to backtrack a bit to point 4.3 raised
> by
> > > > Vladimir.
> > > >
> > > > "ProjectPhysicalRule [6] - transforms logical project to physical
> > > > project *ONLY* if there is an underlying physical input with
> REPLICATED
> > > or
> > > > SINGLETON distribution"
> > > >
> > > > The rule could be modified to do the following two transformations:
> > > > 1. Create a physical project and require the input to be REPLICATED.
> > > > 2. Create a physical project and require
> > > > the input to be SINGLETON.
> > > >
> > > > I would assume that afterwards when your scan rule fires it should go
> > to
> > > > the appropriate subset and give you back the desired plan. Is there a
> > > > problem with this approach?
> > > >
> > > > Best,
> > > > Stamatis
> > > >
> > > > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <gvvinblade@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Unfortunately it requires package-private API usage.
> > > > >
> > > > > Best solution there if Calcite provide it for end users.
> > > > >
> > > > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
> > > > >
> > > > > > “e pension” == “expand conversion” :)
> > > > > >
> > > > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <ppozerov@gmail.com
> >:
> > > > > >
> > > > > > > Yes, that may work. Even if e pension rule is used, for the
> most
> > > > cases
> > > > > it
> > > > > > > will not trigger any real conversions, since we are moving from
> > > > > abstract
> > > > > > > convention to physical, and created converters will have the
> > > opposite
> > > > > > trait
> > > > > > > direction (from physical to abstract).
> > > > > > >
> > > > > > > But again - ideally we only need to re-trigger the rules for a
> > > > specific
> > > > > > > node, no more than that. So API support like
> > > > > > > “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
> > > gvvinblade@gmail.com
> > > > >:
> > > > > > >
> > > > > > >> I considered manual rules calling too, for now we use abstract
> > > > > > converters
> > > > > > >> +
> > > > > > >> ExpandConversionRule for exchanges producing.
> > > > > > >>
> > > > > > >> You may create such converters manually (checking appropriate
> > > > subset)
> > > > > > this
> > > > > > >> case you may reduce created converters count, also, a
> converter
> > > is a
> > > > > > quite
> > > > > > >> special node, that does almost nothing (without corresponding
> > > rule)
> > > > it
> > > > > > may
> > > > > > >> be used just as a rule trigger.
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Igor
> > > > > > >>
> > > > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <
> ppozerov@gmail.com
> > >:
> > > > > > >>
> > > > > > >> > One funny hack which helped me is manual registration of a
> > fake
> > > > > > RelNode
> > > > > > >> > with desired traits through VolcanoPlanner.register()
> method.
> > > But
> > > > > > again,
> > > > > > >> > this leads to trashing. What could really help is a call to
> > > > > > >> > VolcanoPlanner.fireRules() with desired rel. But this
> doesn't
> > > work
> > > > > out
> > > > > > >> of
> > > > > > >> > the box since some internals of the rule queue needs to be
> > > > adjusted.
> > > > > > >> >
> > > > > > >> > What does the community think about adding a method which
> will
> > > > > re-add
> > > > > > >> rules
> > > > > > >> > applicable to the specific RelNode to the rule queue?
> > > > > > >> >
> > > > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
> > > ppozerov@gmail.com
> > > > >:
> > > > > > >> >
> > > > > > >> > > Hi Igor,
> > > > > > >> > >
> > > > > > >> > > Yes, I came to the same conclusion, thank you. This is how
> > it
> > > > > > >> basically
> > > > > > >> > > happens when converters are disabled:
> > > > > > >> > > 1) We start with initial tree: [LogicalProject] <-
> > > [LogicalScan]
> > > > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is
> > added
> > > > to
> > > > > > the
> > > > > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > > > > > >> > > 3) Finally, when it is time to fire a rule for
> PhysicalScan,
> > > we
> > > > > try
> > > > > > to
> > > > > > >> > get
> > > > > > >> > > parents of that scan set with traits of the PhysicalScan.
> > > Since
> > > > > > there
> > > > > > >> are
> > > > > > >> > > no such parents (we skipped it intentionally), the rule is
> > not
> > > > > > queued.
> > > > > > >> > >
> > > > > > >> > > But when converters are enabled, a converter rel is
> created:
> > > > > > >> > [LogicalProject]
> > > > > > >> > > <- [LogicalScan, PhysicalScan,
> > > ConverterFromPhysicalToLogical].
> > > > No
> > > > > > >> rules
> > > > > > >> > > are fired for PhysicalScan again, but they are fired for
> > > > converter
> > > > > > >> since
> > > > > > >> > > it has the necessary LOGICAL trait.
> > > > > > >> > >
> > > > > > >> > > It makes sense, that converters essentially allow forcing
> > rule
> > > > > > >> invocation
> > > > > > >> > > on parents, even if the child was created with different
> > > traits.
> > > > > But
> > > > > > >> it
> > > > > > >> > > seems that this mechanism may be too heavy for complex
> > queries
> > > > > > >> because it
> > > > > > >> > > literally creates hundreds of new converter rels and
> > triggers
> > > > > rules
> > > > > > >> over
> > > > > > >> > > and over again.
> > > > > > >> > >
> > > > > > >> > > We need some fine-grained alternative. Basically, what
> would
> > > be
> > > > > > enough
> > > > > > >> > for
> > > > > > >> > > me is to let the planner know somehow: "I created that
> rel,
> > > and
> > > > I
> > > > > > want
> > > > > > >> > you
> > > > > > >> > > to execute parent rules not only using its trait but also
> on
> > > > this
> > > > > > and
> > > > > > >> > those
> > > > > > >> > > traits."
> > > > > > >> > > Is there any API in Calcite which allows doing this
> without
> > > > > > creating a
> > > > > > >> > new
> > > > > > >> > > rel node?
> > > > > > >> > >
> > > > > > >> > > Regards,
> > > > > > >> > > Vladimir.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> > > > > gvvinblade@gmail.com
> > > > > > >:
> > > > > > >> > >
> > > > > > >> > >> Vladimir,
> > > > > > >> > >>
> > > > > > >> > >> Probably it'll help you:
> > > > > > >> > >>
> > > > > > >> > >> Seems the cause of issue in RelSubset.getParentRels() The
> > > > check
> > > > > > used
> > > > > > >> > when
> > > > > > >> > >> the planner schedules newly matched rules after
> successful
> > > > > > >> > transformation
> > > > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the
> parent
> > > rule
> > > > > be
> > > > > > >> > applied
> > > > > > >> > >> once again (here your logical project with an input
> having
> > > ANY
> > > > > > >> > >> distribution
> > > > > > >> > >> doesn't satisfy a transformed input traits).
> > > > > > >> > >>
> > > > > > >> > >> In our case we use another workaround, so there are also
> > much
> > > > > more
> > > > > > >> > >> transformations than we wanted, so the desired rule is
> > > > triggered.
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
> > > ppozerov@gmail.com
> > > > >:
> > > > > > >> > >>
> > > > > > >> > >> > Hi Vladimir,
> > > > > > >> > >> >
> > > > > > >> > >> > I am sorry. Pushed, it works now.
> > > > > > >> > >> >
> > > > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > > > > > >> > >> > sitnikov.vladimir@gmail.com
> > > > > > >> > >> > >:
> > > > > > >> > >> >
> > > > > > >> > >> > > > mvn clean test
> > > > > > >> > >> > >
> > > > > > >> > >> > > [ERROR] The goal you specified requires a project to
> > > > execute
> > > > > > but
> > > > > > >> > >> there is
> > > > > > >> > >> > > no POM in this directory
> > > > > > >> > >> > >
> > > > > > >> > >> > > Vladimir, please push missing files
> > > > > > >> > >> > >
> > > > > > >> > >> > > Vladimir
> > > > > > >> > >> > >
> > > > > > >> > >> >
> > > > > > >> > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>

Re: Problem with converters and possibly rule matching

Posted by Julian Hyde <jh...@apache.org>.
My definition of a physical property is one that can be changed by a converter. For example, the Ordering physical property can be changed by the Sort operator. And the Distribution physical property can be changed by the Exchange operator. 

Particular physical properties might be baked into the schema — e.g. the fact that table T is partitioned on X and sorted by Y — but doesn’t stop them from being physical properties; it just saves the effort of sorting/partitioning when you execute the query.

Suppose that we have a date column. It could be represented as a string, date, or integer. There is the same information content in each, and hence there are lossless conversions among those representations. So, I’d say that the representation of that column, even though baked into the schema, is a physical property too.

Julian


> On Nov 3, 2019, at 11:33 PM, Haisheng Yuan <h....@alibaba-inc.com> wrote:
> 
> Hi Vladimir,
> 
> This is still can be done through top-down request approach.
> 
> PhysicalFilter operator should request ANY distribution from child operator, unless there is outer reference in the filter condition, in which case, PhysicalFilter should request SINGLETON or BROADCAST distribution. So in your case, PhysicalFilter request ANY, its required distribution will be enforced on filter's output.
> 
> Regarding index usage, you should have a FIlterTableScan2IndexGet logical transformation rule, and a IndexGet2IndexScan physical implementation rule. Note that IndexGet is a logical operator and IndexScan is a physical operator, which are also used by SQL Server.
> 
> - Haisheng
> 
> ------------------------------------------------------------------
> 发件人:Vladimir Ozerov<pp...@gmail.com>
> 日 期:2019年11月01日 17:30:26
> 收件人:<de...@calcite.apache.org>
> 主 题:Re: Problem with converters and possibly rule matching
> 
> Hi Stamatis,
> 
> Thank you for your reply. I also thought that we may set the distribution
> trait during logical planning because it is known in advance. And the
> example I gave will work! :-) But unfortunately, it will work only because
> the tree is very simple, and the Project is adjacent to the Scan. This is
> how my reproducer will work in that case:
> 1) Root: enforce "SINGLETON" on Project
> 2) Project: check the logical Scan, infer already resolved distribution,
> then convert to [PhyiscalProject <- PhysicalScan]
> 3) Resolve Root enforcer, adding and Exchange if needed.
> 
> But this stops working as soon as a plan becomes more complex so that it is
> impossible to infer the distribution from the child immediately. E.g.:
> LogicalRoot [distribution=SINGLETON]
> -> LogicalProject // We are here new and cannot produce the physical project
> -> LogicalFilter[distribution=?]
> -> LogicalScan[distribution=REPLICATED]
> 
> This is where your suggestion with cascading enforcement may kick in. But
> now consider that instead of having REPLICATED distribution, which
> satisfies SINGLETON, we have a PARTITIONED distribution, It doesn't satisfy
> SINGLETON, so we have to insert the exchange.
> 
> Before:
> PhysicalRoot [SINGLETON]
> -> PhysicalProject [SINGLETON]
> -> PhysicalFilter [SINGLETON]
> -> LogicalScan [PARTITIONED] // SINGLETON is enforced here
> 
> After:
> PhysicalRoot [SINGLETON]
> -> PhysicalProject [SINGLETON]
> -> PhysicalFilter [SINGLETON]
> -> SingletonExchange [SINGLETON]
> -> PhysicalScan [PARTITIONED]
> 
> But unfortunately, this plan is not optimal. Since we perform Project and
> Filter after the exchange, we now have to reshuffle the whole table. The
> optimal plan would be to pushdown Project and Filter past exchange:
> PhysicalRoot [SINGLETON]
> -> SingletonExchange [SINGLETON]
> -> PhysicalProject [PARTITIONED]
> -> PhysicalFilter [PARTITIONED]
> -> PhysicalScan [PARTITIONED]
> 
> But in order to achieve that, now I need to introduce new rules which will
> attempt to transpose dozens of different operators past exchange, so most
> likely the planning will never finish :-)
> 
> This is why it seems that the bottom-up approach seems to be more natural
> for such cases. Another example is index support. A table may have a number
> of access methods in addition to plain scan, and every such method may
> produce a physical scan with different collations. It is not practical to
> try to enforce something from the top. Instead, we'd better produce
> alternatives from the bottom.
> 
> Basically, Drill tries to mix two approaches. First, it tries to derive
> traits from the child. If it fails and no transformations were created, it
> just stops trait propagation. As a result, an exchange is created on top of
> the operator where we stopped, which again leads to not optimal plans. You
> may observe this in action if you run my reproducer. "testPass" produces an
> optimal plan with help of abstract converters. "testDrill" produces not
> optimal plan with the unnecessary exchange.
> 
> All in all, I cannot get how we can employ distribution metadata and index
> metadata for optimal planning without using a pure bottom-up approach.
> 
> Regards,
> Vladimir.
> 
> пт, 1 нояб. 2019 г. в 11:42, Stamatis Zampetakis <za...@gmail.com>:
> 
>> The discussion is interesting thanks for the nice examples.
>> I am replying in this thread since it has all the examples and the history
>> of the conversation.
>> 
>> *Part I: Distribution physical or logical property*
>> 
>> The Volcano paper writes the following regarding properties:
>> 
>> "Logical properties can be derived from the logical algebra expression and
>> include schema, expected size, etc., while physical properties depend on
>> algorithms, e.g., sort order, partitioning, etc."
>> 
>> We could say that partitioning is not only a property which depends on an
>> algorithm but a property which depends on the schema. When you have a table
>> you know in advance that the particular table is replicated.
>> 
>> Basically, this means that the starting "logical" plan in your case could
>> be the following:
>> RootLogicalRel[convention=LOGICAL, distribution=ANY]
>> -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
>> -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
>> 
>> When you are about to create your physical projection it is not necessary
>> to create three equivalent paths but the most promising by asking the input
>> what can it offer as traits. Again this partially overlaps with the
>> discussion about "On demand traitset" [1] where as I wrote there is already
>> a mechanism to derive traits from children (collation, distribution, etc.).
>> 
>> Note that I am not saying that there is nothing more to be done to improve
>> the Calcite optimizer; I am just trying to provide a viable alternative
>> given the current situation of the project.
>> 
>> *Part II: Example with the original Volcano optimizer*
>> 
>> Now let me try to show how the original (from the paper) Volcano optimizer
>> could handle your query.
>> 
>> The optimization starts and we request two properties convention and
>> distribution.
>> 
>> RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
>> -> ProjectLogicalRel
>> -> MapScanLogicalRel
>> 
>> Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but cannot
>> satisfy the SINGLETON property so it propagates the demand
>> 
>> RootPhysicalRel
>> -> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>> -> MapScanLogicalRel
>> 
>> Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the algorithm but
>> cannot satisfy the SINGLETON property so it propagates the demand
>> 
>> RootPhysicalRel
>> -> ProjectPhysicalRel
>> -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>> 
>> Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the algorithm and
>> given that the table is replicated it satisfies the distribution property.
>> 
>> RootPhysicalRel
>> -> ProjectPhysicalRel
>> -> MapScanPhysicalRel
>> 
>> So we end up with a complete plan that satisfies all properties and does
>> not have redundant exchange operators.
>> Moreover we didn't require all possible distributions when we applied Rule
>> 2 since we just want to satisfy the SINGLETON property requested by the
>> parent.
>> The example above does not demonstrate the additional optimization paths
>> which would be apply an enforcer (an Exchange operator to satisfy the
>> SINGLETON property) at a higher level than the scan.
>> 
>> Would this be an acceptable approach? Does it still suffer from a big
>> search space?
>> 
>> To make the analog with the actual Volcano optimizer in Calcite the thing
>> that may be missing is to be able to know what trait was requested by the
>> parent during the application of Rule 2.
>> 
>> Best,
>> Stamatis
>> 
>> [1]
>> 
>> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
>> 
>> On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <pp...@gmail.com>
>> wrote:
>> 
>>> Hi Stamatis,
>>> 
>>> The problem that the presented reproducer is a very simplified version of
>>> what is actually needed. In reality, there are several distribution
>> types -
>>> PARTITIONED, REPLICATED, SINGLETON. To make things worse, PARTITIONED may
>>> or may not have concrete distribution fields. In theory, I can create one
>>> transformation per distribution type, but that would increase plan space
>>> significantly. In my sample "Root <- Project <- Scan" plan, there is no
>>> room for optimization at all, we only need to convert the nodes and
>>> propagate traits. But if I follow the proposed approach, the planner
>> would
>>> create three equivalent paths. For complex queries, this may increase
>>> optimization time significantly.
>>> 
>>> What I need instead is to gradually convert and optimize nodes
>> *bottom-up*,
>>> instead of top-bottom. That is, create a physical scan first, then
>> create a
>>> physical project on top of it, etc. But at the same time, some rules
>> still
>>> require the top-bottom approach. So essentially I need the optimizer to
>> do
>>> both. Abstract converters help me establish bottom-up preparation but do
>>> this at the cost of considering too many trait pairs, and as a result,
>> also
>>> pollute the search space.
>>> 
>>> To the contrast, precise command "I transformed the node A to node B,
>>> please re-trigger the rules for A's parents" would allow us to re-trigger
>>> only required rules, without adding more nodes.
>>> 
>>> Does it make sense?
>>> 
>>> Regards,
>>> Vladimir.
>>> 
>>> ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <za...@gmail.com>:
>>> 
>>>> I admit that I didn't thoroughly read the exchanges so far but forcing
>>> the
>>>> rules trigger does not seem the right approach.
>>>> 
>>>> Other than that I would like to backtrack a bit to point 4.3 raised by
>>>> Vladimir.
>>>> 
>>>> "ProjectPhysicalRule [6] - transforms logical project to physical
>>>> project *ONLY* if there is an underlying physical input with REPLICATED
>>> or
>>>> SINGLETON distribution"
>>>> 
>>>> The rule could be modified to do the following two transformations:
>>>> 1. Create a physical project and require the input to be REPLICATED.
>>>> 2. Create a physical project and require
>>>> the input to be SINGLETON.
>>>> 
>>>> I would assume that afterwards when your scan rule fires it should go
>> to
>>>> the appropriate subset and give you back the desired plan. Is there a
>>>> problem with this approach?
>>>> 
>>>> Best,
>>>> Stamatis
>>>> 
>>>> On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <gv...@gmail.com>
>>>> wrote:
>>>> 
>>>>> Unfortunately it requires package-private API usage.
>>>>> 
>>>>> Best solution there if Calcite provide it for end users.
>>>>> 
>>>>> ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
>>>>> 
>>>>>> “e pension” == “expand conversion” :)
>>>>>> 
>>>>>> ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <pp...@gmail.com>:
>>>>>> 
>>>>>>> Yes, that may work. Even if e pension rule is used, for the most
>>>> cases
>>>>> it
>>>>>>> will not trigger any real conversions, since we are moving from
>>>>> abstract
>>>>>>> convention to physical, and created converters will have the
>>> opposite
>>>>>> trait
>>>>>>> direction (from physical to abstract).
>>>>>>> 
>>>>>>> But again - ideally we only need to re-trigger the rules for a
>>>> specific
>>>>>>> node, no more than that. So API support like
>>>>>>> “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
>>>>>>> 
>>>>>>> What do you think?
>>>>>>> 
>>>>>>> ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
>>> gvvinblade@gmail.com
>>>>> :
>>>>>>> 
>>>>>>>> I considered manual rules calling too, for now we use abstract
>>>>>> converters
>>>>>>>> +
>>>>>>>> ExpandConversionRule for exchanges producing.
>>>>>>>> 
>>>>>>>> You may create such converters manually (checking appropriate
>>>> subset)
>>>>>> this
>>>>>>>> case you may reduce created converters count, also, a converter
>>> is a
>>>>>> quite
>>>>>>>> special node, that does almost nothing (without corresponding
>>> rule)
>>>> it
>>>>>> may
>>>>>>>> be used just as a rule trigger.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Igor
>>>>>>>> 
>>>>>>>> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <ppozerov@gmail.com
>>> :
>>>>>>>> 
>>>>>>>>> One funny hack which helped me is manual registration of a
>> fake
>>>>>> RelNode
>>>>>>>>> with desired traits through VolcanoPlanner.register() method.
>>> But
>>>>>> again,
>>>>>>>>> this leads to trashing. What could really help is a call to
>>>>>>>>> VolcanoPlanner.fireRules() with desired rel. But this doesn't
>>> work
>>>>> out
>>>>>>>> of
>>>>>>>>> the box since some internals of the rule queue needs to be
>>>> adjusted.
>>>>>>>>> 
>>>>>>>>> What does the community think about adding a method which will
>>>>> re-add
>>>>>>>> rules
>>>>>>>>> applicable to the specific RelNode to the rule queue?
>>>>>>>>> 
>>>>>>>>> ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
>>> ppozerov@gmail.com
>>>>> :
>>>>>>>>> 
>>>>>>>>>> Hi Igor,
>>>>>>>>>> 
>>>>>>>>>> Yes, I came to the same conclusion, thank you. This is how
>> it
>>>>>>>> basically
>>>>>>>>>> happens when converters are disabled:
>>>>>>>>>> 1) We start with initial tree: [LogicalProject] <-
>>> [LogicalScan]
>>>>>>>>>> 2) Then we convert LogicalScan to PhysicalScan, so it is
>> added
>>>> to
>>>>>> the
>>>>>>>>>> set: [LogicalProject] <- [LogicalScan, PhysicalScan]
>>>>>>>>>> 3) Finally, when it is time to fire a rule for PhysicalScan,
>>> we
>>>>> try
>>>>>> to
>>>>>>>>> get
>>>>>>>>>> parents of that scan set with traits of the PhysicalScan.
>>> Since
>>>>>> there
>>>>>>>> are
>>>>>>>>>> no such parents (we skipped it intentionally), the rule is
>> not
>>>>>> queued.
>>>>>>>>>> 
>>>>>>>>>> But when converters are enabled, a converter rel is created:
>>>>>>>>> [LogicalProject]
>>>>>>>>>> <- [LogicalScan, PhysicalScan,
>>> ConverterFromPhysicalToLogical].
>>>> No
>>>>>>>> rules
>>>>>>>>>> are fired for PhysicalScan again, but they are fired for
>>>> converter
>>>>>>>> since
>>>>>>>>>> it has the necessary LOGICAL trait.
>>>>>>>>>> 
>>>>>>>>>> It makes sense, that converters essentially allow forcing
>> rule
>>>>>>>> invocation
>>>>>>>>>> on parents, even if the child was created with different
>>> traits.
>>>>> But
>>>>>>>> it
>>>>>>>>>> seems that this mechanism may be too heavy for complex
>> queries
>>>>>>>> because it
>>>>>>>>>> literally creates hundreds of new converter rels and
>> triggers
>>>>> rules
>>>>>>>> over
>>>>>>>>>> and over again.
>>>>>>>>>> 
>>>>>>>>>> We need some fine-grained alternative. Basically, what would
>>> be
>>>>>> enough
>>>>>>>>> for
>>>>>>>>>> me is to let the planner know somehow: "I created that rel,
>>> and
>>>> I
>>>>>> want
>>>>>>>>> you
>>>>>>>>>> to execute parent rules not only using its trait but also on
>>>> this
>>>>>> and
>>>>>>>>> those
>>>>>>>>>> traits."
>>>>>>>>>> Is there any API in Calcite which allows doing this without
>>>>>> creating a
>>>>>>>>> new
>>>>>>>>>> rel node?
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Vladimir.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
>>>>> gvvinblade@gmail.com
>>>>>>> :
>>>>>>>>>> 
>>>>>>>>>>> Vladimir,
>>>>>>>>>>> 
>>>>>>>>>>> Probably it'll help you:
>>>>>>>>>>> 
>>>>>>>>>>> Seems the cause of issue in RelSubset.getParentRels() The
>>>> check
>>>>>> used
>>>>>>>>> when
>>>>>>>>>>> the planner schedules newly matched rules after successful
>>>>>>>>> transformation
>>>>>>>>>>> (see VolcanoRuleCall.matchRecurse), it prevents the parent
>>> rule
>>>>> be
>>>>>>>>> applied
>>>>>>>>>>> once again (here your logical project with an input having
>>> ANY
>>>>>>>>>>> distribution
>>>>>>>>>>> doesn't satisfy a transformed input traits).
>>>>>>>>>>> 
>>>>>>>>>>> In our case we use another workaround, so there are also
>> much
>>>>> more
>>>>>>>>>>> transformations than we wanted, so the desired rule is
>>>> triggered.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
>>> ppozerov@gmail.com
>>>>> :
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Vladimir,
>>>>>>>>>>>> 
>>>>>>>>>>>> I am sorry. Pushed, it works now.
>>>>>>>>>>>> 
>>>>>>>>>>>> вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
>>>>>>>>>>>> sitnikov.vladimir@gmail.com
>>>>>>>>>>>>> :
>>>>>>>>>>>> 
>>>>>>>>>>>>>> mvn clean test
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [ERROR] The goal you specified requires a project to
>>>> execute
>>>>>> but
>>>>>>>>>>> there is
>>>>>>>>>>>>> no POM in this directory
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Vladimir, please push missing files
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Vladimir
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


Re: Re: Problem with converters and possibly rule matching

Posted by Haisheng Yuan <h....@alibaba-inc.com>.
Hi Vladimir,

This is still can be done through top-down request approach.

PhysicalFilter operator should request ANY distribution from child operator, unless there is outer reference in the filter condition, in which case, PhysicalFilter should request SINGLETON or BROADCAST distribution. So in your case, PhysicalFilter request ANY, its required distribution will be enforced on filter's output.

Regarding index usage, you should have a FIlterTableScan2IndexGet logical transformation rule, and a IndexGet2IndexScan physical implementation rule. Note that IndexGet is a logical operator and IndexScan is a physical operator, which are also used by SQL Server.

- Haisheng

------------------------------------------------------------------
发件人:Vladimir Ozerov<pp...@gmail.com>
日 期:2019年11月01日 17:30:26
收件人:<de...@calcite.apache.org>
主 题:Re: Problem with converters and possibly rule matching

Hi Stamatis,

Thank you for your reply. I also thought that we may set the distribution
trait during logical planning because it is known in advance. And the
example I gave will work! :-) But unfortunately, it will work only because
the tree is very simple, and the Project is adjacent to the Scan. This is
how my reproducer will work in that case:
1) Root: enforce "SINGLETON" on Project
2) Project: check the logical Scan, infer already resolved distribution,
then convert to [PhyiscalProject <- PhysicalScan]
3) Resolve Root enforcer, adding and Exchange if needed.

But this stops working as soon as a plan becomes more complex so that it is
impossible to infer the distribution from the child immediately. E.g.:
LogicalRoot [distribution=SINGLETON]
-> LogicalProject // We are here new and cannot produce the physical project
 -> LogicalFilter[distribution=?]
 -> LogicalScan[distribution=REPLICATED]

This is where your suggestion with cascading enforcement may kick in. But
now consider that instead of having REPLICATED distribution, which
satisfies SINGLETON, we have a PARTITIONED distribution, It doesn't satisfy
SINGLETON, so we have to insert the exchange.

Before:
PhysicalRoot [SINGLETON]
-> PhysicalProject [SINGLETON]
 -> PhysicalFilter [SINGLETON]
 -> LogicalScan [PARTITIONED] // SINGLETON is enforced here

After:
PhysicalRoot [SINGLETON]
-> PhysicalProject [SINGLETON]
 -> PhysicalFilter [SINGLETON]
 -> SingletonExchange [SINGLETON]
 -> PhysicalScan [PARTITIONED]

But unfortunately, this plan is not optimal. Since we perform Project and
Filter after the exchange, we now have to reshuffle the whole table. The
optimal plan would be to pushdown Project and Filter past exchange:
PhysicalRoot [SINGLETON]
-> SingletonExchange [SINGLETON]
 -> PhysicalProject [PARTITIONED]
 -> PhysicalFilter [PARTITIONED]
 -> PhysicalScan [PARTITIONED]

But in order to achieve that, now I need to introduce new rules which will
attempt to transpose dozens of different operators past exchange, so most
likely the planning will never finish :-)

This is why it seems that the bottom-up approach seems to be more natural
for such cases. Another example is index support. A table may have a number
of access methods in addition to plain scan, and every such method may
produce a physical scan with different collations. It is not practical to
try to enforce something from the top. Instead, we'd better produce
alternatives from the bottom.

Basically, Drill tries to mix two approaches. First, it tries to derive
traits from the child. If it fails and no transformations were created, it
just stops trait propagation. As a result, an exchange is created on top of
the operator where we stopped, which again leads to not optimal plans. You
may observe this in action if you run my reproducer. "testPass" produces an
optimal plan with help of abstract converters. "testDrill" produces not
optimal plan with the unnecessary exchange.

All in all, I cannot get how we can employ distribution metadata and index
metadata for optimal planning without using a pure bottom-up approach.

Regards,
Vladimir.

пт, 1 нояб. 2019 г. в 11:42, Stamatis Zampetakis <za...@gmail.com>:

> The discussion is interesting thanks for the nice examples.
> I am replying in this thread since it has all the examples and the history
> of the conversation.
>
> *Part I: Distribution physical or logical property*
>
> The Volcano paper writes the following regarding properties:
>
> "Logical properties can be derived from the logical algebra expression and
> include schema, expected size, etc., while physical properties depend on
> algorithms, e.g., sort order, partitioning, etc."
>
> We could say that partitioning is not only a property which depends on an
> algorithm but a property which depends on the schema. When you have a table
> you know in advance that the particular table is replicated.
>
> Basically, this means that the starting "logical" plan in your case could
> be the following:
> RootLogicalRel[convention=LOGICAL, distribution=ANY]
> -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
> -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
>
> When you are about to create your physical projection it is not necessary
> to create three equivalent paths but the most promising by asking the input
> what can it offer as traits. Again this partially overlaps with the
> discussion about "On demand traitset" [1] where as I wrote there is already
> a mechanism to derive traits from children (collation, distribution, etc.).
>
> Note that I am not saying that there is nothing more to be done to improve
> the Calcite optimizer; I am just trying to provide a viable alternative
> given the current situation of the project.
>
> *Part II: Example with the original Volcano optimizer*
>
> Now let me try to show how the original (from the paper) Volcano optimizer
> could handle your query.
>
> The optimization starts and we request two properties convention and
> distribution.
>
> RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
> -> ProjectLogicalRel
> -> MapScanLogicalRel
>
> Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but cannot
> satisfy the SINGLETON property so it propagates the demand
>
> RootPhysicalRel
> -> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
> -> MapScanLogicalRel
>
> Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the algorithm but
> cannot satisfy the SINGLETON property so it propagates the demand
>
> RootPhysicalRel
> -> ProjectPhysicalRel
> -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>
> Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the algorithm and
> given that the table is replicated it satisfies the distribution property.
>
> RootPhysicalRel
> -> ProjectPhysicalRel
> -> MapScanPhysicalRel
>
> So we end up with a complete plan that satisfies all properties and does
> not have redundant exchange operators.
> Moreover we didn't require all possible distributions when we applied Rule
> 2 since we just want to satisfy the SINGLETON property requested by the
> parent.
> The example above does not demonstrate the additional optimization paths
> which would be apply an enforcer (an Exchange operator to satisfy the
> SINGLETON property) at a higher level than the scan.
>
> Would this be an acceptable approach? Does it still suffer from a big
> search space?
>
> To make the analog with the actual Volcano optimizer in Calcite the thing
> that may be missing is to be able to know what trait was requested by the
> parent during the application of Rule 2.
>
> Best,
> Stamatis
>
> [1]
>
> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
>
> On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <pp...@gmail.com>
> wrote:
>
> > Hi Stamatis,
> >
> > The problem that the presented reproducer is a very simplified version of
> > what is actually needed. In reality, there are several distribution
> types -
> > PARTITIONED, REPLICATED, SINGLETON. To make things worse, PARTITIONED may
> > or may not have concrete distribution fields. In theory, I can create one
> > transformation per distribution type, but that would increase plan space
> > significantly. In my sample "Root <- Project <- Scan" plan, there is no
> > room for optimization at all, we only need to convert the nodes and
> > propagate traits. But if I follow the proposed approach, the planner
> would
> > create three equivalent paths. For complex queries, this may increase
> > optimization time significantly.
> >
> > What I need instead is to gradually convert and optimize nodes
> *bottom-up*,
> > instead of top-bottom. That is, create a physical scan first, then
> create a
> > physical project on top of it, etc. But at the same time, some rules
> still
> > require the top-bottom approach. So essentially I need the optimizer to
> do
> > both. Abstract converters help me establish bottom-up preparation but do
> > this at the cost of considering too many trait pairs, and as a result,
> also
> > pollute the search space.
> >
> > To the contrast, precise command "I transformed the node A to node B,
> > please re-trigger the rules for A's parents" would allow us to re-trigger
> > only required rules, without adding more nodes.
> >
> > Does it make sense?
> >
> > Regards,
> > Vladimir.
> >
> > ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <za...@gmail.com>:
> >
> > > I admit that I didn't thoroughly read the exchanges so far but forcing
> > the
> > > rules trigger does not seem the right approach.
> > >
> > > Other than that I would like to backtrack a bit to point 4.3 raised by
> > > Vladimir.
> > >
> > > "ProjectPhysicalRule [6] - transforms logical project to physical
> > > project *ONLY* if there is an underlying physical input with REPLICATED
> > or
> > > SINGLETON distribution"
> > >
> > > The rule could be modified to do the following two transformations:
> > > 1. Create a physical project and require the input to be REPLICATED.
> > > 2. Create a physical project and require
> > > the input to be SINGLETON.
> > >
> > > I would assume that afterwards when your scan rule fires it should go
> to
> > > the appropriate subset and give you back the desired plan. Is there a
> > > problem with this approach?
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <gv...@gmail.com>
> > > wrote:
> > >
> > > > Unfortunately it requires package-private API usage.
> > > >
> > > > Best solution there if Calcite provide it for end users.
> > > >
> > > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
> > > >
> > > > > “e pension” == “expand conversion” :)
> > > > >
> > > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <pp...@gmail.com>:
> > > > >
> > > > > > Yes, that may work. Even if e pension rule is used, for the most
> > > cases
> > > > it
> > > > > > will not trigger any real conversions, since we are moving from
> > > > abstract
> > > > > > convention to physical, and created converters will have the
> > opposite
> > > > > trait
> > > > > > direction (from physical to abstract).
> > > > > >
> > > > > > But again - ideally we only need to re-trigger the rules for a
> > > specific
> > > > > > node, no more than that. So API support like
> > > > > > “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
> > gvvinblade@gmail.com
> > > >:
> > > > > >
> > > > > >> I considered manual rules calling too, for now we use abstract
> > > > > converters
> > > > > >> +
> > > > > >> ExpandConversionRule for exchanges producing.
> > > > > >>
> > > > > >> You may create such converters manually (checking appropriate
> > > subset)
> > > > > this
> > > > > >> case you may reduce created converters count, also, a converter
> > is a
> > > > > quite
> > > > > >> special node, that does almost nothing (without corresponding
> > rule)
> > > it
> > > > > may
> > > > > >> be used just as a rule trigger.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Igor
> > > > > >>
> > > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <ppozerov@gmail.com
> >:
> > > > > >>
> > > > > >> > One funny hack which helped me is manual registration of a
> fake
> > > > > RelNode
> > > > > >> > with desired traits through VolcanoPlanner.register() method.
> > But
> > > > > again,
> > > > > >> > this leads to trashing. What could really help is a call to
> > > > > >> > VolcanoPlanner.fireRules() with desired rel. But this doesn't
> > work
> > > > out
> > > > > >> of
> > > > > >> > the box since some internals of the rule queue needs to be
> > > adjusted.
> > > > > >> >
> > > > > >> > What does the community think about adding a method which will
> > > > re-add
> > > > > >> rules
> > > > > >> > applicable to the specific RelNode to the rule queue?
> > > > > >> >
> > > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
> > ppozerov@gmail.com
> > > >:
> > > > > >> >
> > > > > >> > > Hi Igor,
> > > > > >> > >
> > > > > >> > > Yes, I came to the same conclusion, thank you. This is how
> it
> > > > > >> basically
> > > > > >> > > happens when converters are disabled:
> > > > > >> > > 1) We start with initial tree: [LogicalProject] <-
> > [LogicalScan]
> > > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is
> added
> > > to
> > > > > the
> > > > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > > > > >> > > 3) Finally, when it is time to fire a rule for PhysicalScan,
> > we
> > > > try
> > > > > to
> > > > > >> > get
> > > > > >> > > parents of that scan set with traits of the PhysicalScan.
> > Since
> > > > > there
> > > > > >> are
> > > > > >> > > no such parents (we skipped it intentionally), the rule is
> not
> > > > > queued.
> > > > > >> > >
> > > > > >> > > But when converters are enabled, a converter rel is created:
> > > > > >> > [LogicalProject]
> > > > > >> > > <- [LogicalScan, PhysicalScan,
> > ConverterFromPhysicalToLogical].
> > > No
> > > > > >> rules
> > > > > >> > > are fired for PhysicalScan again, but they are fired for
> > > converter
> > > > > >> since
> > > > > >> > > it has the necessary LOGICAL trait.
> > > > > >> > >
> > > > > >> > > It makes sense, that converters essentially allow forcing
> rule
> > > > > >> invocation
> > > > > >> > > on parents, even if the child was created with different
> > traits.
> > > > But
> > > > > >> it
> > > > > >> > > seems that this mechanism may be too heavy for complex
> queries
> > > > > >> because it
> > > > > >> > > literally creates hundreds of new converter rels and
> triggers
> > > > rules
> > > > > >> over
> > > > > >> > > and over again.
> > > > > >> > >
> > > > > >> > > We need some fine-grained alternative. Basically, what would
> > be
> > > > > enough
> > > > > >> > for
> > > > > >> > > me is to let the planner know somehow: "I created that rel,
> > and
> > > I
> > > > > want
> > > > > >> > you
> > > > > >> > > to execute parent rules not only using its trait but also on
> > > this
> > > > > and
> > > > > >> > those
> > > > > >> > > traits."
> > > > > >> > > Is there any API in Calcite which allows doing this without
> > > > > creating a
> > > > > >> > new
> > > > > >> > > rel node?
> > > > > >> > >
> > > > > >> > > Regards,
> > > > > >> > > Vladimir.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> > > > gvvinblade@gmail.com
> > > > > >:
> > > > > >> > >
> > > > > >> > >> Vladimir,
> > > > > >> > >>
> > > > > >> > >> Probably it'll help you:
> > > > > >> > >>
> > > > > >> > >> Seems the cause of issue in RelSubset.getParentRels() The
> > > check
> > > > > used
> > > > > >> > when
> > > > > >> > >> the planner schedules newly matched rules after successful
> > > > > >> > transformation
> > > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the parent
> > rule
> > > > be
> > > > > >> > applied
> > > > > >> > >> once again (here your logical project with an input having
> > ANY
> > > > > >> > >> distribution
> > > > > >> > >> doesn't satisfy a transformed input traits).
> > > > > >> > >>
> > > > > >> > >> In our case we use another workaround, so there are also
> much
> > > > more
> > > > > >> > >> transformations than we wanted, so the desired rule is
> > > triggered.
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
> > ppozerov@gmail.com
> > > >:
> > > > > >> > >>
> > > > > >> > >> > Hi Vladimir,
> > > > > >> > >> >
> > > > > >> > >> > I am sorry. Pushed, it works now.
> > > > > >> > >> >
> > > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > > > > >> > >> > sitnikov.vladimir@gmail.com
> > > > > >> > >> > >:
> > > > > >> > >> >
> > > > > >> > >> > > > mvn clean test
> > > > > >> > >> > >
> > > > > >> > >> > > [ERROR] The goal you specified requires a project to
> > > execute
> > > > > but
> > > > > >> > >> there is
> > > > > >> > >> > > no POM in this directory
> > > > > >> > >> > >
> > > > > >> > >> > > Vladimir, please push missing files
> > > > > >> > >> > >
> > > > > >> > >> > > Vladimir
> > > > > >> > >> > >
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>


Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
Hi Stamatis,

Thank you for your reply. I also thought that we may set the distribution
trait during logical planning because it is known in advance. And the
example I gave will work! :-) But unfortunately, it will work only because
the tree is very simple, and the Project is adjacent to the Scan. This is
how my reproducer will work in that case:
1) Root: enforce "SINGLETON" on Project
2) Project: check the logical Scan, infer already resolved distribution,
then convert to [PhyiscalProject <- PhysicalScan]
3) Resolve Root enforcer, adding and Exchange if needed.

But this stops working as soon as a plan becomes more complex so that it is
impossible to infer the distribution from the child immediately. E.g.:
LogicalRoot [distribution=SINGLETON]
-> LogicalProject // We are here new and cannot produce the physical project
 -> LogicalFilter[distribution=?]
  -> LogicalScan[distribution=REPLICATED]

This is where your suggestion with cascading enforcement may kick in. But
now consider that instead of having REPLICATED distribution, which
satisfies SINGLETON, we have a PARTITIONED distribution, It doesn't satisfy
SINGLETON, so we have to insert the exchange.

Before:
PhysicalRoot       [SINGLETON]
-> PhysicalProject [SINGLETON]
 -> PhysicalFilter [SINGLETON]
  -> LogicalScan   [PARTITIONED] // SINGLETON is enforced here

After:
PhysicalRoot           [SINGLETON]
-> PhysicalProject     [SINGLETON]
 -> PhysicalFilter     [SINGLETON]
  -> SingletonExchange [SINGLETON]
   -> PhysicalScan     [PARTITIONED]

But unfortunately, this plan is not optimal. Since we perform Project and
Filter after the exchange, we now have to reshuffle the whole table. The
optimal plan would be to pushdown Project and Filter past exchange:
PhysicalRoot         [SINGLETON]
-> SingletonExchange [SINGLETON]
 -> PhysicalProject  [PARTITIONED]
  -> PhysicalFilter  [PARTITIONED]
   -> PhysicalScan   [PARTITIONED]

But in order to achieve that, now I need to introduce new rules which will
attempt to transpose dozens of different operators past exchange, so most
likely the planning will never finish :-)

This is why it seems that the bottom-up approach seems to be more natural
for such cases. Another example is index support. A table may have a number
of access methods in addition to plain scan, and every such method may
produce a physical scan with different collations. It is not practical to
try to enforce something from the top. Instead, we'd better produce
alternatives from the bottom.

Basically, Drill tries to mix two approaches. First, it tries to derive
traits from the child. If it fails and no transformations were created, it
just stops trait propagation. As a result, an exchange is created on top of
the operator where we stopped, which again leads to not optimal plans. You
may observe this in action if you run my reproducer. "testPass" produces an
optimal plan with help of abstract converters. "testDrill" produces not
optimal plan with the unnecessary exchange.

All in all, I cannot get how we can employ distribution metadata and index
metadata for optimal planning without using a pure bottom-up approach.

Regards,
Vladimir.

пт, 1 нояб. 2019 г. в 11:42, Stamatis Zampetakis <za...@gmail.com>:

> The discussion is interesting thanks for the nice examples.
> I am replying in this thread since it has all the examples and the history
> of the conversation.
>
> *Part I: Distribution physical or logical property*
>
> The Volcano paper writes the following regarding properties:
>
> "Logical properties can be derived from the logical algebra expression and
> include schema, expected size, etc., while physical properties depend on
> algorithms, e.g., sort order, partitioning, etc."
>
> We could say that partitioning is not only a property which depends on an
> algorithm but a property which depends on the schema. When you have a table
> you know in advance that the particular table is replicated.
>
> Basically, this means that the starting "logical" plan in your case could
> be the following:
> RootLogicalRel[convention=LOGICAL, distribution=ANY]
> -> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
>   -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]
>
> When you are about to create your physical projection it is not necessary
> to create three equivalent paths but the most promising by asking the input
> what can it offer as traits. Again this partially overlaps with the
> discussion about "On demand traitset" [1] where as I wrote there is already
> a mechanism to derive traits from children (collation, distribution, etc.).
>
> Note that I am not saying that there is nothing more to be done to improve
> the Calcite optimizer; I am just trying to provide a viable alternative
> given the current situation of the project.
>
> *Part II: Example with the original Volcano optimizer*
>
> Now let me try to show how the original (from the paper) Volcano optimizer
> could handle your query.
>
> The optimization starts and we request two properties convention and
> distribution.
>
> RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
> -> ProjectLogicalRel
>   -> MapScanLogicalRel
>
> Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but cannot
> satisfy the SINGLETON property so it propagates the demand
>
> RootPhysicalRel
> -> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>   -> MapScanLogicalRel
>
> Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the algorithm but
> cannot satisfy the SINGLETON property so it propagates the demand
>
> RootPhysicalRel
> -> ProjectPhysicalRel
>   -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
>
> Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the algorithm and
> given that the table is replicated it satisfies the distribution property.
>
> RootPhysicalRel
> -> ProjectPhysicalRel
>   -> MapScanPhysicalRel
>
> So we end up with a complete plan that satisfies all properties and does
> not have redundant exchange operators.
> Moreover we didn't require all possible distributions when we applied Rule
> 2 since we just want to satisfy the SINGLETON property requested by the
> parent.
> The example above does not demonstrate the additional optimization paths
> which would be apply an enforcer (an Exchange operator to satisfy the
> SINGLETON property) at a higher level than the scan.
>
> Would this be an acceptable approach? Does it still suffer from a big
> search space?
>
> To make the analog with the actual Volcano optimizer in Calcite the thing
> that may be missing is to be able to know what trait was requested by the
> parent during the application of Rule 2.
>
> Best,
> Stamatis
>
> [1]
>
> https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E
>
> On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <pp...@gmail.com>
> wrote:
>
> > Hi Stamatis,
> >
> > The problem that the presented reproducer is a very simplified version of
> > what is actually needed. In reality, there are several distribution
> types -
> > PARTITIONED, REPLICATED, SINGLETON. To make things worse, PARTITIONED may
> > or may not have concrete distribution fields. In theory, I can create one
> > transformation per distribution type, but that would increase plan space
> > significantly. In my sample "Root <- Project <- Scan" plan, there is no
> > room for optimization at all, we only need to convert the nodes and
> > propagate traits. But if I follow the proposed approach, the planner
> would
> > create three equivalent paths. For complex queries, this may increase
> > optimization time significantly.
> >
> > What I need instead is to gradually convert and optimize nodes
> *bottom-up*,
> > instead of top-bottom. That is, create a physical scan first, then
> create a
> > physical project on top of it, etc. But at the same time, some rules
> still
> > require the top-bottom approach. So essentially I need the optimizer to
> do
> > both. Abstract converters help me establish bottom-up preparation but do
> > this at the cost of considering too many trait pairs, and as a result,
> also
> > pollute the search space.
> >
> > To the contrast, precise command "I transformed the node A to node B,
> > please re-trigger the rules for A's parents" would allow us to re-trigger
> > only required rules, without adding more nodes.
> >
> > Does it make sense?
> >
> > Regards,
> > Vladimir.
> >
> > ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <za...@gmail.com>:
> >
> > > I admit that I didn't thoroughly read the exchanges so far but forcing
> > the
> > > rules trigger does not seem the right approach.
> > >
> > > Other than that I would like to backtrack a bit to point 4.3 raised by
> > > Vladimir.
> > >
> > > "ProjectPhysicalRule [6] - transforms logical project to physical
> > > project *ONLY* if there is an underlying physical input with REPLICATED
> > or
> > > SINGLETON distribution"
> > >
> > > The rule could be modified to do the following two transformations:
> > > 1. Create a physical project and require the input to be REPLICATED.
> > > 2. Create a physical project and require
> > > the input to be SINGLETON.
> > >
> > > I would assume that afterwards when your scan rule fires it should go
> to
> > > the appropriate subset and give you back the desired plan. Is there a
> > > problem with this approach?
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <gv...@gmail.com>
> > > wrote:
> > >
> > > > Unfortunately it requires package-private API usage.
> > > >
> > > > Best solution there if Calcite provide it for end users.
> > > >
> > > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
> > > >
> > > > > “e pension” == “expand conversion” :)
> > > > >
> > > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <pp...@gmail.com>:
> > > > >
> > > > > > Yes, that may work. Even if e pension rule is used, for the most
> > > cases
> > > > it
> > > > > > will not trigger any real conversions, since we are moving from
> > > > abstract
> > > > > > convention to physical, and created converters will have the
> > opposite
> > > > > trait
> > > > > > direction (from physical to abstract).
> > > > > >
> > > > > > But again - ideally we only need to re-trigger the rules for a
> > > specific
> > > > > > node, no more than that. So API support like
> > > > > > “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
> > gvvinblade@gmail.com
> > > >:
> > > > > >
> > > > > >> I considered manual rules calling too, for now we use abstract
> > > > > converters
> > > > > >> +
> > > > > >> ExpandConversionRule for exchanges producing.
> > > > > >>
> > > > > >> You may create such converters manually (checking appropriate
> > > subset)
> > > > > this
> > > > > >> case you may reduce created converters count, also, a converter
> > is a
> > > > > quite
> > > > > >> special node, that does almost nothing (without corresponding
> > rule)
> > > it
> > > > > may
> > > > > >> be used just as a rule trigger.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Igor
> > > > > >>
> > > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <ppozerov@gmail.com
> >:
> > > > > >>
> > > > > >> > One funny hack which helped me is manual registration of a
> fake
> > > > > RelNode
> > > > > >> > with desired traits through VolcanoPlanner.register() method.
> > But
> > > > > again,
> > > > > >> > this leads to trashing. What could really help is a call to
> > > > > >> > VolcanoPlanner.fireRules() with desired rel. But this doesn't
> > work
> > > > out
> > > > > >> of
> > > > > >> > the box since some internals of the rule queue needs to be
> > > adjusted.
> > > > > >> >
> > > > > >> > What does the community think about adding a method which will
> > > > re-add
> > > > > >> rules
> > > > > >> > applicable to the specific RelNode to the rule queue?
> > > > > >> >
> > > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
> > ppozerov@gmail.com
> > > >:
> > > > > >> >
> > > > > >> > > Hi Igor,
> > > > > >> > >
> > > > > >> > > Yes, I came to the same conclusion, thank you. This is how
> it
> > > > > >> basically
> > > > > >> > > happens when converters are disabled:
> > > > > >> > > 1) We start with initial tree: [LogicalProject] <-
> > [LogicalScan]
> > > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is
> added
> > > to
> > > > > the
> > > > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > > > > >> > > 3) Finally, when it is time to fire a rule for PhysicalScan,
> > we
> > > > try
> > > > > to
> > > > > >> > get
> > > > > >> > > parents of that scan set with traits of the PhysicalScan.
> > Since
> > > > > there
> > > > > >> are
> > > > > >> > > no such parents (we skipped it intentionally), the rule is
> not
> > > > > queued.
> > > > > >> > >
> > > > > >> > > But when converters are enabled, a converter rel is created:
> > > > > >> > [LogicalProject]
> > > > > >> > > <- [LogicalScan, PhysicalScan,
> > ConverterFromPhysicalToLogical].
> > > No
> > > > > >> rules
> > > > > >> > > are fired for PhysicalScan again, but they are fired for
> > > converter
> > > > > >> since
> > > > > >> > > it has the necessary LOGICAL trait.
> > > > > >> > >
> > > > > >> > > It makes sense, that converters essentially allow forcing
> rule
> > > > > >> invocation
> > > > > >> > > on parents, even if the child was created with different
> > traits.
> > > > But
> > > > > >> it
> > > > > >> > > seems that this mechanism may be too heavy for complex
> queries
> > > > > >> because it
> > > > > >> > > literally creates hundreds of new converter rels and
> triggers
> > > > rules
> > > > > >> over
> > > > > >> > > and over again.
> > > > > >> > >
> > > > > >> > > We need some fine-grained alternative. Basically, what would
> > be
> > > > > enough
> > > > > >> > for
> > > > > >> > > me is to let the planner know somehow: "I created that rel,
> > and
> > > I
> > > > > want
> > > > > >> > you
> > > > > >> > > to execute parent rules not only using its trait but also on
> > > this
> > > > > and
> > > > > >> > those
> > > > > >> > > traits."
> > > > > >> > > Is there any API in Calcite which allows doing this without
> > > > > creating a
> > > > > >> > new
> > > > > >> > > rel node?
> > > > > >> > >
> > > > > >> > > Regards,
> > > > > >> > > Vladimir.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> > > > gvvinblade@gmail.com
> > > > > >:
> > > > > >> > >
> > > > > >> > >> Vladimir,
> > > > > >> > >>
> > > > > >> > >> Probably it'll help you:
> > > > > >> > >>
> > > > > >> > >> Seems the cause of issue in RelSubset.getParentRels()  The
> > > check
> > > > > used
> > > > > >> > when
> > > > > >> > >> the planner schedules newly matched rules after successful
> > > > > >> > transformation
> > > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the parent
> > rule
> > > > be
> > > > > >> > applied
> > > > > >> > >> once again (here your logical project with an input having
> > ANY
> > > > > >> > >> distribution
> > > > > >> > >> doesn't satisfy a transformed input traits).
> > > > > >> > >>
> > > > > >> > >> In our case we use another workaround, so there are also
> much
> > > > more
> > > > > >> > >> transformations than we wanted, so the desired rule is
> > > triggered.
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
> > ppozerov@gmail.com
> > > >:
> > > > > >> > >>
> > > > > >> > >> > Hi Vladimir,
> > > > > >> > >> >
> > > > > >> > >> > I am sorry. Pushed, it works now.
> > > > > >> > >> >
> > > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > > > > >> > >> > sitnikov.vladimir@gmail.com
> > > > > >> > >> > >:
> > > > > >> > >> >
> > > > > >> > >> > > > mvn clean test
> > > > > >> > >> > >
> > > > > >> > >> > > [ERROR] The goal you specified requires a project to
> > > execute
> > > > > but
> > > > > >> > >> there is
> > > > > >> > >> > > no POM in this directory
> > > > > >> > >> > >
> > > > > >> > >> > > Vladimir, please push missing files
> > > > > >> > >> > >
> > > > > >> > >> > > Vladimir
> > > > > >> > >> > >
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Problem with converters and possibly rule matching

Posted by Stamatis Zampetakis <za...@gmail.com>.
The discussion is interesting thanks for the nice examples.
I am replying in this thread since it has all the examples and the history
of the conversation.

*Part I: Distribution physical or logical property*

The Volcano paper writes the following regarding properties:

"Logical properties can be derived from the logical algebra expression and
include schema, expected size, etc., while physical properties depend on
algorithms, e.g., sort order, partitioning, etc."

We could say that partitioning is not only a property which depends on an
algorithm but a property which depends on the schema. When you have a table
you know in advance that the particular table is replicated.

Basically, this means that the starting "logical" plan in your case could
be the following:
RootLogicalRel[convention=LOGICAL, distribution=ANY]
-> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
  -> MapScanLogicalRel[convention=LOGICAL, distribution=REPLICATED]

When you are about to create your physical projection it is not necessary
to create three equivalent paths but the most promising by asking the input
what can it offer as traits. Again this partially overlaps with the
discussion about "On demand traitset" [1] where as I wrote there is already
a mechanism to derive traits from children (collation, distribution, etc.).

Note that I am not saying that there is nothing more to be done to improve
the Calcite optimizer; I am just trying to provide a viable alternative
given the current situation of the project.

*Part II: Example with the original Volcano optimizer*

Now let me try to show how the original (from the paper) Volcano optimizer
could handle your query.

The optimization starts and we request two properties convention and
distribution.

RootLogicalRel[convention=PHYSICAL, distribution=SINGLETON]
-> ProjectLogicalRel
  -> MapScanLogicalRel

Rule 1: RootLogicalRel -> RootPhysicalRel applies the algorithm but cannot
satisfy the SINGLETON property so it propagates the demand

RootPhysicalRel
-> ProjectLogicalRel [convention=PHYSICAL, distribution=SINGLETON]
  -> MapScanLogicalRel

Rule 2: ProjectLogicalRel -> ProjectPhysicalRel applies the algorithm but
cannot satisfy the SINGLETON property so it propagates the demand

RootPhysicalRel
-> ProjectPhysicalRel
  -> MapScanLogicalRel [convention=PHYSICAL, distribution=SINGLETON]

Rule 3: MapScanLogicalRel -> MapScanPhysicalRel applies the algorithm and
given that the table is replicated it satisfies the distribution property.

RootPhysicalRel
-> ProjectPhysicalRel
  -> MapScanPhysicalRel

So we end up with a complete plan that satisfies all properties and does
not have redundant exchange operators.
Moreover we didn't require all possible distributions when we applied Rule
2 since we just want to satisfy the SINGLETON property requested by the
parent.
The example above does not demonstrate the additional optimization paths
which would be apply an enforcer (an Exchange operator to satisfy the
SINGLETON property) at a higher level than the scan.

Would this be an acceptable approach? Does it still suffer from a big
search space?

To make the analog with the actual Volcano optimizer in Calcite the thing
that may be missing is to be able to know what trait was requested by the
parent during the application of Rule 2.

Best,
Stamatis

[1]
https://lists.apache.org/thread.html/79dac47ea50b5dfbd3f234e368ed61d247fb0eb989f87fe01aedaf25@%3Cdev.calcite.apache.org%3E

On Wed, Oct 30, 2019 at 7:24 PM Vladimir Ozerov <pp...@gmail.com> wrote:

> Hi Stamatis,
>
> The problem that the presented reproducer is a very simplified version of
> what is actually needed. In reality, there are several distribution types -
> PARTITIONED, REPLICATED, SINGLETON. To make things worse, PARTITIONED may
> or may not have concrete distribution fields. In theory, I can create one
> transformation per distribution type, but that would increase plan space
> significantly. In my sample "Root <- Project <- Scan" plan, there is no
> room for optimization at all, we only need to convert the nodes and
> propagate traits. But if I follow the proposed approach, the planner would
> create three equivalent paths. For complex queries, this may increase
> optimization time significantly.
>
> What I need instead is to gradually convert and optimize nodes *bottom-up*,
> instead of top-bottom. That is, create a physical scan first, then create a
> physical project on top of it, etc. But at the same time, some rules still
> require the top-bottom approach. So essentially I need the optimizer to do
> both. Abstract converters help me establish bottom-up preparation but do
> this at the cost of considering too many trait pairs, and as a result, also
> pollute the search space.
>
> To the contrast, precise command "I transformed the node A to node B,
> please re-trigger the rules for A's parents" would allow us to re-trigger
> only required rules, without adding more nodes.
>
> Does it make sense?
>
> Regards,
> Vladimir.
>
> ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <za...@gmail.com>:
>
> > I admit that I didn't thoroughly read the exchanges so far but forcing
> the
> > rules trigger does not seem the right approach.
> >
> > Other than that I would like to backtrack a bit to point 4.3 raised by
> > Vladimir.
> >
> > "ProjectPhysicalRule [6] - transforms logical project to physical
> > project *ONLY* if there is an underlying physical input with REPLICATED
> or
> > SINGLETON distribution"
> >
> > The rule could be modified to do the following two transformations:
> > 1. Create a physical project and require the input to be REPLICATED.
> > 2. Create a physical project and require
> > the input to be SINGLETON.
> >
> > I would assume that afterwards when your scan rule fires it should go to
> > the appropriate subset and give you back the desired plan. Is there a
> > problem with this approach?
> >
> > Best,
> > Stamatis
> >
> > On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <gv...@gmail.com>
> > wrote:
> >
> > > Unfortunately it requires package-private API usage.
> > >
> > > Best solution there if Calcite provide it for end users.
> > >
> > > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
> > >
> > > > “e pension” == “expand conversion” :)
> > > >
> > > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <pp...@gmail.com>:
> > > >
> > > > > Yes, that may work. Even if e pension rule is used, for the most
> > cases
> > > it
> > > > > will not trigger any real conversions, since we are moving from
> > > abstract
> > > > > convention to physical, and created converters will have the
> opposite
> > > > trait
> > > > > direction (from physical to abstract).
> > > > >
> > > > > But again - ideally we only need to re-trigger the rules for a
> > specific
> > > > > node, no more than that. So API support like
> > > > > “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <
> gvvinblade@gmail.com
> > >:
> > > > >
> > > > >> I considered manual rules calling too, for now we use abstract
> > > > converters
> > > > >> +
> > > > >> ExpandConversionRule for exchanges producing.
> > > > >>
> > > > >> You may create such converters manually (checking appropriate
> > subset)
> > > > this
> > > > >> case you may reduce created converters count, also, a converter
> is a
> > > > quite
> > > > >> special node, that does almost nothing (without corresponding
> rule)
> > it
> > > > may
> > > > >> be used just as a rule trigger.
> > > > >>
> > > > >> Regards,
> > > > >> Igor
> > > > >>
> > > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <pp...@gmail.com>:
> > > > >>
> > > > >> > One funny hack which helped me is manual registration of a fake
> > > > RelNode
> > > > >> > with desired traits through VolcanoPlanner.register() method.
> But
> > > > again,
> > > > >> > this leads to trashing. What could really help is a call to
> > > > >> > VolcanoPlanner.fireRules() with desired rel. But this doesn't
> work
> > > out
> > > > >> of
> > > > >> > the box since some internals of the rule queue needs to be
> > adjusted.
> > > > >> >
> > > > >> > What does the community think about adding a method which will
> > > re-add
> > > > >> rules
> > > > >> > applicable to the specific RelNode to the rule queue?
> > > > >> >
> > > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <
> ppozerov@gmail.com
> > >:
> > > > >> >
> > > > >> > > Hi Igor,
> > > > >> > >
> > > > >> > > Yes, I came to the same conclusion, thank you. This is how it
> > > > >> basically
> > > > >> > > happens when converters are disabled:
> > > > >> > > 1) We start with initial tree: [LogicalProject] <-
> [LogicalScan]
> > > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is added
> > to
> > > > the
> > > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > > > >> > > 3) Finally, when it is time to fire a rule for PhysicalScan,
> we
> > > try
> > > > to
> > > > >> > get
> > > > >> > > parents of that scan set with traits of the PhysicalScan.
> Since
> > > > there
> > > > >> are
> > > > >> > > no such parents (we skipped it intentionally), the rule is not
> > > > queued.
> > > > >> > >
> > > > >> > > But when converters are enabled, a converter rel is created:
> > > > >> > [LogicalProject]
> > > > >> > > <- [LogicalScan, PhysicalScan,
> ConverterFromPhysicalToLogical].
> > No
> > > > >> rules
> > > > >> > > are fired for PhysicalScan again, but they are fired for
> > converter
> > > > >> since
> > > > >> > > it has the necessary LOGICAL trait.
> > > > >> > >
> > > > >> > > It makes sense, that converters essentially allow forcing rule
> > > > >> invocation
> > > > >> > > on parents, even if the child was created with different
> traits.
> > > But
> > > > >> it
> > > > >> > > seems that this mechanism may be too heavy for complex queries
> > > > >> because it
> > > > >> > > literally creates hundreds of new converter rels and triggers
> > > rules
> > > > >> over
> > > > >> > > and over again.
> > > > >> > >
> > > > >> > > We need some fine-grained alternative. Basically, what would
> be
> > > > enough
> > > > >> > for
> > > > >> > > me is to let the planner know somehow: "I created that rel,
> and
> > I
> > > > want
> > > > >> > you
> > > > >> > > to execute parent rules not only using its trait but also on
> > this
> > > > and
> > > > >> > those
> > > > >> > > traits."
> > > > >> > > Is there any API in Calcite which allows doing this without
> > > > creating a
> > > > >> > new
> > > > >> > > rel node?
> > > > >> > >
> > > > >> > > Regards,
> > > > >> > > Vladimir.
> > > > >> > >
> > > > >> > >
> > > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> > > gvvinblade@gmail.com
> > > > >:
> > > > >> > >
> > > > >> > >> Vladimir,
> > > > >> > >>
> > > > >> > >> Probably it'll help you:
> > > > >> > >>
> > > > >> > >> Seems the cause of issue in RelSubset.getParentRels()  The
> > check
> > > > used
> > > > >> > when
> > > > >> > >> the planner schedules newly matched rules after successful
> > > > >> > transformation
> > > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the parent
> rule
> > > be
> > > > >> > applied
> > > > >> > >> once again (here your logical project with an input having
> ANY
> > > > >> > >> distribution
> > > > >> > >> doesn't satisfy a transformed input traits).
> > > > >> > >>
> > > > >> > >> In our case we use another workaround, so there are also much
> > > more
> > > > >> > >> transformations than we wanted, so the desired rule is
> > triggered.
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <
> ppozerov@gmail.com
> > >:
> > > > >> > >>
> > > > >> > >> > Hi Vladimir,
> > > > >> > >> >
> > > > >> > >> > I am sorry. Pushed, it works now.
> > > > >> > >> >
> > > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > > > >> > >> > sitnikov.vladimir@gmail.com
> > > > >> > >> > >:
> > > > >> > >> >
> > > > >> > >> > > > mvn clean test
> > > > >> > >> > >
> > > > >> > >> > > [ERROR] The goal you specified requires a project to
> > execute
> > > > but
> > > > >> > >> there is
> > > > >> > >> > > no POM in this directory
> > > > >> > >> > >
> > > > >> > >> > > Vladimir, please push missing files
> > > > >> > >> > >
> > > > >> > >> > > Vladimir
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
Hi Stamatis,

The problem that the presented reproducer is a very simplified version of
what is actually needed. In reality, there are several distribution types -
PARTITIONED, REPLICATED, SINGLETON. To make things worse, PARTITIONED may
or may not have concrete distribution fields. In theory, I can create one
transformation per distribution type, but that would increase plan space
significantly. In my sample "Root <- Project <- Scan" plan, there is no
room for optimization at all, we only need to convert the nodes and
propagate traits. But if I follow the proposed approach, the planner would
create three equivalent paths. For complex queries, this may increase
optimization time significantly.

What I need instead is to gradually convert and optimize nodes *bottom-up*,
instead of top-bottom. That is, create a physical scan first, then create a
physical project on top of it, etc. But at the same time, some rules still
require the top-bottom approach. So essentially I need the optimizer to do
both. Abstract converters help me establish bottom-up preparation but do
this at the cost of considering too many trait pairs, and as a result, also
pollute the search space.

To the contrast, precise command "I transformed the node A to node B,
please re-trigger the rules for A's parents" would allow us to re-trigger
only required rules, without adding more nodes.

Does it make sense?

Regards,
Vladimir.

ср, 30 окт. 2019 г. в 21:02, Stamatis Zampetakis <za...@gmail.com>:

> I admit that I didn't thoroughly read the exchanges so far but forcing the
> rules trigger does not seem the right approach.
>
> Other than that I would like to backtrack a bit to point 4.3 raised by
> Vladimir.
>
> "ProjectPhysicalRule [6] - transforms logical project to physical
> project *ONLY* if there is an underlying physical input with REPLICATED or
> SINGLETON distribution"
>
> The rule could be modified to do the following two transformations:
> 1. Create a physical project and require the input to be REPLICATED.
> 2. Create a physical project and require
> the input to be SINGLETON.
>
> I would assume that afterwards when your scan rule fires it should go to
> the appropriate subset and give you back the desired plan. Is there a
> problem with this approach?
>
> Best,
> Stamatis
>
> On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <gv...@gmail.com>
> wrote:
>
> > Unfortunately it requires package-private API usage.
> >
> > Best solution there if Calcite provide it for end users.
> >
> > ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
> >
> > > “e pension” == “expand conversion” :)
> > >
> > > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <pp...@gmail.com>:
> > >
> > > > Yes, that may work. Even if e pension rule is used, for the most
> cases
> > it
> > > > will not trigger any real conversions, since we are moving from
> > abstract
> > > > convention to physical, and created converters will have the opposite
> > > trait
> > > > direction (from physical to abstract).
> > > >
> > > > But again - ideally we only need to re-trigger the rules for a
> specific
> > > > node, no more than that. So API support like
> > > > “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
> > > >
> > > > What do you think?
> > > >
> > > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <gvvinblade@gmail.com
> >:
> > > >
> > > >> I considered manual rules calling too, for now we use abstract
> > > converters
> > > >> +
> > > >> ExpandConversionRule for exchanges producing.
> > > >>
> > > >> You may create such converters manually (checking appropriate
> subset)
> > > this
> > > >> case you may reduce created converters count, also, a converter is a
> > > quite
> > > >> special node, that does almost nothing (without corresponding rule)
> it
> > > may
> > > >> be used just as a rule trigger.
> > > >>
> > > >> Regards,
> > > >> Igor
> > > >>
> > > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <pp...@gmail.com>:
> > > >>
> > > >> > One funny hack which helped me is manual registration of a fake
> > > RelNode
> > > >> > with desired traits through VolcanoPlanner.register() method. But
> > > again,
> > > >> > this leads to trashing. What could really help is a call to
> > > >> > VolcanoPlanner.fireRules() with desired rel. But this doesn't work
> > out
> > > >> of
> > > >> > the box since some internals of the rule queue needs to be
> adjusted.
> > > >> >
> > > >> > What does the community think about adding a method which will
> > re-add
> > > >> rules
> > > >> > applicable to the specific RelNode to the rule queue?
> > > >> >
> > > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <ppozerov@gmail.com
> >:
> > > >> >
> > > >> > > Hi Igor,
> > > >> > >
> > > >> > > Yes, I came to the same conclusion, thank you. This is how it
> > > >> basically
> > > >> > > happens when converters are disabled:
> > > >> > > 1) We start with initial tree: [LogicalProject] <- [LogicalScan]
> > > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is added
> to
> > > the
> > > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > > >> > > 3) Finally, when it is time to fire a rule for PhysicalScan, we
> > try
> > > to
> > > >> > get
> > > >> > > parents of that scan set with traits of the PhysicalScan. Since
> > > there
> > > >> are
> > > >> > > no such parents (we skipped it intentionally), the rule is not
> > > queued.
> > > >> > >
> > > >> > > But when converters are enabled, a converter rel is created:
> > > >> > [LogicalProject]
> > > >> > > <- [LogicalScan, PhysicalScan, ConverterFromPhysicalToLogical].
> No
> > > >> rules
> > > >> > > are fired for PhysicalScan again, but they are fired for
> converter
> > > >> since
> > > >> > > it has the necessary LOGICAL trait.
> > > >> > >
> > > >> > > It makes sense, that converters essentially allow forcing rule
> > > >> invocation
> > > >> > > on parents, even if the child was created with different traits.
> > But
> > > >> it
> > > >> > > seems that this mechanism may be too heavy for complex queries
> > > >> because it
> > > >> > > literally creates hundreds of new converter rels and triggers
> > rules
> > > >> over
> > > >> > > and over again.
> > > >> > >
> > > >> > > We need some fine-grained alternative. Basically, what would be
> > > enough
> > > >> > for
> > > >> > > me is to let the planner know somehow: "I created that rel, and
> I
> > > want
> > > >> > you
> > > >> > > to execute parent rules not only using its trait but also on
> this
> > > and
> > > >> > those
> > > >> > > traits."
> > > >> > > Is there any API in Calcite which allows doing this without
> > > creating a
> > > >> > new
> > > >> > > rel node?
> > > >> > >
> > > >> > > Regards,
> > > >> > > Vladimir.
> > > >> > >
> > > >> > >
> > > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> > gvvinblade@gmail.com
> > > >:
> > > >> > >
> > > >> > >> Vladimir,
> > > >> > >>
> > > >> > >> Probably it'll help you:
> > > >> > >>
> > > >> > >> Seems the cause of issue in RelSubset.getParentRels()  The
> check
> > > used
> > > >> > when
> > > >> > >> the planner schedules newly matched rules after successful
> > > >> > transformation
> > > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the parent rule
> > be
> > > >> > applied
> > > >> > >> once again (here your logical project with an input having ANY
> > > >> > >> distribution
> > > >> > >> doesn't satisfy a transformed input traits).
> > > >> > >>
> > > >> > >> In our case we use another workaround, so there are also much
> > more
> > > >> > >> transformations than we wanted, so the desired rule is
> triggered.
> > > >> > >>
> > > >> > >>
> > > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <ppozerov@gmail.com
> >:
> > > >> > >>
> > > >> > >> > Hi Vladimir,
> > > >> > >> >
> > > >> > >> > I am sorry. Pushed, it works now.
> > > >> > >> >
> > > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > > >> > >> > sitnikov.vladimir@gmail.com
> > > >> > >> > >:
> > > >> > >> >
> > > >> > >> > > > mvn clean test
> > > >> > >> > >
> > > >> > >> > > [ERROR] The goal you specified requires a project to
> execute
> > > but
> > > >> > >> there is
> > > >> > >> > > no POM in this directory
> > > >> > >> > >
> > > >> > >> > > Vladimir, please push missing files
> > > >> > >> > >
> > > >> > >> > > Vladimir
> > > >> > >> > >
> > > >> > >> >
> > > >> > >>
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: Problem with converters and possibly rule matching

Posted by Stamatis Zampetakis <za...@gmail.com>.
I admit that I didn't thoroughly read the exchanges so far but forcing the
rules trigger does not seem the right approach.

Other than that I would like to backtrack a bit to point 4.3 raised by
Vladimir.

"ProjectPhysicalRule [6] - transforms logical project to physical
project *ONLY* if there is an underlying physical input with REPLICATED or
SINGLETON distribution"

The rule could be modified to do the following two transformations:
1. Create a physical project and require the input to be REPLICATED.
2. Create a physical project and require
the input to be SINGLETON.

I would assume that afterwards when your scan rule fires it should go to
the appropriate subset and give you back the desired plan. Is there a
problem with this approach?

Best,
Stamatis

On Wed, Oct 30, 2019, 5:52 PM Seliverstov Igor <gv...@gmail.com> wrote:

> Unfortunately it requires package-private API usage.
>
> Best solution there if Calcite provide it for end users.
>
> ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:
>
> > “e pension” == “expand conversion” :)
> >
> > ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <pp...@gmail.com>:
> >
> > > Yes, that may work. Even if e pension rule is used, for the most cases
> it
> > > will not trigger any real conversions, since we are moving from
> abstract
> > > convention to physical, and created converters will have the opposite
> > trait
> > > direction (from physical to abstract).
> > >
> > > But again - ideally we only need to re-trigger the rules for a specific
> > > node, no more than that. So API support like
> > > “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
> > >
> > > What do you think?
> > >
> > > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <gv...@gmail.com>:
> > >
> > >> I considered manual rules calling too, for now we use abstract
> > converters
> > >> +
> > >> ExpandConversionRule for exchanges producing.
> > >>
> > >> You may create such converters manually (checking appropriate subset)
> > this
> > >> case you may reduce created converters count, also, a converter is a
> > quite
> > >> special node, that does almost nothing (without corresponding rule) it
> > may
> > >> be used just as a rule trigger.
> > >>
> > >> Regards,
> > >> Igor
> > >>
> > >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <pp...@gmail.com>:
> > >>
> > >> > One funny hack which helped me is manual registration of a fake
> > RelNode
> > >> > with desired traits through VolcanoPlanner.register() method. But
> > again,
> > >> > this leads to trashing. What could really help is a call to
> > >> > VolcanoPlanner.fireRules() with desired rel. But this doesn't work
> out
> > >> of
> > >> > the box since some internals of the rule queue needs to be adjusted.
> > >> >
> > >> > What does the community think about adding a method which will
> re-add
> > >> rules
> > >> > applicable to the specific RelNode to the rule queue?
> > >> >
> > >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <pp...@gmail.com>:
> > >> >
> > >> > > Hi Igor,
> > >> > >
> > >> > > Yes, I came to the same conclusion, thank you. This is how it
> > >> basically
> > >> > > happens when converters are disabled:
> > >> > > 1) We start with initial tree: [LogicalProject] <- [LogicalScan]
> > >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is added to
> > the
> > >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > >> > > 3) Finally, when it is time to fire a rule for PhysicalScan, we
> try
> > to
> > >> > get
> > >> > > parents of that scan set with traits of the PhysicalScan. Since
> > there
> > >> are
> > >> > > no such parents (we skipped it intentionally), the rule is not
> > queued.
> > >> > >
> > >> > > But when converters are enabled, a converter rel is created:
> > >> > [LogicalProject]
> > >> > > <- [LogicalScan, PhysicalScan, ConverterFromPhysicalToLogical]. No
> > >> rules
> > >> > > are fired for PhysicalScan again, but they are fired for converter
> > >> since
> > >> > > it has the necessary LOGICAL trait.
> > >> > >
> > >> > > It makes sense, that converters essentially allow forcing rule
> > >> invocation
> > >> > > on parents, even if the child was created with different traits.
> But
> > >> it
> > >> > > seems that this mechanism may be too heavy for complex queries
> > >> because it
> > >> > > literally creates hundreds of new converter rels and triggers
> rules
> > >> over
> > >> > > and over again.
> > >> > >
> > >> > > We need some fine-grained alternative. Basically, what would be
> > enough
> > >> > for
> > >> > > me is to let the planner know somehow: "I created that rel, and I
> > want
> > >> > you
> > >> > > to execute parent rules not only using its trait but also on this
> > and
> > >> > those
> > >> > > traits."
> > >> > > Is there any API in Calcite which allows doing this without
> > creating a
> > >> > new
> > >> > > rel node?
> > >> > >
> > >> > > Regards,
> > >> > > Vladimir.
> > >> > >
> > >> > >
> > >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <
> gvvinblade@gmail.com
> > >:
> > >> > >
> > >> > >> Vladimir,
> > >> > >>
> > >> > >> Probably it'll help you:
> > >> > >>
> > >> > >> Seems the cause of issue in RelSubset.getParentRels()  The check
> > used
> > >> > when
> > >> > >> the planner schedules newly matched rules after successful
> > >> > transformation
> > >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the parent rule
> be
> > >> > applied
> > >> > >> once again (here your logical project with an input having ANY
> > >> > >> distribution
> > >> > >> doesn't satisfy a transformed input traits).
> > >> > >>
> > >> > >> In our case we use another workaround, so there are also much
> more
> > >> > >> transformations than we wanted, so the desired rule is triggered.
> > >> > >>
> > >> > >>
> > >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <pp...@gmail.com>:
> > >> > >>
> > >> > >> > Hi Vladimir,
> > >> > >> >
> > >> > >> > I am sorry. Pushed, it works now.
> > >> > >> >
> > >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > >> > >> > sitnikov.vladimir@gmail.com
> > >> > >> > >:
> > >> > >> >
> > >> > >> > > > mvn clean test
> > >> > >> > >
> > >> > >> > > [ERROR] The goal you specified requires a project to execute
> > but
> > >> > >> there is
> > >> > >> > > no POM in this directory
> > >> > >> > >
> > >> > >> > > Vladimir, please push missing files
> > >> > >> > >
> > >> > >> > > Vladimir
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: Problem with converters and possibly rule matching

Posted by Seliverstov Igor <gv...@gmail.com>.
Unfortunately it requires package-private API usage.

Best solution there if Calcite provide it for end users.

ср, 30 окт. 2019 г., 18:48 Vladimir Ozerov <pp...@gmail.com>:

> “e pension” == “expand conversion” :)
>
> ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <pp...@gmail.com>:
>
> > Yes, that may work. Even if e pension rule is used, for the most cases it
> > will not trigger any real conversions, since we are moving from abstract
> > convention to physical, and created converters will have the opposite
> trait
> > direction (from physical to abstract).
> >
> > But again - ideally we only need to re-trigger the rules for a specific
> > node, no more than that. So API support like
> > “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
> >
> > What do you think?
> >
> > ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <gv...@gmail.com>:
> >
> >> I considered manual rules calling too, for now we use abstract
> converters
> >> +
> >> ExpandConversionRule for exchanges producing.
> >>
> >> You may create such converters manually (checking appropriate subset)
> this
> >> case you may reduce created converters count, also, a converter is a
> quite
> >> special node, that does almost nothing (without corresponding rule) it
> may
> >> be used just as a rule trigger.
> >>
> >> Regards,
> >> Igor
> >>
> >> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <pp...@gmail.com>:
> >>
> >> > One funny hack which helped me is manual registration of a fake
> RelNode
> >> > with desired traits through VolcanoPlanner.register() method. But
> again,
> >> > this leads to trashing. What could really help is a call to
> >> > VolcanoPlanner.fireRules() with desired rel. But this doesn't work out
> >> of
> >> > the box since some internals of the rule queue needs to be adjusted.
> >> >
> >> > What does the community think about adding a method which will re-add
> >> rules
> >> > applicable to the specific RelNode to the rule queue?
> >> >
> >> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <pp...@gmail.com>:
> >> >
> >> > > Hi Igor,
> >> > >
> >> > > Yes, I came to the same conclusion, thank you. This is how it
> >> basically
> >> > > happens when converters are disabled:
> >> > > 1) We start with initial tree: [LogicalProject] <- [LogicalScan]
> >> > > 2) Then we convert LogicalScan to PhysicalScan, so it is added to
> the
> >> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> >> > > 3) Finally, when it is time to fire a rule for PhysicalScan, we try
> to
> >> > get
> >> > > parents of that scan set with traits of the PhysicalScan. Since
> there
> >> are
> >> > > no such parents (we skipped it intentionally), the rule is not
> queued.
> >> > >
> >> > > But when converters are enabled, a converter rel is created:
> >> > [LogicalProject]
> >> > > <- [LogicalScan, PhysicalScan, ConverterFromPhysicalToLogical]. No
> >> rules
> >> > > are fired for PhysicalScan again, but they are fired for converter
> >> since
> >> > > it has the necessary LOGICAL trait.
> >> > >
> >> > > It makes sense, that converters essentially allow forcing rule
> >> invocation
> >> > > on parents, even if the child was created with different traits. But
> >> it
> >> > > seems that this mechanism may be too heavy for complex queries
> >> because it
> >> > > literally creates hundreds of new converter rels and triggers rules
> >> over
> >> > > and over again.
> >> > >
> >> > > We need some fine-grained alternative. Basically, what would be
> enough
> >> > for
> >> > > me is to let the planner know somehow: "I created that rel, and I
> want
> >> > you
> >> > > to execute parent rules not only using its trait but also on this
> and
> >> > those
> >> > > traits."
> >> > > Is there any API in Calcite which allows doing this without
> creating a
> >> > new
> >> > > rel node?
> >> > >
> >> > > Regards,
> >> > > Vladimir.
> >> > >
> >> > >
> >> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <gvvinblade@gmail.com
> >:
> >> > >
> >> > >> Vladimir,
> >> > >>
> >> > >> Probably it'll help you:
> >> > >>
> >> > >> Seems the cause of issue in RelSubset.getParentRels()  The check
> used
> >> > when
> >> > >> the planner schedules newly matched rules after successful
> >> > transformation
> >> > >> (see VolcanoRuleCall.matchRecurse), it prevents the parent rule be
> >> > applied
> >> > >> once again (here your logical project with an input having ANY
> >> > >> distribution
> >> > >> doesn't satisfy a transformed input traits).
> >> > >>
> >> > >> In our case we use another workaround, so there are also much more
> >> > >> transformations than we wanted, so the desired rule is triggered.
> >> > >>
> >> > >>
> >> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <pp...@gmail.com>:
> >> > >>
> >> > >> > Hi Vladimir,
> >> > >> >
> >> > >> > I am sorry. Pushed, it works now.
> >> > >> >
> >> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> >> > >> > sitnikov.vladimir@gmail.com
> >> > >> > >:
> >> > >> >
> >> > >> > > > mvn clean test
> >> > >> > >
> >> > >> > > [ERROR] The goal you specified requires a project to execute
> but
> >> > >> there is
> >> > >> > > no POM in this directory
> >> > >> > >
> >> > >> > > Vladimir, please push missing files
> >> > >> > >
> >> > >> > > Vladimir
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >
> >> >
> >>
> >
>

Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
“e pension” == “expand conversion” :)

ср, 30 окт. 2019 г. в 18:46, Vladimir Ozerov <pp...@gmail.com>:

> Yes, that may work. Even if e pension rule is used, for the most cases it
> will not trigger any real conversions, since we are moving from abstract
> convention to physical, and created converters will have the opposite trait
> direction (from physical to abstract).
>
> But again - ideally we only need to re-trigger the rules for a specific
> node, no more than that. So API support like
> “VolcanoPlanner.forceRules(RelNode)” would be very convenient.
>
> What do you think?
>
> ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <gv...@gmail.com>:
>
>> I considered manual rules calling too, for now we use abstract converters
>> +
>> ExpandConversionRule for exchanges producing.
>>
>> You may create such converters manually (checking appropriate subset) this
>> case you may reduce created converters count, also, a converter is a quite
>> special node, that does almost nothing (without corresponding rule) it may
>> be used just as a rule trigger.
>>
>> Regards,
>> Igor
>>
>> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <pp...@gmail.com>:
>>
>> > One funny hack which helped me is manual registration of a fake RelNode
>> > with desired traits through VolcanoPlanner.register() method. But again,
>> > this leads to trashing. What could really help is a call to
>> > VolcanoPlanner.fireRules() with desired rel. But this doesn't work out
>> of
>> > the box since some internals of the rule queue needs to be adjusted.
>> >
>> > What does the community think about adding a method which will re-add
>> rules
>> > applicable to the specific RelNode to the rule queue?
>> >
>> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <pp...@gmail.com>:
>> >
>> > > Hi Igor,
>> > >
>> > > Yes, I came to the same conclusion, thank you. This is how it
>> basically
>> > > happens when converters are disabled:
>> > > 1) We start with initial tree: [LogicalProject] <- [LogicalScan]
>> > > 2) Then we convert LogicalScan to PhysicalScan, so it is added to the
>> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
>> > > 3) Finally, when it is time to fire a rule for PhysicalScan, we try to
>> > get
>> > > parents of that scan set with traits of the PhysicalScan. Since there
>> are
>> > > no such parents (we skipped it intentionally), the rule is not queued.
>> > >
>> > > But when converters are enabled, a converter rel is created:
>> > [LogicalProject]
>> > > <- [LogicalScan, PhysicalScan, ConverterFromPhysicalToLogical]. No
>> rules
>> > > are fired for PhysicalScan again, but they are fired for converter
>> since
>> > > it has the necessary LOGICAL trait.
>> > >
>> > > It makes sense, that converters essentially allow forcing rule
>> invocation
>> > > on parents, even if the child was created with different traits. But
>> it
>> > > seems that this mechanism may be too heavy for complex queries
>> because it
>> > > literally creates hundreds of new converter rels and triggers rules
>> over
>> > > and over again.
>> > >
>> > > We need some fine-grained alternative. Basically, what would be enough
>> > for
>> > > me is to let the planner know somehow: "I created that rel, and I want
>> > you
>> > > to execute parent rules not only using its trait but also on this and
>> > those
>> > > traits."
>> > > Is there any API in Calcite which allows doing this without creating a
>> > new
>> > > rel node?
>> > >
>> > > Regards,
>> > > Vladimir.
>> > >
>> > >
>> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <gv...@gmail.com>:
>> > >
>> > >> Vladimir,
>> > >>
>> > >> Probably it'll help you:
>> > >>
>> > >> Seems the cause of issue in RelSubset.getParentRels()  The check used
>> > when
>> > >> the planner schedules newly matched rules after successful
>> > transformation
>> > >> (see VolcanoRuleCall.matchRecurse), it prevents the parent rule be
>> > applied
>> > >> once again (here your logical project with an input having ANY
>> > >> distribution
>> > >> doesn't satisfy a transformed input traits).
>> > >>
>> > >> In our case we use another workaround, so there are also much more
>> > >> transformations than we wanted, so the desired rule is triggered.
>> > >>
>> > >>
>> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <pp...@gmail.com>:
>> > >>
>> > >> > Hi Vladimir,
>> > >> >
>> > >> > I am sorry. Pushed, it works now.
>> > >> >
>> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
>> > >> > sitnikov.vladimir@gmail.com
>> > >> > >:
>> > >> >
>> > >> > > > mvn clean test
>> > >> > >
>> > >> > > [ERROR] The goal you specified requires a project to execute but
>> > >> there is
>> > >> > > no POM in this directory
>> > >> > >
>> > >> > > Vladimir, please push missing files
>> > >> > >
>> > >> > > Vladimir
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>

Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
Yes, that may work. Even if e pension rule is used, for the most cases it
will not trigger any real conversions, since we are moving from abstract
convention to physical, and created converters will have the opposite trait
direction (from physical to abstract).

But again - ideally we only need to re-trigger the rules for a specific
node, no more than that. So API support like
“VolcanoPlanner.forceRules(RelNode)” would be very convenient.

What do you think?

ср, 30 окт. 2019 г. в 17:56, Seliverstov Igor <gv...@gmail.com>:

> I considered manual rules calling too, for now we use abstract converters +
> ExpandConversionRule for exchanges producing.
>
> You may create such converters manually (checking appropriate subset) this
> case you may reduce created converters count, also, a converter is a quite
> special node, that does almost nothing (without corresponding rule) it may
> be used just as a rule trigger.
>
> Regards,
> Igor
>
> ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <pp...@gmail.com>:
>
> > One funny hack which helped me is manual registration of a fake RelNode
> > with desired traits through VolcanoPlanner.register() method. But again,
> > this leads to trashing. What could really help is a call to
> > VolcanoPlanner.fireRules() with desired rel. But this doesn't work out of
> > the box since some internals of the rule queue needs to be adjusted.
> >
> > What does the community think about adding a method which will re-add
> rules
> > applicable to the specific RelNode to the rule queue?
> >
> > ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <pp...@gmail.com>:
> >
> > > Hi Igor,
> > >
> > > Yes, I came to the same conclusion, thank you. This is how it basically
> > > happens when converters are disabled:
> > > 1) We start with initial tree: [LogicalProject] <- [LogicalScan]
> > > 2) Then we convert LogicalScan to PhysicalScan, so it is added to the
> > > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > > 3) Finally, when it is time to fire a rule for PhysicalScan, we try to
> > get
> > > parents of that scan set with traits of the PhysicalScan. Since there
> are
> > > no such parents (we skipped it intentionally), the rule is not queued.
> > >
> > > But when converters are enabled, a converter rel is created:
> > [LogicalProject]
> > > <- [LogicalScan, PhysicalScan, ConverterFromPhysicalToLogical]. No
> rules
> > > are fired for PhysicalScan again, but they are fired for converter
> since
> > > it has the necessary LOGICAL trait.
> > >
> > > It makes sense, that converters essentially allow forcing rule
> invocation
> > > on parents, even if the child was created with different traits. But it
> > > seems that this mechanism may be too heavy for complex queries because
> it
> > > literally creates hundreds of new converter rels and triggers rules
> over
> > > and over again.
> > >
> > > We need some fine-grained alternative. Basically, what would be enough
> > for
> > > me is to let the planner know somehow: "I created that rel, and I want
> > you
> > > to execute parent rules not only using its trait but also on this and
> > those
> > > traits."
> > > Is there any API in Calcite which allows doing this without creating a
> > new
> > > rel node?
> > >
> > > Regards,
> > > Vladimir.
> > >
> > >
> > > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <gv...@gmail.com>:
> > >
> > >> Vladimir,
> > >>
> > >> Probably it'll help you:
> > >>
> > >> Seems the cause of issue in RelSubset.getParentRels()  The check used
> > when
> > >> the planner schedules newly matched rules after successful
> > transformation
> > >> (see VolcanoRuleCall.matchRecurse), it prevents the parent rule be
> > applied
> > >> once again (here your logical project with an input having ANY
> > >> distribution
> > >> doesn't satisfy a transformed input traits).
> > >>
> > >> In our case we use another workaround, so there are also much more
> > >> transformations than we wanted, so the desired rule is triggered.
> > >>
> > >>
> > >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <pp...@gmail.com>:
> > >>
> > >> > Hi Vladimir,
> > >> >
> > >> > I am sorry. Pushed, it works now.
> > >> >
> > >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > >> > sitnikov.vladimir@gmail.com
> > >> > >:
> > >> >
> > >> > > > mvn clean test
> > >> > >
> > >> > > [ERROR] The goal you specified requires a project to execute but
> > >> there is
> > >> > > no POM in this directory
> > >> > >
> > >> > > Vladimir, please push missing files
> > >> > >
> > >> > > Vladimir
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: Problem with converters and possibly rule matching

Posted by Seliverstov Igor <gv...@gmail.com>.
I considered manual rules calling too, for now we use abstract converters +
ExpandConversionRule for exchanges producing.

You may create such converters manually (checking appropriate subset) this
case you may reduce created converters count, also, a converter is a quite
special node, that does almost nothing (without corresponding rule) it may
be used just as a rule trigger.

Regards,
Igor

ср, 30 окт. 2019 г., 17:31 Vladimir Ozerov <pp...@gmail.com>:

> One funny hack which helped me is manual registration of a fake RelNode
> with desired traits through VolcanoPlanner.register() method. But again,
> this leads to trashing. What could really help is a call to
> VolcanoPlanner.fireRules() with desired rel. But this doesn't work out of
> the box since some internals of the rule queue needs to be adjusted.
>
> What does the community think about adding a method which will re-add rules
> applicable to the specific RelNode to the rule queue?
>
> ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <pp...@gmail.com>:
>
> > Hi Igor,
> >
> > Yes, I came to the same conclusion, thank you. This is how it basically
> > happens when converters are disabled:
> > 1) We start with initial tree: [LogicalProject] <- [LogicalScan]
> > 2) Then we convert LogicalScan to PhysicalScan, so it is added to the
> > set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> > 3) Finally, when it is time to fire a rule for PhysicalScan, we try to
> get
> > parents of that scan set with traits of the PhysicalScan. Since there are
> > no such parents (we skipped it intentionally), the rule is not queued.
> >
> > But when converters are enabled, a converter rel is created:
> [LogicalProject]
> > <- [LogicalScan, PhysicalScan, ConverterFromPhysicalToLogical]. No rules
> > are fired for PhysicalScan again, but they are fired for converter since
> > it has the necessary LOGICAL trait.
> >
> > It makes sense, that converters essentially allow forcing rule invocation
> > on parents, even if the child was created with different traits. But it
> > seems that this mechanism may be too heavy for complex queries because it
> > literally creates hundreds of new converter rels and triggers rules over
> > and over again.
> >
> > We need some fine-grained alternative. Basically, what would be enough
> for
> > me is to let the planner know somehow: "I created that rel, and I want
> you
> > to execute parent rules not only using its trait but also on this and
> those
> > traits."
> > Is there any API in Calcite which allows doing this without creating a
> new
> > rel node?
> >
> > Regards,
> > Vladimir.
> >
> >
> > ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <gv...@gmail.com>:
> >
> >> Vladimir,
> >>
> >> Probably it'll help you:
> >>
> >> Seems the cause of issue in RelSubset.getParentRels()  The check used
> when
> >> the planner schedules newly matched rules after successful
> transformation
> >> (see VolcanoRuleCall.matchRecurse), it prevents the parent rule be
> applied
> >> once again (here your logical project with an input having ANY
> >> distribution
> >> doesn't satisfy a transformed input traits).
> >>
> >> In our case we use another workaround, so there are also much more
> >> transformations than we wanted, so the desired rule is triggered.
> >>
> >>
> >> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <pp...@gmail.com>:
> >>
> >> > Hi Vladimir,
> >> >
> >> > I am sorry. Pushed, it works now.
> >> >
> >> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> >> > sitnikov.vladimir@gmail.com
> >> > >:
> >> >
> >> > > > mvn clean test
> >> > >
> >> > > [ERROR] The goal you specified requires a project to execute but
> >> there is
> >> > > no POM in this directory
> >> > >
> >> > > Vladimir, please push missing files
> >> > >
> >> > > Vladimir
> >> > >
> >> >
> >>
> >
>

Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
One funny hack which helped me is manual registration of a fake RelNode
with desired traits through VolcanoPlanner.register() method. But again,
this leads to trashing. What could really help is a call to
VolcanoPlanner.fireRules() with desired rel. But this doesn't work out of
the box since some internals of the rule queue needs to be adjusted.

What does the community think about adding a method which will re-add rules
applicable to the specific RelNode to the rule queue?

ср, 30 окт. 2019 г. в 17:00, Vladimir Ozerov <pp...@gmail.com>:

> Hi Igor,
>
> Yes, I came to the same conclusion, thank you. This is how it basically
> happens when converters are disabled:
> 1) We start with initial tree: [LogicalProject] <- [LogicalScan]
> 2) Then we convert LogicalScan to PhysicalScan, so it is added to the
> set: [LogicalProject] <- [LogicalScan, PhysicalScan]
> 3) Finally, when it is time to fire a rule for PhysicalScan, we try to get
> parents of that scan set with traits of the PhysicalScan. Since there are
> no such parents (we skipped it intentionally), the rule is not queued.
>
> But when converters are enabled, a converter rel is created: [LogicalProject]
> <- [LogicalScan, PhysicalScan, ConverterFromPhysicalToLogical]. No rules
> are fired for PhysicalScan again, but they are fired for converter since
> it has the necessary LOGICAL trait.
>
> It makes sense, that converters essentially allow forcing rule invocation
> on parents, even if the child was created with different traits. But it
> seems that this mechanism may be too heavy for complex queries because it
> literally creates hundreds of new converter rels and triggers rules over
> and over again.
>
> We need some fine-grained alternative. Basically, what would be enough for
> me is to let the planner know somehow: "I created that rel, and I want you
> to execute parent rules not only using its trait but also on this and those
> traits."
> Is there any API in Calcite which allows doing this without creating a new
> rel node?
>
> Regards,
> Vladimir.
>
>
> ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <gv...@gmail.com>:
>
>> Vladimir,
>>
>> Probably it'll help you:
>>
>> Seems the cause of issue in RelSubset.getParentRels()  The check used when
>> the planner schedules newly matched rules after successful transformation
>> (see VolcanoRuleCall.matchRecurse), it prevents the parent rule be applied
>> once again (here your logical project with an input having ANY
>> distribution
>> doesn't satisfy a transformed input traits).
>>
>> In our case we use another workaround, so there are also much more
>> transformations than we wanted, so the desired rule is triggered.
>>
>>
>> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <pp...@gmail.com>:
>>
>> > Hi Vladimir,
>> >
>> > I am sorry. Pushed, it works now.
>> >
>> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
>> > sitnikov.vladimir@gmail.com
>> > >:
>> >
>> > > > mvn clean test
>> > >
>> > > [ERROR] The goal you specified requires a project to execute but
>> there is
>> > > no POM in this directory
>> > >
>> > > Vladimir, please push missing files
>> > >
>> > > Vladimir
>> > >
>> >
>>
>

Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
Hi Igor,

Yes, I came to the same conclusion, thank you. This is how it basically
happens when converters are disabled:
1) We start with initial tree: [LogicalProject] <- [LogicalScan]
2) Then we convert LogicalScan to PhysicalScan, so it is added to the
set: [LogicalProject]
<- [LogicalScan, PhysicalScan]
3) Finally, when it is time to fire a rule for PhysicalScan, we try to get
parents of that scan set with traits of the PhysicalScan. Since there are
no such parents (we skipped it intentionally), the rule is not queued.

But when converters are enabled, a converter rel is created: [LogicalProject]
<- [LogicalScan, PhysicalScan, ConverterFromPhysicalToLogical]. No rules
are fired for PhysicalScan again, but they are fired for converter since it
has the necessary LOGICAL trait.

It makes sense, that converters essentially allow forcing rule invocation
on parents, even if the child was created with different traits. But it
seems that this mechanism may be too heavy for complex queries because it
literally creates hundreds of new converter rels and triggers rules over
and over again.

We need some fine-grained alternative. Basically, what would be enough for
me is to let the planner know somehow: "I created that rel, and I want you
to execute parent rules not only using its trait but also on this and those
traits."
Is there any API in Calcite which allows doing this without creating a new
rel node?

Regards,
Vladimir.


ср, 30 окт. 2019 г. в 09:25, Seliverstov Igor <gv...@gmail.com>:

> Vladimir,
>
> Probably it'll help you:
>
> Seems the cause of issue in RelSubset.getParentRels()  The check used when
> the planner schedules newly matched rules after successful transformation
> (see VolcanoRuleCall.matchRecurse), it prevents the parent rule be applied
> once again (here your logical project with an input having ANY distribution
> doesn't satisfy a transformed input traits).
>
> In our case we use another workaround, so there are also much more
> transformations than we wanted, so the desired rule is triggered.
>
>
> вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <pp...@gmail.com>:
>
> > Hi Vladimir,
> >
> > I am sorry. Pushed, it works now.
> >
> > вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> > sitnikov.vladimir@gmail.com
> > >:
> >
> > > > mvn clean test
> > >
> > > [ERROR] The goal you specified requires a project to execute but there
> is
> > > no POM in this directory
> > >
> > > Vladimir, please push missing files
> > >
> > > Vladimir
> > >
> >
>

Re: Problem with converters and possibly rule matching

Posted by Seliverstov Igor <gv...@gmail.com>.
Vladimir,

Probably it'll help you:

Seems the cause of issue in RelSubset.getParentRels()  The check used when
the planner schedules newly matched rules after successful transformation
(see VolcanoRuleCall.matchRecurse), it prevents the parent rule be applied
once again (here your logical project with an input having ANY distribution
doesn't satisfy a transformed input traits).

In our case we use another workaround, so there are also much more
transformations than we wanted, so the desired rule is triggered.


вт, 29 окт. 2019 г., 14:46 Vladimir Ozerov <pp...@gmail.com>:

> Hi Vladimir,
>
> I am sorry. Pushed, it works now.
>
> вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <
> sitnikov.vladimir@gmail.com
> >:
>
> > > mvn clean test
> >
> > [ERROR] The goal you specified requires a project to execute but there is
> > no POM in this directory
> >
> > Vladimir, please push missing files
> >
> > Vladimir
> >
>

Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
Hi Vladimir,

I am sorry. Pushed, it works now.

вт, 29 окт. 2019 г. в 14:41, Vladimir Sitnikov <sitnikov.vladimir@gmail.com
>:

> > mvn clean test
>
> [ERROR] The goal you specified requires a project to execute but there is
> no POM in this directory
>
> Vladimir, please push missing files
>
> Vladimir
>

Re: Problem with converters and possibly rule matching

Posted by Vladimir Sitnikov <si...@gmail.com>.
> mvn clean test

[ERROR] The goal you specified requires a project to execute but there is
no POM in this directory

Vladimir, please push missing files

Vladimir

Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
Hi colleagues,

I prepared (relatively) simple reproducer for the problem [1]. It can be
executed with three commands:
git clone https://github.com/devozerov/calcite-optimizer.git
cd calcite-optimizer
mvn clean test

Please do not pay attention to stuff located inside "org.apache.calcite"
package, as it is most likely of little interest for the given problem. I
removed a lot of unrelated stuff for the sake of simplicity, so some things
may seem strange or unnecessary.

So what is going on here?
1) We have a distributed engine that should assemble data from remote
nodes. Let's say, we have three distribution types [2]:
ANY - abstract unknown distribution, which is yet to be resolved during
physical planning. This is the default
REPLICATED - this is how the tables are organized - they are copied to all
nodes
SINGLETON - in this example, this is the distribution of a final aggregator
node which is enforced during physical planning

REPLICATED satisfies SINGLETON. SINGLETON doesn't satisfy REPLICATED.

2) There is a query "SELECT FUNC(f) FROM t", which is converted to the
following form with LOGICAL convention during logical planning:
RootLogicalRel[convention=LOGICAL, distribution=ANY]
-> ProjectLogicalRel[convention=LOGICAL, distribution=ANY]
  -> MapScanLogicalRel[convention=LOGICAL, distribution=ANY]

3) Then, during physical planning we need to resolve and propagate
distributions, and enforce [convention=PHYSICAL, distribution=SINGLETON],
possibly injecting exchange operators in between. In this specific case, no
exchanges are needed, because scans are always REPLICATED, projection
inherits scan distribution, and for the root node is ok to use REPLICATED
input instead of SINGLETON, since the first one satisfies the latter. The
expected physical tree is:
RootPhysicalRel[convention=PHYSICAL, distribution=SINGLETON]
-> ProjectPhysicalRel[convention=PHYSICAL, distribution=REPLICATED]
  -> MapScanPhysicalRel[convention=PHYSICAL, distribution=REPLICATED]

4) There are three physical rules for each operator, which are organized as
follows:
4.1) RootPhysicalRule [4] - converts RootLogicalRel node, enforcing
SINGLETON distribution on its input. As a result, an appropriate exchange
is injected later from DistributionTraitDef [3]. As I said, in this
specific query exchange is not needed
4.2) MapScanPhysicalRule [5] - converts MapScanLogicalRel. At this point,
we resolve input distribution.
4.3) ProjectPhysicalRule [6] - transforms logical project to physical
project *ONLY* if there is an underlying physical input with REPLICATED or
SINGLETON distribution.

5) Original case fails because the rules are executed as follows:
5.1) RootPhysicalRule - do transform
5.2) ProjectPhysicalRule - do nothing at this point
5.3) MapScanPhysicalRule - do transform
I would like ProjectPhysicalRule to be re-fired again, but this doesn't
happen.

There are two workarounds for this, and both concern me.

*Workaround 1*: set "canConvertConvention" to "true". In this case, rules
are executed in the way I would expect. But I observe too many rule
invocations and unnecessary trait conversions because enabling this flag
leads to matching every rel in a set with every other. For example, Volcano
tries to convert SINGLETON -> REPLICATED, while it is never actually needed
by my engine. And this is how rules are executed:
1) RootPhysicalRule - expected
2) RootPhysicalRule - fired again, because I created a converted node on
the previous step
3) ProjectPhysicalRule - expected
4) ProjectPhysicalRule - fired again for the same reason as p.2
5) MapScanPhysicalRule - expected
6) ProjectPhysicalRule - parent rule is called as I would expect - this is
where the missing physical project appear
7) RootPhysicalRule - called because its child was transformed
+ ExpandConversionRule is called multiple times, and sometimes it creates
nodes, which are otherwise completely unnecessary. Such as the
aforementioned SINGLETON -> REPLICATED conversion.
So in the end, it works as I would expect even for much more complex plans.
But produces a litter which I cannot control

Resulting plan:
RootPhysicalRel.PHYSICAL.SINGLETON.[](input=ProjectPhysicalRel#91)
  ProjectPhysicalRel.PHYSICAL.REPLICATED.[](input=MapScanPhysicalRel#81,
...))
    MapScanPhysicalRel.PHYSICAL.REPLICATED.[](table=[t],projects=[0])

*Workaround 2*: add exchanges for ANY distribution and produce
transformation with ANY distribution in the project rule. This is what
Drill does. But it also produces a litter. This time these are unnecessary
exchanges; we pessimistically create them when input is ANY, and if it
eventually resolved to any compatible distribution, the exchange stays
there. As far as I understand, Drill's ExcessiveExchangeIdentifier removes
that stuff after planning separately.

Resulting plan:
RootPhysicalRel.PHYSICAL.SINGLETON.[](input=UnicastExchangePhysicalRel#30)

UnicastExchangePhysicalRel.PHYSICAL.SINGLETON.[](input=ProjectPhysicalRel#29,hashFields=[0])
// Not needed!
    ProjectPhysicalRel.PHYSICAL.ANY.[](input=MapScanPhysicalRel#25,...))
      MapScanPhysicalRel.PHYSICAL.REPLICATED.[](table=[t],projects=[0])

So as you can see, it is possible to force Calcite to do what I need in
this specific case - to re-execute the rules of parent nodes after the
child has changed. But this workarounds creates some unnecessary
intermediate nodes. With complex queries, this may increase planning time
significantly.

My main question is - what key part of converters logic or Calcite
framework I am missing? Why I cannot force the Calcite to call parent rules
out of the box, but converters seem to do that at the cost of additional
litter?

Thank you very much in advance.
Vladimir.

[1]
https://github.com/devozerov/calcite-optimizer/blob/master/src/test/java/devozerov/OptimizerTest.java
[2]
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/distribution/DistributionTrait.java
[3]
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/distribution/DistributionTraitDef.java
[4]
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/RootPhysicalRule.java

[5]
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/MapScanPhysicalRule.java
[6]
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/physical/ProjectPhysicalRule.java
[7]
https://github.com/devozerov/calcite-optimizer/blob/master/src/main/java/devozerov/HazelcastConventions.java#L38

вт, 29 окт. 2019 г. в 11:26, Vladimir Ozerov <pp...@gmail.com>:

> Hi everybody,
>
> First of all, thank you for answers and suggestions. Let me address them
> briefly:
> 1) I use two conventions at the moment, LOGICAL and PHYSICAL. I agree with
> you that this might be overkill, and there is a chance that in the final
> solution we may end up with only one. But meanwhile having this separation
> seems handy, because on the first stage I enforce the optimizer to
> propagate NONE -> LOGICAL conversions. Then I have a clean logical tree
> *before* any physical distribution stuff is involved, which I can use for
> internal post-processing before going logical. I would propose to keep it
> out of scope at the moment. Let's just consider that the goal is to convert
> from one convention to another.
> 2) Operands with "any" matchers are already used
> 3) The reason why I would like to avoid the "Project" rule fire on the
> first run, is that it doesn't enforce any distribution on its child
> ("Scan"). Instead, it needs to derive the distribution from the scan.
>
> To make the problem more clear, let me prepare a simple reproducer for the
> issue.
>
> Regards,
> Vladimir.
>
> вт, 29 окт. 2019 г. в 10:01, Seliverstov Igor <gv...@gmail.com>:
>
>> Vladimir,
>>
>> I guess Project rule doesn't have a child matcher. Put into it child "any"
>> match rule and it will apply on each child node transformation.
>>
>> Regards,
>> Igor
>>
>>
>> вт, 29 окт. 2019 г., 7:07 Danny Chan <yu...@gmail.com>:
>>
>> > Vladimir, all you need to do is to change the convention of the root
>> node,
>> > the volcano would propagate the convention to all its input nodes when
>> > registering them to the planner. You can take this code [1] for
>> reference :)
>> >
>> > [1]
>> >
>> https://github.com/apache/calcite/blob/1ef2821695ca6e10fbad7b8efe7246c4a20143af/core/src/main/java/org/apache/calcite/tools/Programs.java#L324
>> >
>> > Best,
>> > Danny Chan
>> > 在 2019年10月29日 +0800 AM5:24,dev@calcite.apache.org,写道:
>> > >
>> > > n of the scan.
>> >
>>
>

Re: Problem with converters and possibly rule matching

Posted by Vladimir Ozerov <pp...@gmail.com>.
Hi everybody,

First of all, thank you for answers and suggestions. Let me address them
briefly:
1) I use two conventions at the moment, LOGICAL and PHYSICAL. I agree with
you that this might be overkill, and there is a chance that in the final
solution we may end up with only one. But meanwhile having this separation
seems handy, because on the first stage I enforce the optimizer to
propagate NONE -> LOGICAL conversions. Then I have a clean logical tree
*before* any physical distribution stuff is involved, which I can use for
internal post-processing before going logical. I would propose to keep it
out of scope at the moment. Let's just consider that the goal is to convert
from one convention to another.
2) Operands with "any" matchers are already used
3) The reason why I would like to avoid the "Project" rule fire on the
first run, is that it doesn't enforce any distribution on its child
("Scan"). Instead, it needs to derive the distribution from the scan.

To make the problem more clear, let me prepare a simple reproducer for the
issue.

Regards,
Vladimir.

вт, 29 окт. 2019 г. в 10:01, Seliverstov Igor <gv...@gmail.com>:

> Vladimir,
>
> I guess Project rule doesn't have a child matcher. Put into it child "any"
> match rule and it will apply on each child node transformation.
>
> Regards,
> Igor
>
>
> вт, 29 окт. 2019 г., 7:07 Danny Chan <yu...@gmail.com>:
>
> > Vladimir, all you need to do is to change the convention of the root
> node,
> > the volcano would propagate the convention to all its input nodes when
> > registering them to the planner. You can take this code [1] for
> reference :)
> >
> > [1]
> >
> https://github.com/apache/calcite/blob/1ef2821695ca6e10fbad7b8efe7246c4a20143af/core/src/main/java/org/apache/calcite/tools/Programs.java#L324
> >
> > Best,
> > Danny Chan
> > 在 2019年10月29日 +0800 AM5:24,dev@calcite.apache.org,写道:
> > >
> > > n of the scan.
> >
>

Re: Problem with converters and possibly rule matching

Posted by Seliverstov Igor <gv...@gmail.com>.
Vladimir,

I guess Project rule doesn't have a child matcher. Put into it child "any"
match rule and it will apply on each child node transformation.

Regards,
Igor


вт, 29 окт. 2019 г., 7:07 Danny Chan <yu...@gmail.com>:

> Vladimir, all you need to do is to change the convention of the root node,
> the volcano would propagate the convention to all its input nodes when
> registering them to the planner. You can take this code [1] for reference :)
>
> [1]
> https://github.com/apache/calcite/blob/1ef2821695ca6e10fbad7b8efe7246c4a20143af/core/src/main/java/org/apache/calcite/tools/Programs.java#L324
>
> Best,
> Danny Chan
> 在 2019年10月29日 +0800 AM5:24,dev@calcite.apache.org,写道:
> >
> > n of the scan.
>

Re: Problem with converters and possibly rule matching

Posted by Danny Chan <yu...@gmail.com>.
Vladimir, all you need to do is to change the convention of the root node, the volcano would propagate the convention to all its input nodes when registering them to the planner. You can take this code [1] for reference :)

[1] https://github.com/apache/calcite/blob/1ef2821695ca6e10fbad7b8efe7246c4a20143af/core/src/main/java/org/apache/calcite/tools/Programs.java#L324

Best,
Danny Chan
在 2019年10月29日 +0800 AM5:24,dev@calcite.apache.org,写道:
>
> n of the scan.

Re: Re: Problem with converters and possibly rule matching

Posted by Haisheng Yuan <h....@alibaba-inc.com>.
Why do you need 2 conventions? I think 1 "Hazelcast" is enough. Usually logical operators are ones with NONE convention.

Regarding with physical->logical, you can modify the rule instance to match logical operators or NONE convention only.

- Haisheng

------------------------------------------------------------------
发件人:Stamatis Zampetakis<za...@gmail.com>
日 期:2019年10月29日 06:10:45
收件人:<de...@calcite.apache.org>
主 题:Re: Problem with converters and possibly rule matching

Hi Vladimir,

You can get an idea of how the Volcano planner works by reading [1].
The implementation of the Volcano in Calcite has many differences but the
main ideas are there.

Normally you do not need to set canConvertConvention to true; especially in
your case it doesn't seem necessary.

I think the main problem with your approach is that the Project rule does
not produce any transformation when it is first invoked.
The Volcano planner works mainly in a top-down fashion so rules tend to
match from top to bottom.
For example, in [1] if for a particular operator there is no algorithm,
transformation rule, or enforcer to apply at a given step the planning
process would stop and you wouldn't even see the transformation of the scan.

When the project rule matches you should transform it to the
HZPhysicalProject and then require that the child operator has certain
traits (the distribution that is necessary).
The rule should look like the EnumerableProjectRule [2] but instead of
requiring only the convention you should pass also the requirement for the
distribution of the scan.

Best,
Stamatis

[1]
https://www.cse.iitb.ac.in/infolab/Data/Courses/CS632/Papers/Volcano-graefe.pdf
[2]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableProjectRule.java



On Mon, Oct 28, 2019 at 10:24 PM Vladimir Ozerov <pp...@gmail.com> wrote:

> Hi colleagues,
>
> We are building a Calcite-based optimizer for Hazelcast, and I have some
> problems understanding Calcite's logic with respect to converters. Let me
> briefly explain the problem.
>
> We have an execution backend, so we do not need Bindable or Enumerable.
> Instead, we would like to use Calcite to convert original SQL to a tree
> with our own convention, then convert it to our internal representation,
> and finally, execute.
>
> We started with looking at other Calcite integrations and eventually came
> to a classical two-phase optimization approach. We have two internal
> conventions - LOGICAL and PHYSICAL. The goal is to optimize the tree as
> follows:
> 1) NONE -> LOGICAL - heuristical optimizations
> 2) LOGICAL -> PHYSICAL - cost-based planning
>
> Suppose that after the first phase I have the following tree of our own
> operators:
> HZLogicalRoot
> -> HZLogicalProject
>   -> HZLogicalScan
>
> For this specific case, there is not much to optimize, so we only need to
> transition to physical nodes and do some boilerplate with traits
> propagation:
> HZPhysicalRoot
> -> HZPhysicalProject
>   -> HZPhysicalScan
>
> In order to achieve this, I define three rules, which just do a conversion
> of relevant nodes. Volcano optimizer is used.
>
> Now, the problem - somehow it works only when I override
> Convention.Impl.canConvertConvention to true for our PHYSICAL convention,
> but that blows the search space and the same rules are called many times. A
> lot of time is spent on endless PHYSICAL -> LOGICAL conversions, which are
> of no use.
>
> If I change canConvertConvention to false, then rules are called a sensible
> number of times, but cannot produce a complete PHYSICAL tree. Here is how
> it works:
> 1) "Root" rule is invoked, which converts "HZLogicalRoot" to
> "HZPhysicalRoot"
> 2) "Project" rule is invoked, but do not produce any transformations, since
> it needs Scan distribution, which is not known yet. This desired behavior
> at this point.
> 3) "Scan" rule is invoked, "HZLogicalScan" is converted to
> "HZPhysicalScan". Distribution is resolved
> 4) At this point, we have [LogicalRoot, PhysicalRoot] -> [LogicalProject]
> -> [LogicalScan, PhysicalScan] sets . I expect that since new scan was
> installed, the "Project" rule will be fired again. This time we know the
> distribution, so the transformation is possible. But the rule is not called
> and we fail with an error.
>
> So my questions are:
> 1) What is the real role of converters in this process? For some reason,
> when unnecessary (from a logical standpoint) PHYSICAL -> LOGICAL conversion
> is allowed, even complex plans could be built. And Drill does it for some
> reason. But it costs multiple additional invocations of the same rules. Are
> there any docs or presentations explaining the mechanics behind?
> 2) What are the minimum requirements, that will allow a rule on the parent
> to be fired again after it's child node has changed?
>
> I can provide any additional information, source code or even working
> example of this problem if needed. I don't want to bother you with it at
> the moment, because it feels like I miss something very simple.
>
> Would appreciate your help.
>
> Regards,
> Vladimir.
>


Re: Problem with converters and possibly rule matching

Posted by Stamatis Zampetakis <za...@gmail.com>.
Hi Vladimir,

You can get an idea of how the Volcano planner works by reading [1].
The implementation of the Volcano in Calcite has many differences but the
main ideas are there.

Normally you do not need to set canConvertConvention to true; especially in
your case it doesn't seem necessary.

I think the main problem with your approach is that the Project rule does
not produce any transformation when it is first invoked.
The Volcano planner works mainly in a top-down fashion so rules tend to
match from top to bottom.
For example, in [1] if for a particular operator there is no algorithm,
transformation rule, or enforcer to apply at a given step the planning
process would stop and you wouldn't even see the transformation of the scan.

When the project rule matches you should transform it to the
HZPhysicalProject and then require that the child operator has certain
traits (the distribution that is necessary).
The rule should look like the EnumerableProjectRule [2] but instead of
requiring only the convention you should pass also the requirement for the
distribution of the scan.

Best,
Stamatis

[1]
https://www.cse.iitb.ac.in/infolab/Data/Courses/CS632/Papers/Volcano-graefe.pdf
[2]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableProjectRule.java



On Mon, Oct 28, 2019 at 10:24 PM Vladimir Ozerov <pp...@gmail.com> wrote:

> Hi colleagues,
>
> We are building a Calcite-based optimizer for Hazelcast, and I have some
> problems understanding Calcite's logic with respect to converters. Let me
> briefly explain the problem.
>
> We have an execution backend, so we do not need Bindable or Enumerable.
> Instead, we would like to use Calcite to convert original SQL to a tree
> with our own convention, then convert it to our internal representation,
> and finally, execute.
>
> We started with looking at other Calcite integrations and eventually came
> to a classical two-phase optimization approach. We have two internal
> conventions - LOGICAL and PHYSICAL. The goal is to optimize the tree as
> follows:
> 1) NONE -> LOGICAL - heuristical optimizations
> 2) LOGICAL -> PHYSICAL - cost-based planning
>
> Suppose that after the first phase I have the following tree of our own
> operators:
> HZLogicalRoot
> -> HZLogicalProject
>   -> HZLogicalScan
>
> For this specific case, there is not much to optimize, so we only need to
> transition to physical nodes and do some boilerplate with traits
> propagation:
> HZPhysicalRoot
> -> HZPhysicalProject
>   -> HZPhysicalScan
>
> In order to achieve this, I define three rules, which just do a conversion
> of relevant nodes. Volcano optimizer is used.
>
> Now, the problem - somehow it works only when I override
> Convention.Impl.canConvertConvention to true for our PHYSICAL convention,
> but that blows the search space and the same rules are called many times. A
> lot of time is spent on endless PHYSICAL -> LOGICAL conversions, which are
> of no use.
>
> If I change canConvertConvention to false, then rules are called a sensible
> number of times, but cannot produce a complete PHYSICAL tree. Here is how
> it works:
> 1) "Root" rule is invoked, which converts "HZLogicalRoot" to
> "HZPhysicalRoot"
> 2) "Project" rule is invoked, but do not produce any transformations, since
> it needs Scan distribution, which is not known yet. This desired behavior
> at this point.
> 3) "Scan" rule is invoked, "HZLogicalScan" is converted to
> "HZPhysicalScan". Distribution is resolved
> 4) At this point, we have [LogicalRoot, PhysicalRoot] -> [LogicalProject]
> -> [LogicalScan, PhysicalScan] sets . I expect that since new scan was
> installed, the "Project" rule will be fired again. This time we know the
> distribution, so the transformation is possible. But the rule is not called
> and we fail with an error.
>
> So my questions are:
> 1) What is the real role of converters in this process? For some reason,
> when unnecessary (from a logical standpoint) PHYSICAL -> LOGICAL conversion
> is allowed, even complex plans could be built. And Drill does it for some
> reason. But it costs multiple additional invocations of the same rules. Are
> there any docs or presentations explaining the mechanics behind?
> 2) What are the minimum requirements, that will allow a rule on the parent
> to be fired again after it's child node has changed?
>
> I can provide any additional information, source code or even working
> example of this problem if needed. I don't want to bother you with it at
> the moment, because it feels like I miss something very simple.
>
> Would appreciate your help.
>
> Regards,
> Vladimir.
>