You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Jay Narale <ja...@gmail.com> on 2022/08/09 21:27:37 UTC
Question - Better human readable plan functionality
Hello dev community,
Long calcite plans are very hard to debug sometimes because of our
filter, aggregate expressions involve InputReferences which are hard to
quickly follow down the tree.
eg -> JoinRel(condition=[=($0, $2)], joinType=[inner])
Filter ( condition=[$0 = 100])
AggregateRel(group=[{0}], sun_sales=[SUM($1)], mon_sales=[SUM($2)]
Better Plans ->
JoinRel( condition=[(tableName1.columnName, tableName2.columnName2)] )
...
I can override RelMdExpressionLineage for this use case for RexNodes like
for Joins and Filters but AggregateCalls cannot be handled properly by this
relMd.
Is there any existing functionality that I missed that already has the
functionality for this use case or should we work on a new writer or add a
visitor internally in calcite to transform this plan into a better human
readable plan. There can also be a few more operators than need this
"readable transformation"
Warm Regards,
Jay Narale
[1]
https://github.com/apache/calcite/blob/7e0057e8de93930f1b2952a1cbcee8ad7a6bfb4b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdExpressionLineage.java
Re: Question - Better human readable plan functionality
Posted by Jay Narale <ja...@gmail.com>.
Two ways that I can think of
1) To reroute explain in AbstractRelNode[1] to
a new function explainVerboseTerms
which would fall back to the explainTerms if unavailable but not sure we
would want to go down that path.
2) Add a new writer which switches on RelNode types and tries to update the
info
stored in valueList but this is not a principled approach.
[1]
https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/main/java/org/apache/calcite/rel/AbstractRelNode.java#L251
On Wed, Aug 10, 2022 at 2:33 PM Jay Narale <ja...@gmail.com> wrote:
> Hi Chunwei,
> Thanks for confirming, I will check if it's easily doable in calcite.
>
>
>
> On Tue, Aug 9, 2022 at 7:27 PM Chunwei Lei <ch...@gmail.com> wrote:
>
>> Hi, Jay.
>>
>> As far as I know, there is no existing functionality that can do this job.
>> We have the same requirement, especially when having to check whether
>> the plan is correct. So I'm more than glad to have this feature.
>>
>> P.S. Apache Flink has done the job for its physical plan.
>>
>> Best,
>> Chunwei
>>
>>
>> On Wed, Aug 10, 2022 at 5:28 AM Jay Narale <ja...@gmail.com>
>> wrote:
>>
>> > Hello dev community,
>> > Long calcite plans are very hard to debug sometimes because of our
>> > filter, aggregate expressions involve InputReferences which are hard to
>> > quickly follow down the tree.
>> >
>> > eg -> JoinRel(condition=[=($0, $2)], joinType=[inner])
>> > Filter ( condition=[$0 = 100])
>> > AggregateRel(group=[{0}], sun_sales=[SUM($1)], mon_sales=[SUM($2)]
>> >
>> > Better Plans ->
>> > JoinRel( condition=[(tableName1.columnName, tableName2.columnName2)] )
>> > ...
>> >
>> > I can override RelMdExpressionLineage for this use case for RexNodes
>> like
>> > for Joins and Filters but AggregateCalls cannot be handled properly by
>> this
>> > relMd.
>> > Is there any existing functionality that I missed that already has
>> the
>> > functionality for this use case or should we work on a new writer or
>> add a
>> > visitor internally in calcite to transform this plan into a better human
>> > readable plan. There can also be a few more operators than need this
>> > "readable transformation"
>> >
>> > Warm Regards,
>> >
>> > Jay Narale
>> >
>> > [1]
>> >
>> >
>> https://github.com/apache/calcite/blob/7e0057e8de93930f1b2952a1cbcee8ad7a6bfb4b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdExpressionLineage.java
>> >
>>
>
>
> --
> Warm Regards,
>
> Jay Narale
>
--
Warm Regards,
Jay Narale
Re: Question - Better human readable plan functionality
Posted by Jay Narale <ja...@gmail.com>.
Hi Chunwei,
Thanks for confirming, I will check if it's easily doable in calcite.
On Tue, Aug 9, 2022 at 7:27 PM Chunwei Lei <ch...@gmail.com> wrote:
> Hi, Jay.
>
> As far as I know, there is no existing functionality that can do this job.
> We have the same requirement, especially when having to check whether
> the plan is correct. So I'm more than glad to have this feature.
>
> P.S. Apache Flink has done the job for its physical plan.
>
> Best,
> Chunwei
>
>
> On Wed, Aug 10, 2022 at 5:28 AM Jay Narale <ja...@gmail.com>
> wrote:
>
> > Hello dev community,
> > Long calcite plans are very hard to debug sometimes because of our
> > filter, aggregate expressions involve InputReferences which are hard to
> > quickly follow down the tree.
> >
> > eg -> JoinRel(condition=[=($0, $2)], joinType=[inner])
> > Filter ( condition=[$0 = 100])
> > AggregateRel(group=[{0}], sun_sales=[SUM($1)], mon_sales=[SUM($2)]
> >
> > Better Plans ->
> > JoinRel( condition=[(tableName1.columnName, tableName2.columnName2)] )
> > ...
> >
> > I can override RelMdExpressionLineage for this use case for RexNodes like
> > for Joins and Filters but AggregateCalls cannot be handled properly by
> this
> > relMd.
> > Is there any existing functionality that I missed that already has
> the
> > functionality for this use case or should we work on a new writer or add
> a
> > visitor internally in calcite to transform this plan into a better human
> > readable plan. There can also be a few more operators than need this
> > "readable transformation"
> >
> > Warm Regards,
> >
> > Jay Narale
> >
> > [1]
> >
> >
> https://github.com/apache/calcite/blob/7e0057e8de93930f1b2952a1cbcee8ad7a6bfb4b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdExpressionLineage.java
> >
>
--
Warm Regards,
Jay Narale
Re: Question - Better human readable plan functionality
Posted by Chunwei Lei <ch...@gmail.com>.
Hi, Jay.
As far as I know, there is no existing functionality that can do this job.
We have the same requirement, especially when having to check whether
the plan is correct. So I'm more than glad to have this feature.
P.S. Apache Flink has done the job for its physical plan.
Best,
Chunwei
On Wed, Aug 10, 2022 at 5:28 AM Jay Narale <ja...@gmail.com> wrote:
> Hello dev community,
> Long calcite plans are very hard to debug sometimes because of our
> filter, aggregate expressions involve InputReferences which are hard to
> quickly follow down the tree.
>
> eg -> JoinRel(condition=[=($0, $2)], joinType=[inner])
> Filter ( condition=[$0 = 100])
> AggregateRel(group=[{0}], sun_sales=[SUM($1)], mon_sales=[SUM($2)]
>
> Better Plans ->
> JoinRel( condition=[(tableName1.columnName, tableName2.columnName2)] )
> ...
>
> I can override RelMdExpressionLineage for this use case for RexNodes like
> for Joins and Filters but AggregateCalls cannot be handled properly by this
> relMd.
> Is there any existing functionality that I missed that already has the
> functionality for this use case or should we work on a new writer or add a
> visitor internally in calcite to transform this plan into a better human
> readable plan. There can also be a few more operators than need this
> "readable transformation"
>
> Warm Regards,
>
> Jay Narale
>
> [1]
>
> https://github.com/apache/calcite/blob/7e0057e8de93930f1b2952a1cbcee8ad7a6bfb4b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdExpressionLineage.java
>