Lateral views and CBO

[Stephen Carlin <sc...@cloudera.com.INVALID>]

Hi,

I noticed recently that for most lateral views, we do not convert to a CBO
plan.  I was hoping to make some changes to make this possible.

I was wondering if anyone out there had any thoughts on how to do this.

If not, I did have one in mind and wanted to bring it up just to start the
conversation before I did any work on it.

I was thinking maybe we could have our own RelNode deriving off of
LogicalCorrelate.  I know we already handle correlated queries and
decorrelate them.  But in this special case, I think we should not
decorrelate the RelNode.  We should leave this new LogicalCorrelate node in
through the whole optimization process and then put in code to translate
the RelNode into the physical plan.

Thoughts?

Thanks!

Re: Lateral views and CBO

[Stamatis Zampetakis <za...@gmail.com>]

Hi Steve,

The way that we currently represent lateral views on the physical plan is
not great. I'm sure there are good reasons of why people went forward with
the approach of introducing specialized operators for that purpose but as
the history has shown the current representation is causing us lots of
problems in many parts of the compiler where we need to traverse the plan
DAG (rules + hooks + explain) leading to exponential visit of the plan
nodes.

If we decorrelate the plan as usual maybe we could get rid of the problems
mentioned above and the need for specialized lateral view physical
operators which I think would be a good thing in the long run.


Best,
Stamatis

On Thu, Apr 6, 2023, 6:31 PM Stephen Carlin <sc...@cloudera.com.invalid>
wrote:

> Hi,
>
> I noticed recently that for most lateral views, we do not convert to a CBO
> plan.  I was hoping to make some changes to make this possible.
>
> I was wondering if anyone out there had any thoughts on how to do this.
>
> If not, I did have one in mind and wanted to bring it up just to start the
> conversation before I did any work on it.
>
> I was thinking maybe we could have our own RelNode deriving off of
> LogicalCorrelate.  I know we already handle correlated queries and
> decorrelate them.  But in this special case, I think we should not
> decorrelate the RelNode.  We should leave this new LogicalCorrelate node in
> through the whole optimization process and then put in code to translate
> the RelNode into the physical plan.
>
> Thoughts?
>
> Thanks!
>