Share intermediate data between nodes during plan execution

[Vladislav Matyushevski <vl...@gmail.com>]

Greetings Calcite team!

I`m using the Calcite + Avatica and curious to know:
Is there a way how I can share an Object (e.g. Map) between left and right
parts of join?
Sharing is needed due to the fact that the left part is producing data that
must be re-used in the right part.
I`m thinking of the following solution:
There is a getProvider() method, which returns me CalciteJDBC41Connection.
And if I have my own implementation of this class then I`ll be able to
extend it and enrich the class with methods I need. However, it
doesn't feel like a good idea. I believe there might be a proper solution
for this.


Thanks in advance,
Vladislav

Re: Share intermediate data between nodes during plan execution

[Dan Zou <zo...@163.com>]

Hi Vladislav,
Could you please provide more details? Do you want to reuse part of RelNode trees?

Best,
Dan Zou   





> 2023年5月24日 06:44，Vladislav Matyushevski <vl...@gmail.com> 写道：
> 
> Vladislav

Re: Share intermediate data between nodes during plan execution

[Stamatis Zampetakis <za...@gmail.com>]

Hey Vladislav,

I see two operators relevant to your use-case:
* Spool [1]
* Correlate [2]

Both allow some form of propagating data from one side of the join to
the other. Check the code and relevant discussions to see how they are
used.

Best,
Stamatis

[1] https://github.com/apache/calcite/blob/b0420947ddeb2c9e18afe896dbf7adaf49a715a2/core/src/main/java/org/apache/calcite/rel/core/Spool.java
[2] https://github.com/apache/calcite/blob/b0420947ddeb2c9e18afe896dbf7adaf49a715a2/core/src/main/java/org/apache/calcite/rel/core/Correlate.java

On Wed, May 24, 2023 at 8:32 AM Roman Kondakov
<ko...@mail.ru.invalid> wrote:
>
> Hi Vladislav,
>
> Can you share more details regarding your use case?
>
> If you want to share the same object between join inputs, you can use
> RelShuttleImpl [1] to traverse the RelNode tree and every time you are
> visiting a Join relation you need to put the object to the stack and
> remove it from there when you are leaving the Join. In this case the
> shared object will be available on the top of the stack for both sides
> of the Join.
>
> [1]
> https://github.com/apache/calcite/blob/b0420947ddeb2c9e18afe896dbf7adaf49a715a2/core/src/main/java/org/apache/calcite/rel/RelShuttleImpl.java#L49
>
> Thanks,
>
> Roman
>
> On 24.05.2023 05:44, Vladislav Matyushevski wrote:
> > Greetings Calcite team!
> >
> > I`m using the Calcite + Avatica and curious to know:
> > Is there a way how I can share an Object (e.g. Map) between left and right
> > parts of join?
> > Sharing is needed due to the fact that the left part is producing data that
> > must be re-used in the right part.
> > I`m thinking of the following solution:
> > There is a getProvider() method, which returns me CalciteJDBC41Connection.
> > And if I have my own implementation of this class then I`ll be able to
> > extend it and enrich the class with methods I need. However, it
> > doesn't feel like a good idea. I believe there might be a proper solution
> > for this.
> >
> >
> > Thanks in advance,
> > Vladislav
> >

Re: Share intermediate data between nodes during plan execution

[Roman Kondakov <ko...@mail.ru.INVALID>]

Hi Vladislav,

Can you share more details regarding your use case?

If you want to share the same object between join inputs, you can use 
RelShuttleImpl [1] to traverse the RelNode tree and every time you are 
visiting a Join relation you need to put the object to the stack and 
remove it from there when you are leaving the Join. In this case the 
shared object will be available on the top of the stack for both sides 
of the Join.

[1] 
https://github.com/apache/calcite/blob/b0420947ddeb2c9e18afe896dbf7adaf49a715a2/core/src/main/java/org/apache/calcite/rel/RelShuttleImpl.java#L49

Thanks,

Roman

On 24.05.2023 05:44, Vladislav Matyushevski wrote:
> Greetings Calcite team!
>
> I`m using the Calcite + Avatica and curious to know:
> Is there a way how I can share an Object (e.g. Map) between left and right
> parts of join?
> Sharing is needed due to the fact that the left part is producing data that
> must be re-used in the right part.
> I`m thinking of the following solution:
> There is a getProvider() method, which returns me CalciteJDBC41Connection.
> And if I have my own implementation of this class then I`ll be able to
> extend it and enrich the class with methods I need. However, it
> doesn't feel like a good idea. I believe there might be a proper solution
> for this.
>
>
> Thanks in advance,
> Vladislav
>