You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Austin Richardson <au...@teampicnic.com.INVALID> on 2024/09/30 09:54:59 UTC
Serialisation of optimised query plans
Hi Calcite devs,
Our team is currently experimenting with ideas for serialising and
deserialising optimised query plans using RelJsonWriter and RelJsonReader.
We noticed this is possible for pre-optimisation logical plans, but ran
into issues when trying to do so for post-optimisation execution plans. For
example, LogicalJoin implements the required constructor for serialisation
here
<https://calcite.apache.org/javadocAggregate/org/apache/calcite/rel/logical/LogicalJoin.html#%3Cinit%3E(org.apache.calcite.rel.RelInput)>,
but the specific JOIN implementations (e.g. EnumerableBatchNestedLoopJoin)
do not.
Considering this, is there a preferred way for serialising and
deserialising optimised query plans? We wanted to ensure we aren’t missing
something here.
Thank you,
Austin
Re: Serialisation of optimised query plans
Posted by Austin Richardson <au...@teampicnic.com.INVALID>.
Excellent point Julian, and thank you for the swift reply. We'll keep
tinkering in this space to see what we can find.
Thank you,
Austin
On Fri, 4 Oct 2024 at 00:47, Julian Hyde <jh...@apache.org> wrote:
> No, that would not be acceptable. EnumerableInterpretable#getBindable
> is a package-protected method with no javadoc or tests. If you "relax"
> the visibility modifier it becomes a public API and Calcite committers
> are responsible for maintaining its behavior in perpetuity.
>
> You need to come up with a better proposal that is a net benefit to
> Calcite.
>
>
> On Thu, Oct 3, 2024 at 5:54 AM Austin Richardson
> <au...@teampicnic.com.invalid> wrote:
> >
> > Hi Stamatis,
> >
> > Thank you for the reply.
> >
> > For some additional context: we're experimenting with this
> > (de)serialisation because we're interested in storing optimised query
> plans
> > in a distributed cache. Another idea we're exploring is caching the
> > generated source code for those query plans, and then compiling it
> > as-needed later. If possible, we'd like to re-use the Calcite code
> > responsible for this, though we'd need to relax the visibility modifier
> of
> > EnumerableInterpretable#getBindable
> > <
> https://github.com/apache/calcite/blob/b1308feff49c8b747b3bbb52e1519d334bc984ec/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableInterpretable.java#L134
> >
> > from
> > package-private to public so it'd be invokable from our code.
> >
> > Do you have any thoughts on whether that'd be acceptable?
> >
> > Thank you,
> > Austin
> >
> > On Tue, 1 Oct 2024 at 09:10, Stamatis Zampetakis <za...@gmail.com>
> wrote:
> >
> > > Hey Austin,
> > >
> > > Indeed the most popular serialization/deserialization method for query
> > > plans is by using the RelJsonReader and RelJsonWriter. However, we are
> > > lacking a bit of test coverage around this feature so when new
> > > RelNodes are added it's easy to miss the required changes. Moreover,
> > > it's quite common for other projects to plug-in their own custom
> > > RelNode's so serialization happens outside the classes present in
> > > calcite core. We would happily accept contributions in this area so
> > > feel free to create a JIRA ticket and submit a PR.
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Mon, Sep 30, 2024 at 11:55 AM Austin Richardson
> > > <au...@teampicnic.com.invalid> wrote:
> > > >
> > > > Hi Calcite devs,
> > > >
> > > > Our team is currently experimenting with ideas for serialising and
> > > > deserialising optimised query plans using RelJsonWriter and
> > > RelJsonReader.
> > > > We noticed this is possible for pre-optimisation logical plans, but
> ran
> > > > into issues when trying to do so for post-optimisation execution
> plans.
> > > For
> > > > example, LogicalJoin implements the required constructor for
> > > serialisation
> > > > here
> > > > <
> > >
> https://calcite.apache.org/javadocAggregate/org/apache/calcite/rel/logical/LogicalJoin.html#%3Cinit%3E(org.apache.calcite.rel.RelInput)
> > > >,
> > > > but the specific JOIN implementations (e.g.
> > > EnumerableBatchNestedLoopJoin)
> > > > do not.
> > > >
> > > > Considering this, is there a preferred way for serialising and
> > > > deserialising optimised query plans? We wanted to ensure we aren’t
> > > missing
> > > > something here.
> > > >
> > > > Thank you,
> > > >
> > > > Austin
> > >
>
Re: Serialisation of optimised query plans
Posted by Julian Hyde <jh...@apache.org>.
No, that would not be acceptable. EnumerableInterpretable#getBindable
is a package-protected method with no javadoc or tests. If you "relax"
the visibility modifier it becomes a public API and Calcite committers
are responsible for maintaining its behavior in perpetuity.
You need to come up with a better proposal that is a net benefit to Calcite.
On Thu, Oct 3, 2024 at 5:54 AM Austin Richardson
<au...@teampicnic.com.invalid> wrote:
>
> Hi Stamatis,
>
> Thank you for the reply.
>
> For some additional context: we're experimenting with this
> (de)serialisation because we're interested in storing optimised query plans
> in a distributed cache. Another idea we're exploring is caching the
> generated source code for those query plans, and then compiling it
> as-needed later. If possible, we'd like to re-use the Calcite code
> responsible for this, though we'd need to relax the visibility modifier of
> EnumerableInterpretable#getBindable
> <https://github.com/apache/calcite/blob/b1308feff49c8b747b3bbb52e1519d334bc984ec/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableInterpretable.java#L134>
> from
> package-private to public so it'd be invokable from our code.
>
> Do you have any thoughts on whether that'd be acceptable?
>
> Thank you,
> Austin
>
> On Tue, 1 Oct 2024 at 09:10, Stamatis Zampetakis <za...@gmail.com> wrote:
>
> > Hey Austin,
> >
> > Indeed the most popular serialization/deserialization method for query
> > plans is by using the RelJsonReader and RelJsonWriter. However, we are
> > lacking a bit of test coverage around this feature so when new
> > RelNodes are added it's easy to miss the required changes. Moreover,
> > it's quite common for other projects to plug-in their own custom
> > RelNode's so serialization happens outside the classes present in
> > calcite core. We would happily accept contributions in this area so
> > feel free to create a JIRA ticket and submit a PR.
> >
> > Best,
> > Stamatis
> >
> > On Mon, Sep 30, 2024 at 11:55 AM Austin Richardson
> > <au...@teampicnic.com.invalid> wrote:
> > >
> > > Hi Calcite devs,
> > >
> > > Our team is currently experimenting with ideas for serialising and
> > > deserialising optimised query plans using RelJsonWriter and
> > RelJsonReader.
> > > We noticed this is possible for pre-optimisation logical plans, but ran
> > > into issues when trying to do so for post-optimisation execution plans.
> > For
> > > example, LogicalJoin implements the required constructor for
> > serialisation
> > > here
> > > <
> > https://calcite.apache.org/javadocAggregate/org/apache/calcite/rel/logical/LogicalJoin.html#%3Cinit%3E(org.apache.calcite.rel.RelInput)
> > >,
> > > but the specific JOIN implementations (e.g.
> > EnumerableBatchNestedLoopJoin)
> > > do not.
> > >
> > > Considering this, is there a preferred way for serialising and
> > > deserialising optimised query plans? We wanted to ensure we aren’t
> > missing
> > > something here.
> > >
> > > Thank you,
> > >
> > > Austin
> >
Re: Serialisation of optimised query plans
Posted by Austin Richardson <au...@teampicnic.com.INVALID>.
Hi Stamatis,
Thank you for the reply.
For some additional context: we're experimenting with this
(de)serialisation because we're interested in storing optimised query plans
in a distributed cache. Another idea we're exploring is caching the
generated source code for those query plans, and then compiling it
as-needed later. If possible, we'd like to re-use the Calcite code
responsible for this, though we'd need to relax the visibility modifier of
EnumerableInterpretable#getBindable
<https://github.com/apache/calcite/blob/b1308feff49c8b747b3bbb52e1519d334bc984ec/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableInterpretable.java#L134>
from
package-private to public so it'd be invokable from our code.
Do you have any thoughts on whether that'd be acceptable?
Thank you,
Austin
On Tue, 1 Oct 2024 at 09:10, Stamatis Zampetakis <za...@gmail.com> wrote:
> Hey Austin,
>
> Indeed the most popular serialization/deserialization method for query
> plans is by using the RelJsonReader and RelJsonWriter. However, we are
> lacking a bit of test coverage around this feature so when new
> RelNodes are added it's easy to miss the required changes. Moreover,
> it's quite common for other projects to plug-in their own custom
> RelNode's so serialization happens outside the classes present in
> calcite core. We would happily accept contributions in this area so
> feel free to create a JIRA ticket and submit a PR.
>
> Best,
> Stamatis
>
> On Mon, Sep 30, 2024 at 11:55 AM Austin Richardson
> <au...@teampicnic.com.invalid> wrote:
> >
> > Hi Calcite devs,
> >
> > Our team is currently experimenting with ideas for serialising and
> > deserialising optimised query plans using RelJsonWriter and
> RelJsonReader.
> > We noticed this is possible for pre-optimisation logical plans, but ran
> > into issues when trying to do so for post-optimisation execution plans.
> For
> > example, LogicalJoin implements the required constructor for
> serialisation
> > here
> > <
> https://calcite.apache.org/javadocAggregate/org/apache/calcite/rel/logical/LogicalJoin.html#%3Cinit%3E(org.apache.calcite.rel.RelInput)
> >,
> > but the specific JOIN implementations (e.g.
> EnumerableBatchNestedLoopJoin)
> > do not.
> >
> > Considering this, is there a preferred way for serialising and
> > deserialising optimised query plans? We wanted to ensure we aren’t
> missing
> > something here.
> >
> > Thank you,
> >
> > Austin
>
Re: Serialisation of optimised query plans
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hey Austin,
Indeed the most popular serialization/deserialization method for query
plans is by using the RelJsonReader and RelJsonWriter. However, we are
lacking a bit of test coverage around this feature so when new
RelNodes are added it's easy to miss the required changes. Moreover,
it's quite common for other projects to plug-in their own custom
RelNode's so serialization happens outside the classes present in
calcite core. We would happily accept contributions in this area so
feel free to create a JIRA ticket and submit a PR.
Best,
Stamatis
On Mon, Sep 30, 2024 at 11:55 AM Austin Richardson
<au...@teampicnic.com.invalid> wrote:
>
> Hi Calcite devs,
>
> Our team is currently experimenting with ideas for serialising and
> deserialising optimised query plans using RelJsonWriter and RelJsonReader.
> We noticed this is possible for pre-optimisation logical plans, but ran
> into issues when trying to do so for post-optimisation execution plans. For
> example, LogicalJoin implements the required constructor for serialisation
> here
> <https://calcite.apache.org/javadocAggregate/org/apache/calcite/rel/logical/LogicalJoin.html#%3Cinit%3E(org.apache.calcite.rel.RelInput)>,
> but the specific JOIN implementations (e.g. EnumerableBatchNestedLoopJoin)
> do not.
>
> Considering this, is there a preferred way for serialising and
> deserialising optimised query plans? We wanted to ensure we aren’t missing
> something here.
>
> Thank you,
>
> Austin