You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Pranav Deshpande <de...@gmail.com> on 2022/09/16 18:41:03 UTC
Requesting Information Regarding Enumerable
Dear Apache Calcite Team,
I am trying to modify some parts of the RelNode Tree in Calcite for my own
custom logic. For this, I am extending existing nodes (eg.
EnumerableAggregate etc.) and then also extending the respective rule (eg.
EnumerableAggregateRule) and overriding the respective functions in both so
that my custom node is used in the tree (logical -> custom node which is
basically an extended enumerable) instead of logical -> enumerable.
Some of the rules here (eg. EnumerableLimit) have references to Immutable
classes which are package private and I hence I am unable to extend these
(EnumerableLimitRule is itself one such example:
https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableLimitRule.java#L72
)
Any advice on how I can solve this problem?
Thanks & Regards,
Pranav
Re: Requesting Information Regarding Enumerable
Posted by Ruben Q L <ru...@gmail.com>.
Hello Pranav,
Inside YourRelMdNonCumulativeCost class, you'd need to create one method
per operator (for those whose RelOptCost you want to "override").
You can take a look at the code of the existing RelMd* classes in Calcite
to have an idea [1].
Once you have your metadata provider, i.e. something like:
RelMetadataProvider YOUR_INSTANCE =
ChainedRelMetadataProvider.of(ImmutableList.of(YourRelMdNonCumulativeCost.INSTANCE,
DefaultRelMetadataProvider.INSTANCE));
Then you just need to use it in the RelOptCluster:
RelOptCluster cluster = ...
cluster.setMetadataProvider(YOUR_INSTANCE);
Best,
Ruben
[1]
https://github.com/apache/calcite/tree/main/core/src/main/java/org/apache/calcite/rel/metadata
On Wed, Sep 21, 2022 at 4:26 PM Pranav Deshpande <
deshpande.v.pranav@gmail.com> wrote:
> Hi Ruben,
> Thank you very much for your reply!
>
> Will I have to create a different Metadata Handler per operator (eg.
> Project, Correlate etc.) or can I make a common one?
>
> Also, I am having some trouble finding where to put this statement:
>
>
> ChainedRelMetadataProvider.of(ImmutableList.of(YourRelMdNonCumulativeCost.INSTANCE,
> DefaultRelMetadataProvider.INSTANCE));
>
> Could you please help me out by pointing me to a code example?
>
> I viewed this tutorial: https://www.youtube.com/watch?v=p1O3E33FIs8, but
> nowhere in the query processing code do I see where to put this (it is
> mentioned in the architecture though).
>
> Thanks & Regards,
> Pranav
>
> On Sat, Sep 17, 2022 at 6:07 AM Ruben Q L <ru...@gmail.com> wrote:
>
> > Hello Pranav,
> >
> > If you just need to "override" the cost function of certain operators,
> > there is no need to create new operators (and rules), you can do that by
> > extending the Metadata.
> > Let's say you want to override the cost of the Correlate, you should be
> > able to do that by creating your own class:
> >
> > public final class YourRelMdNonCumulativeCost implements
> > MetadataHandler<NonCumulativeCost>
> > {
> > static final RelMetadataProvider INSTANCE =
> > ReflectiveRelMetadataProvider.reflectiveSource(new
> > YourRelMdNonCumulativeCost(),
> > BuiltInMetadata.NonCumulativeCost.Handler.class);
> >
> > private YourRelMdNonCumulativeCost() { }
> >
> > @Override
> > public MetadataDef<NonCumulativeCost> getDef() { return
> > NonCumulativeCost.DEF; }
> >
> > public RelOptCost getNonCumulativeCost(Correlate correlate,
> > RelMetadataQuery mq)
> > {
> > // your code here...
> > }
> > }
> >
> > And then as MetadataProvider, instead of using just Calcite's default
> one,
> > you would use a chain of yours + Calcite's default:
> >
> >
> >
> ChainedRelMetadataProvider.of(ImmutableList.of(YourRelMdNonCumulativeCost.INSTANCE,
> > DefaultRelMetadataProvider.INSTANCE));
> >
> >
> > Having said that, if you really need to create your own operators, the
> > easiest way, as you mention, is to create your operator (which would
> extend
> > the Calcite operator) and your rule (which could extend the Calcite rule,
> > but not necessarily). As you say, it is not possible to use the
> > Immutable*Rule Config, but you can use the rule's default Config or
> create
> > your own default Config. You can even do all that without Immutables in
> > your project, this thread might be of interest [1].
> >
> > Best,
> > Ruben
> >
> > [1] https://lists.apache.org/thread/2h0mcdqs4q1psgjy9sxlk8wxzo3w1zq9
> >
> >
> >
> > On Fri, Sep 16, 2022 at 10:10 PM Pranav Deshpande <
> > deshpande.v.pranav@gmail.com> wrote:
> >
> > > Hi Julian/Apache Calcite Dev Team,
> > > Thank you very much for your reply
> > > I see that's a good point and it makes a lot of sense. I think I should
> > > exactly tell you what my problem is: I am only trying to override the
> > cost
> > > function so that I can group operators in a certain way in the
> Relational
> > > Tree.
> > >
> > > That is the only functionality that is required (for my example) right
> > now.
> > > I was extending the rules and the nodes to override this 1 function
> only.
> > > Is there another way to do this? If not, what is the recommended
> approach
> > > (is it to duplicate the enumerable rels and rules)?
> > >
> > > Please let me know.
> > >
> > > Thanks & Regards,
> > > Pranav
> > >
> > > On Fri, Sep 16, 2022 at 3:13 PM Julian Hyde <jh...@gmail.com>
> > > wrote:
> > >
> > > > We don’t necessary want you to extend existing classes. If we change
> > the
> > > > base class in future, your code breaks, and you complain that we have
> > > > broken semantic versioning. Making things private is, in that sense,
> a
> > > > feature.
> > > >
> > > > If what you are doing is a feature that would benefit other Calcite
> > > users,
> > > > you should propose that feature.
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On Sep 16, 2022, at 11:41 AM, Pranav Deshpande <
> > > > deshpande.v.pranav@gmail.com> wrote:
> > > > >
> > > > > Dear Apache Calcite Team,
> > > > > I am trying to modify some parts of the RelNode Tree in Calcite for
> > my
> > > > own
> > > > > custom logic. For this, I am extending existing nodes (eg.
> > > > > EnumerableAggregate etc.) and then also extending the respective
> rule
> > > > (eg.
> > > > > EnumerableAggregateRule) and overriding the respective functions in
> > > both
> > > > so
> > > > > that my custom node is used in the tree (logical -> custom node
> which
> > > is
> > > > > basically an extended enumerable) instead of logical -> enumerable.
> > > > >
> > > > > Some of the rules here (eg. EnumerableLimit) have references to
> > > Immutable
> > > > > classes which are package private and I hence I am unable to extend
> > > these
> > > > > (EnumerableLimitRule is itself one such example:
> > > > >
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableLimitRule.java#L72
> > > > > )
> > > > >
> > > > > Any advice on how I can solve this problem?
> > > > >
> > > > > Thanks & Regards,
> > > > > Pranav
> > > >
> > > >
> > >
> >
>
Re: Requesting Information Regarding Enumerable
Posted by Pranav Deshpande <de...@gmail.com>.
Hi Ruben,
Thank you very much for your reply!
Will I have to create a different Metadata Handler per operator (eg.
Project, Correlate etc.) or can I make a common one?
Also, I am having some trouble finding where to put this statement:
ChainedRelMetadataProvider.of(ImmutableList.of(YourRelMdNonCumulativeCost.INSTANCE,
DefaultRelMetadataProvider.INSTANCE));
Could you please help me out by pointing me to a code example?
I viewed this tutorial: https://www.youtube.com/watch?v=p1O3E33FIs8, but
nowhere in the query processing code do I see where to put this (it is
mentioned in the architecture though).
Thanks & Regards,
Pranav
On Sat, Sep 17, 2022 at 6:07 AM Ruben Q L <ru...@gmail.com> wrote:
> Hello Pranav,
>
> If you just need to "override" the cost function of certain operators,
> there is no need to create new operators (and rules), you can do that by
> extending the Metadata.
> Let's say you want to override the cost of the Correlate, you should be
> able to do that by creating your own class:
>
> public final class YourRelMdNonCumulativeCost implements
> MetadataHandler<NonCumulativeCost>
> {
> static final RelMetadataProvider INSTANCE =
> ReflectiveRelMetadataProvider.reflectiveSource(new
> YourRelMdNonCumulativeCost(),
> BuiltInMetadata.NonCumulativeCost.Handler.class);
>
> private YourRelMdNonCumulativeCost() { }
>
> @Override
> public MetadataDef<NonCumulativeCost> getDef() { return
> NonCumulativeCost.DEF; }
>
> public RelOptCost getNonCumulativeCost(Correlate correlate,
> RelMetadataQuery mq)
> {
> // your code here...
> }
> }
>
> And then as MetadataProvider, instead of using just Calcite's default one,
> you would use a chain of yours + Calcite's default:
>
>
> ChainedRelMetadataProvider.of(ImmutableList.of(YourRelMdNonCumulativeCost.INSTANCE,
> DefaultRelMetadataProvider.INSTANCE));
>
>
> Having said that, if you really need to create your own operators, the
> easiest way, as you mention, is to create your operator (which would extend
> the Calcite operator) and your rule (which could extend the Calcite rule,
> but not necessarily). As you say, it is not possible to use the
> Immutable*Rule Config, but you can use the rule's default Config or create
> your own default Config. You can even do all that without Immutables in
> your project, this thread might be of interest [1].
>
> Best,
> Ruben
>
> [1] https://lists.apache.org/thread/2h0mcdqs4q1psgjy9sxlk8wxzo3w1zq9
>
>
>
> On Fri, Sep 16, 2022 at 10:10 PM Pranav Deshpande <
> deshpande.v.pranav@gmail.com> wrote:
>
> > Hi Julian/Apache Calcite Dev Team,
> > Thank you very much for your reply
> > I see that's a good point and it makes a lot of sense. I think I should
> > exactly tell you what my problem is: I am only trying to override the
> cost
> > function so that I can group operators in a certain way in the Relational
> > Tree.
> >
> > That is the only functionality that is required (for my example) right
> now.
> > I was extending the rules and the nodes to override this 1 function only.
> > Is there another way to do this? If not, what is the recommended approach
> > (is it to duplicate the enumerable rels and rules)?
> >
> > Please let me know.
> >
> > Thanks & Regards,
> > Pranav
> >
> > On Fri, Sep 16, 2022 at 3:13 PM Julian Hyde <jh...@gmail.com>
> > wrote:
> >
> > > We don’t necessary want you to extend existing classes. If we change
> the
> > > base class in future, your code breaks, and you complain that we have
> > > broken semantic versioning. Making things private is, in that sense, a
> > > feature.
> > >
> > > If what you are doing is a feature that would benefit other Calcite
> > users,
> > > you should propose that feature.
> > >
> > > Julian
> > >
> > >
> > > > On Sep 16, 2022, at 11:41 AM, Pranav Deshpande <
> > > deshpande.v.pranav@gmail.com> wrote:
> > > >
> > > > Dear Apache Calcite Team,
> > > > I am trying to modify some parts of the RelNode Tree in Calcite for
> my
> > > own
> > > > custom logic. For this, I am extending existing nodes (eg.
> > > > EnumerableAggregate etc.) and then also extending the respective rule
> > > (eg.
> > > > EnumerableAggregateRule) and overriding the respective functions in
> > both
> > > so
> > > > that my custom node is used in the tree (logical -> custom node which
> > is
> > > > basically an extended enumerable) instead of logical -> enumerable.
> > > >
> > > > Some of the rules here (eg. EnumerableLimit) have references to
> > Immutable
> > > > classes which are package private and I hence I am unable to extend
> > these
> > > > (EnumerableLimitRule is itself one such example:
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableLimitRule.java#L72
> > > > )
> > > >
> > > > Any advice on how I can solve this problem?
> > > >
> > > > Thanks & Regards,
> > > > Pranav
> > >
> > >
> >
>
Re: Requesting Information Regarding Enumerable
Posted by Ruben Q L <ru...@gmail.com>.
Hello Pranav,
If you just need to "override" the cost function of certain operators,
there is no need to create new operators (and rules), you can do that by
extending the Metadata.
Let's say you want to override the cost of the Correlate, you should be
able to do that by creating your own class:
public final class YourRelMdNonCumulativeCost implements
MetadataHandler<NonCumulativeCost>
{
static final RelMetadataProvider INSTANCE =
ReflectiveRelMetadataProvider.reflectiveSource(new
YourRelMdNonCumulativeCost(),
BuiltInMetadata.NonCumulativeCost.Handler.class);
private YourRelMdNonCumulativeCost() { }
@Override
public MetadataDef<NonCumulativeCost> getDef() { return
NonCumulativeCost.DEF; }
public RelOptCost getNonCumulativeCost(Correlate correlate,
RelMetadataQuery mq)
{
// your code here...
}
}
And then as MetadataProvider, instead of using just Calcite's default one,
you would use a chain of yours + Calcite's default:
ChainedRelMetadataProvider.of(ImmutableList.of(YourRelMdNonCumulativeCost.INSTANCE,
DefaultRelMetadataProvider.INSTANCE));
Having said that, if you really need to create your own operators, the
easiest way, as you mention, is to create your operator (which would extend
the Calcite operator) and your rule (which could extend the Calcite rule,
but not necessarily). As you say, it is not possible to use the
Immutable*Rule Config, but you can use the rule's default Config or create
your own default Config. You can even do all that without Immutables in
your project, this thread might be of interest [1].
Best,
Ruben
[1] https://lists.apache.org/thread/2h0mcdqs4q1psgjy9sxlk8wxzo3w1zq9
On Fri, Sep 16, 2022 at 10:10 PM Pranav Deshpande <
deshpande.v.pranav@gmail.com> wrote:
> Hi Julian/Apache Calcite Dev Team,
> Thank you very much for your reply
> I see that's a good point and it makes a lot of sense. I think I should
> exactly tell you what my problem is: I am only trying to override the cost
> function so that I can group operators in a certain way in the Relational
> Tree.
>
> That is the only functionality that is required (for my example) right now.
> I was extending the rules and the nodes to override this 1 function only.
> Is there another way to do this? If not, what is the recommended approach
> (is it to duplicate the enumerable rels and rules)?
>
> Please let me know.
>
> Thanks & Regards,
> Pranav
>
> On Fri, Sep 16, 2022 at 3:13 PM Julian Hyde <jh...@gmail.com>
> wrote:
>
> > We don’t necessary want you to extend existing classes. If we change the
> > base class in future, your code breaks, and you complain that we have
> > broken semantic versioning. Making things private is, in that sense, a
> > feature.
> >
> > If what you are doing is a feature that would benefit other Calcite
> users,
> > you should propose that feature.
> >
> > Julian
> >
> >
> > > On Sep 16, 2022, at 11:41 AM, Pranav Deshpande <
> > deshpande.v.pranav@gmail.com> wrote:
> > >
> > > Dear Apache Calcite Team,
> > > I am trying to modify some parts of the RelNode Tree in Calcite for my
> > own
> > > custom logic. For this, I am extending existing nodes (eg.
> > > EnumerableAggregate etc.) and then also extending the respective rule
> > (eg.
> > > EnumerableAggregateRule) and overriding the respective functions in
> both
> > so
> > > that my custom node is used in the tree (logical -> custom node which
> is
> > > basically an extended enumerable) instead of logical -> enumerable.
> > >
> > > Some of the rules here (eg. EnumerableLimit) have references to
> Immutable
> > > classes which are package private and I hence I am unable to extend
> these
> > > (EnumerableLimitRule is itself one such example:
> > >
> >
> https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableLimitRule.java#L72
> > > )
> > >
> > > Any advice on how I can solve this problem?
> > >
> > > Thanks & Regards,
> > > Pranav
> >
> >
>
Re: Requesting Information Regarding Enumerable
Posted by Pranav Deshpande <de...@gmail.com>.
Hi Julian/Apache Calcite Dev Team,
Thank you very much for your reply
I see that's a good point and it makes a lot of sense. I think I should
exactly tell you what my problem is: I am only trying to override the cost
function so that I can group operators in a certain way in the Relational
Tree.
That is the only functionality that is required (for my example) right now.
I was extending the rules and the nodes to override this 1 function only.
Is there another way to do this? If not, what is the recommended approach
(is it to duplicate the enumerable rels and rules)?
Please let me know.
Thanks & Regards,
Pranav
On Fri, Sep 16, 2022 at 3:13 PM Julian Hyde <jh...@gmail.com> wrote:
> We don’t necessary want you to extend existing classes. If we change the
> base class in future, your code breaks, and you complain that we have
> broken semantic versioning. Making things private is, in that sense, a
> feature.
>
> If what you are doing is a feature that would benefit other Calcite users,
> you should propose that feature.
>
> Julian
>
>
> > On Sep 16, 2022, at 11:41 AM, Pranav Deshpande <
> deshpande.v.pranav@gmail.com> wrote:
> >
> > Dear Apache Calcite Team,
> > I am trying to modify some parts of the RelNode Tree in Calcite for my
> own
> > custom logic. For this, I am extending existing nodes (eg.
> > EnumerableAggregate etc.) and then also extending the respective rule
> (eg.
> > EnumerableAggregateRule) and overriding the respective functions in both
> so
> > that my custom node is used in the tree (logical -> custom node which is
> > basically an extended enumerable) instead of logical -> enumerable.
> >
> > Some of the rules here (eg. EnumerableLimit) have references to Immutable
> > classes which are package private and I hence I am unable to extend these
> > (EnumerableLimitRule is itself one such example:
> >
> https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableLimitRule.java#L72
> > )
> >
> > Any advice on how I can solve this problem?
> >
> > Thanks & Regards,
> > Pranav
>
>
Re: Requesting Information Regarding Enumerable
Posted by Julian Hyde <jh...@gmail.com>.
We don’t necessary want you to extend existing classes. If we change the base class in future, your code breaks, and you complain that we have broken semantic versioning. Making things private is, in that sense, a feature.
If what you are doing is a feature that would benefit other Calcite users, you should propose that feature.
Julian
> On Sep 16, 2022, at 11:41 AM, Pranav Deshpande <de...@gmail.com> wrote:
>
> Dear Apache Calcite Team,
> I am trying to modify some parts of the RelNode Tree in Calcite for my own
> custom logic. For this, I am extending existing nodes (eg.
> EnumerableAggregate etc.) and then also extending the respective rule (eg.
> EnumerableAggregateRule) and overriding the respective functions in both so
> that my custom node is used in the tree (logical -> custom node which is
> basically an extended enumerable) instead of logical -> enumerable.
>
> Some of the rules here (eg. EnumerableLimit) have references to Immutable
> classes which are package private and I hence I am unable to extend these
> (EnumerableLimitRule is itself one such example:
> https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableLimitRule.java#L72
> )
>
> Any advice on how I can solve this problem?
>
> Thanks & Regards,
> Pranav