You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Soumyadeep Mukhopadhyay <so...@gmail.com> on 2023/06/14 09:32:02 UTC
Calcite to Beam mapping, possible?
Hello All,
I have a use case where I need Apache Calcite to parse, validate the query
and then apply some planner rules.
I was hoping to leverage the capabilities of the Apache Beam on top of
Calcite so that I can build on top of what we have already done, and apply
it at scale. One approach that came to mind was to either convert the
RelNode to something equivalent in Beam (like PTransform maybe?
https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.html
).
Any opinion/suggestion on this will be really appreciated! Thanks for your
time and consideration! If you feel this is not intended forum for me to
ask, please let me know! :)
With regards,
Soumyadeep Mukhopadhyay.
Re: Calcite to Beam mapping, possible?
Posted by Byron Ellis via dev <de...@beam.apache.org>.
Not knowing what your custom rules are intended to do my first question
would be “do you need them if you can just write SQL.” My second thought
would be if they implement some part of SQL that SqlTransform doesn’t
perhaps contributing them. My third thought would be looking at the
SqlTransform code and using that as a basis for a CalciteTransform.
On Sat, Jun 17, 2023 at 11:59 PM Soumyadeep Mukhopadhyay <
soumyamyb95@gmail.com> wrote:
> Hey Byron,
>
> Thanks for your response. My apologies for getting back to you so late.
>
> SqlTransform does seem a potent candidate for my use case but I am unsure
> about one thing - what happens to the custom planner rules that we were
> hoping to build? Shall we rely solely on Beam's implementation of the
> planner rules or is there a way to integrate our own rules too?
>
> If answer to both the questions above is no then the only way I see I
> could use Calcite with Beam would be as a query parsing layer (the parser
> being Calcite) and passing on the SqlString.toString() into SqlTransform
> from the resulting SqlNode. Were you approaching from that point of view?
>
> Thanks again for your help!
>
> With regards,
> Soumyadeep Mukhopadhyay.
>
>
>
>
> On Wed, Jun 14, 2023 at 7:47 PM deepak kumar <kd...@gmail.com> wrote:
>
>> I would be really interested to contribute as well on calcite to
>> PTransform kind of conversions
>>
>> On Wed, Jun 14, 2023 at 7:03 PM Byron Ellis via dev <de...@beam.apache.org>
>> wrote:
>>
>>> Any particular reason you're not using the SqlTransform to do that?
>>> That'd probably be a good place to start code-wise (perhaps with something
>>> a bit newer than 2.29) to see how Beam uses Calcite today to produce a
>>> PTransform from a query plan.
>>>
>>> On Wed, Jun 14, 2023 at 5:32 AM Soumyadeep Mukhopadhyay <
>>> soumyamyb95@gmail.com> wrote:
>>>
>>>> Hello All,
>>>>
>>>> I have a use case where I need Apache Calcite to parse, validate the
>>>> query and then apply some planner rules.
>>>>
>>>> I was hoping to leverage the capabilities of the Apache Beam on top of
>>>> Calcite so that I can build on top of what we have already done, and apply
>>>> it at scale. One approach that came to mind was to either convert the
>>>> RelNode to something equivalent in Beam (like PTransform maybe?
>>>> https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.html
>>>> ).
>>>>
>>>> Any opinion/suggestion on this will be really appreciated! Thanks for
>>>> your time and consideration! If you feel this is not intended forum for me
>>>> to ask, please let me know! :)
>>>>
>>>> With regards,
>>>> Soumyadeep Mukhopadhyay.
>>>>
>>>
Re: Calcite to Beam mapping, possible?
Posted by Soumyadeep Mukhopadhyay <so...@gmail.com>.
Hey Byron,
Thanks for your response. My apologies for getting back to you so late.
SqlTransform does seem a potent candidate for my use case but I am unsure
about one thing - what happens to the custom planner rules that we were
hoping to build? Shall we rely solely on Beam's implementation of the
planner rules or is there a way to integrate our own rules too?
If answer to both the questions above is no then the only way I see I could
use Calcite with Beam would be as a query parsing layer (the parser being
Calcite) and passing on the SqlString.toString() into SqlTransform from the
resulting SqlNode. Were you approaching from that point of view?
Thanks again for your help!
With regards,
Soumyadeep Mukhopadhyay.
On Wed, Jun 14, 2023 at 7:47 PM deepak kumar <kd...@gmail.com> wrote:
> I would be really interested to contribute as well on calcite to
> PTransform kind of conversions
>
> On Wed, Jun 14, 2023 at 7:03 PM Byron Ellis via dev <de...@beam.apache.org>
> wrote:
>
>> Any particular reason you're not using the SqlTransform to do that?
>> That'd probably be a good place to start code-wise (perhaps with something
>> a bit newer than 2.29) to see how Beam uses Calcite today to produce a
>> PTransform from a query plan.
>>
>> On Wed, Jun 14, 2023 at 5:32 AM Soumyadeep Mukhopadhyay <
>> soumyamyb95@gmail.com> wrote:
>>
>>> Hello All,
>>>
>>> I have a use case where I need Apache Calcite to parse, validate the
>>> query and then apply some planner rules.
>>>
>>> I was hoping to leverage the capabilities of the Apache Beam on top of
>>> Calcite so that I can build on top of what we have already done, and apply
>>> it at scale. One approach that came to mind was to either convert the
>>> RelNode to something equivalent in Beam (like PTransform maybe?
>>> https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.html
>>> ).
>>>
>>> Any opinion/suggestion on this will be really appreciated! Thanks for
>>> your time and consideration! If you feel this is not intended forum for me
>>> to ask, please let me know! :)
>>>
>>> With regards,
>>> Soumyadeep Mukhopadhyay.
>>>
>>
Re: Calcite to Beam mapping, possible?
Posted by deepak kumar <kd...@gmail.com>.
I would be really interested to contribute as well on calcite to PTransform
kind of conversions
On Wed, Jun 14, 2023 at 7:03 PM Byron Ellis via dev <de...@beam.apache.org>
wrote:
> Any particular reason you're not using the SqlTransform to do that? That'd
> probably be a good place to start code-wise (perhaps with something a bit
> newer than 2.29) to see how Beam uses Calcite today to produce a PTransform
> from a query plan.
>
> On Wed, Jun 14, 2023 at 5:32 AM Soumyadeep Mukhopadhyay <
> soumyamyb95@gmail.com> wrote:
>
>> Hello All,
>>
>> I have a use case where I need Apache Calcite to parse, validate the
>> query and then apply some planner rules.
>>
>> I was hoping to leverage the capabilities of the Apache Beam on top of
>> Calcite so that I can build on top of what we have already done, and apply
>> it at scale. One approach that came to mind was to either convert the
>> RelNode to something equivalent in Beam (like PTransform maybe?
>> https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.html
>> ).
>>
>> Any opinion/suggestion on this will be really appreciated! Thanks for
>> your time and consideration! If you feel this is not intended forum for me
>> to ask, please let me know! :)
>>
>> With regards,
>> Soumyadeep Mukhopadhyay.
>>
>
Re: Calcite to Beam mapping, possible?
Posted by Byron Ellis via dev <de...@beam.apache.org>.
Any particular reason you're not using the SqlTransform to do that? That'd
probably be a good place to start code-wise (perhaps with something a bit
newer than 2.29) to see how Beam uses Calcite today to produce a PTransform
from a query plan.
On Wed, Jun 14, 2023 at 5:32 AM Soumyadeep Mukhopadhyay <
soumyamyb95@gmail.com> wrote:
> Hello All,
>
> I have a use case where I need Apache Calcite to parse, validate the query
> and then apply some planner rules.
>
> I was hoping to leverage the capabilities of the Apache Beam on top of
> Calcite so that I can build on top of what we have already done, and apply
> it at scale. One approach that came to mind was to either convert the
> RelNode to something equivalent in Beam (like PTransform maybe?
> https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.html
> ).
>
> Any opinion/suggestion on this will be really appreciated! Thanks for your
> time and consideration! If you feel this is not intended forum for me to
> ask, please let me know! :)
>
> With regards,
> Soumyadeep Mukhopadhyay.
>