You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "dominik.werner.groeninger@student.uni-augsburg.de" <do...@student.uni-augsburg.de> on 2019/09/16 16:16:27 UTC
Extending Flink's SQL-Parser
Hey there,
I have to extend Flink's SQL-parser such that it accepts and evaluates
select-queries with different syntax.
Furthermore I use Eclipse Oxygen with Maven plugin and Flink Release 1.8.
0.
What I believe to know:
For parsing SQL-queries Flink-Table uses Apache Calcite's SQL-parser.
Flink-Table-Planner is the only module that references the Calcite-Core
which contains the parser ("Parser.jj" ?).
Therefore I want to import Flink-Table-Planner and Calcite-Core as local
projects in Eclipse and edit the files "config.fmpp" and "parserImpls.ftl".
After that I want to create a new "Parser.jj" file with Apache Freemaker (I
assume there are some tutorials?).
What I don't know:
Is it a promising plan or are there better strategies to extend the parser?
I already tried to import Flink-Table-Planner but I got many errors which
might refer to a Scala-problem with Eclipse. Do I have to switch to say
IntelliJ? Furthermore I'm not yet clear about how exactly I can extend the
parser. Are there any manuals/tutorials to teach me adding a new
SELECT-syntax? I already came across the parser extension test but it
didn't give me the answers I was looking for.
Thanks for your help!
Regards,
Dominik Gröninger
Re: Extending Flink's SQL-Parser
Posted by Rong Rong <wa...@gmail.com>.
Hi Dominik,
To add to Rui's answer. there are other examples I can think of on how to
extend Calcite's DDL syntax is already in Calcite's Server module [1] and
one of our open-sourced project [2]. you might want to check them out.
--
Rong
[1]
https://github.com/apache/calcite/blob/master/server/src/main/codegen/includes/parserImpls.ftl
[2]
https://github.com/uber/AthenaX/blob/master/athenax-vm-compiler/src/codegen/includes/parserImpls.ftl
On Mon, Sep 16, 2019 at 8:28 PM Rui Li <li...@gmail.com> wrote:
> Hi Dominik,
>
> I think you can check "parserImpls.ftl" to find out how Flink extends
> Calcite's original syntax to support features like CREATE TABLE and DROP
> TABLE, and follow those examples to implement your own syntax. It may also
> be helpful to check the pom.xml of flink-sql-parser to see how we use
> javacc plugin to generate the parser code.
> At the moment I don't think there's any tutorials about extending the SQL
> parser because it's quite internal to Flink. But perhaps the following
> answer provides some insights about how to extend Calcite parser in
> general:
> https://stackoverflow.com/questions/44382826/how-to-change-calcites-default-sql-grammar
>
> On Tue, Sep 17, 2019 at 12:16 AM
> dominik.werner.groeninger@student.uni-augsburg.de <
> dominik.werner.groeninger@student.uni-augsburg.de> wrote:
>
>> Hey there,
>>
>>
>>
>> I have to extend Flink's SQL-parser such that it accepts and evaluates
>> select-queries with different syntax.
>>
>> Furthermore I use Eclipse Oxygen with Maven plugin and Flink Release 1.8.
>> 0.
>>
>>
>>
>> What I believe to know:
>>
>> For parsing SQL-queries Flink-Table uses Apache Calcite's SQL-parser.
>> Flink-Table-Planner is the only module that references the Calcite-Core
>> which contains the parser ("Parser.jj" ?).
>>
>> Therefore I want to import Flink-Table-Planner and Calcite-Core as local
>> projects in Eclipse and edit the files "config.fmpp" and "parserImpls.ftl".
>> After that I want to create a new "Parser.jj" file with Apache Freemaker (I
>> assume there are some tutorials?).
>>
>>
>>
>> What I don't know:
>>
>> Is it a promising plan or are there better strategies to extend the
>> parser?
>>
>> I already tried to import Flink-Table-Planner but I got many errors which
>> might refer to a Scala-problem with Eclipse. Do I have to switch to say
>> IntelliJ? Furthermore I'm not yet clear about how exactly I can extend the
>> parser. Are there any manuals/tutorials to teach me adding a new
>> SELECT-syntax? I already came across the parser extension test but it
>> didn't give me the answers I was looking for.
>>
>>
>>
>> Thanks for your help!
>>
>>
>>
>> Regards,
>>
>> Dominik Gröninger
>>
>>
>>
>>
>
>
> --
> Best regards!
> Rui Li
>
Re: Extending Flink's SQL-Parser
Posted by Rui Li <li...@gmail.com>.
Hi Dominik,
I think you can check "parserImpls.ftl" to find out how Flink extends
Calcite's original syntax to support features like CREATE TABLE and DROP
TABLE, and follow those examples to implement your own syntax. It may also
be helpful to check the pom.xml of flink-sql-parser to see how we use
javacc plugin to generate the parser code.
At the moment I don't think there's any tutorials about extending the SQL
parser because it's quite internal to Flink. But perhaps the following
answer provides some insights about how to extend Calcite parser in
general:
https://stackoverflow.com/questions/44382826/how-to-change-calcites-default-sql-grammar
On Tue, Sep 17, 2019 at 12:16 AM
dominik.werner.groeninger@student.uni-augsburg.de <
dominik.werner.groeninger@student.uni-augsburg.de> wrote:
> Hey there,
>
>
>
> I have to extend Flink's SQL-parser such that it accepts and evaluates
> select-queries with different syntax.
>
> Furthermore I use Eclipse Oxygen with Maven plugin and Flink Release 1.8.
> 0.
>
>
>
> What I believe to know:
>
> For parsing SQL-queries Flink-Table uses Apache Calcite's SQL-parser.
> Flink-Table-Planner is the only module that references the Calcite-Core
> which contains the parser ("Parser.jj" ?).
>
> Therefore I want to import Flink-Table-Planner and Calcite-Core as local
> projects in Eclipse and edit the files "config.fmpp" and "parserImpls.ftl".
> After that I want to create a new "Parser.jj" file with Apache Freemaker (I
> assume there are some tutorials?).
>
>
>
> What I don't know:
>
> Is it a promising plan or are there better strategies to extend the parser?
>
> I already tried to import Flink-Table-Planner but I got many errors which
> might refer to a Scala-problem with Eclipse. Do I have to switch to say
> IntelliJ? Furthermore I'm not yet clear about how exactly I can extend the
> parser. Are there any manuals/tutorials to teach me adding a new
> SELECT-syntax? I already came across the parser extension test but it
> didn't give me the answers I was looking for.
>
>
>
> Thanks for your help!
>
>
>
> Regards,
>
> Dominik Gröninger
>
>
>
>
--
Best regards!
Rui Li