You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jark Wu <im...@gmail.com> on 2019/03/21 04:21:05 UTC
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Hi Timo,
I'm +1 on the proposal. I like the idea to provide a Java DSL which is more
friendly than string-based approach in programming.
My concern is if/when we can drop the string-based expression parser. If it
takes a very long time, we have to paid more development
cost on the three Table APIs. As far as I know, the string-based API is
used in many companies.
We should also get some feedbacks from users. So I'm CCing this email to
user mailing list.
Best,
Jark
On Wed, 20 Mar 2019 at 08:51, Rong Rong <wa...@gmail.com> wrote:
> Thanks for sharing the initiative of improving Java side Table expression
> DSL.
>
> I agree as in the doc stated that Java DSL was always a "3rd class citizen"
> and we've run into many hand holding scenarios with our Flink developers
> trying to get the Stringify syntax working.
> Overall I am a +1 on this, it also help reduce the development cost of the
> Table API so that we no longer need to maintain different DSL and
> documentations.
>
> I left a few comments in the doc. and also some features that I think will
> be beneficial to the final outcome. Please kindly take a look @Timo.
>
> Many thanks,
> Rong
>
> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther <tw...@apache.org> wrote:
>
> > Hi everyone,
> >
> > some of you might have already noticed the JIRA issue that I opened
> > recently [1] about introducing a proper Java expression DSL for the
> > Table API. Instead of using string-based expressions, we should aim for
> > a unified, maintainable, programmatic Java DSL.
> >
> > Some background: The Blink merging efforts and the big refactorings as
> > part of FLIP-32 have revealed many shortcomings in the current Table &
> > SQL API design. Most of these legacy issues cause problems nowadays in
> > making the Table API a first-class API next to the DataStream API. An
> > example is the ExpressionParser class[2]. It was implemented in the
> > early days of the Table API using Scala parser combinators. During the
> > last years, this parser caused many JIRA issues and user confusion on
> > the mailing list. Because the exceptions and syntax might not be
> > straight forward.
> >
> > For FLINK-11908, we added a temporary bridge instead of reimplementing
> > the parser in Java for FLIP-32. However, this is only a intermediate
> > solution until we made a final decision.
> >
> > I would like to propose a new, parser-free version of the Java Table API:
> >
> >
> >
> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
> >
> > I already implemented an early protoype that shows that such a DSL is
> > not much implementation effort and integrates nicely with all existing
> > API methods.
> >
> > What do you think?
> >
> > Thanks for your feedback,
> >
> > Timo
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-11890
> >
> > [2]
> >
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
> >
> >
>
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Posted by Timo Walther <tw...@apache.org>.
Hi all,
thanks for the great feedback! I see a clear majority for a new Java DSL.
I agree with Dawid that we should aim for 1.9. Compared to other
efforts, this is a relatively small change ;-)
I will further develop my protoype and come back to you once I have a
production-ready solution.
Regards,
Timo
Am 26.03.19 um 11:06 schrieb Elias Saalmann:
> Hi all,
>
> I'm investigating using Table-API as an abstraction for Batch-API in
> our GRADOOP project [1]. As a user of Flink I like to give you some
> feedback from the user perspective.
>
> I really like Timo's ideas - since we built another framework which is
> based on Apache Flink we are just providing another API for our users.
> Using java string literals (or building boolean expressions via string
> builder, for instance) in our code isn't convenient. Therefore I built
> little helpers for our special use cases:
>
> - A builder [2] to build a single Expression, e.g.
>
> ExpressionBuilder builder = new ExpressionBuilder();
> Expression e = builder
> .field("foo")
> .as("bar")
> .toExpression();
>
> - A builder [3] to build a Seq<Expression>, e.g.
>
> ExpressionSeqBuilder builder = new ExpressionSeqBuilder();
> Seq<Expression> seq = builder
> .field("foo")
> .as("a")
> .field("bar")
> .as("b")
> .buildSeq()
>
> Using those builders we make use of those "select(Expression)" API
> methods. Of course, we would appreciate a DSL provided by Flink!
>
> Best,
> Elias
>
> [1] https://dbs.uni-leipzig.de/en/research/projects/gradoop
>
> [2]
> https://github.com/lordon/gradoop/blob/table-api/gradoop-flink/src/main/java/org/gradoop/flink/model/impl/layouts/table/util/ExpressionBuilder.java
>
> [3]
> https://github.com/lordon/gradoop/blob/table-api/gradoop-flink/src/main/java/org/gradoop/flink/model/impl/layouts/table/util/ExpressionSeqBuilder.java
>
> Am 26.03.19 um 10:22 schrieb jincheng sun:
>> Thanks for bringing up this DISCUSS Timo!
>>
>> Java Expression DSL is pretty useful for java user. When we have the
>> Java
>> Expression DSL, Java API will become very rich and easy to use!
>>
>> +1 from my side.
>>
>> Best,
>> Jincheng
>>
>>
>> Dawid Wysakowicz <dw...@apache.org> 于2019年3月26日周二
>> 下午5:08写道：
>>
>>> Hi,
>>>
>>> I really like the idea of introducing Java Expression DSL. I think this
>>> will solve many problems e.g. right now it's quite tricky how string
>>> literals work in scala (sometimes it might go through the
>>> ExpressionParser
>>> and it will end up as an UnresolvedFieldReference), another important
>>> problem we could solve with this is the need for unique column names in
>>> tables right now. We could at some point introduce sth like:
>>>
>>> Table table = ...
>>>
>>> table.field("fieldName")
>>>
>>> and etc. A common "entry point" to expressions should simplify a lot.
>>>
>>> Therefore I am strongly +1 for introducing this feature.
>>>
>>> @Jark I think we could aim to introduce the new Java DSL API in 1.9 and
>>> once we do that we could deprecate the string approach.
>>>
>>> Best,
>>>
>>> Dawid
>>> On 22/03/2019 03:36, Jark Wu wrote:
>>>
>>> Hi Timo,
>>>
>>> Sounds good to me.
>>>
>>> Do you want to deprecate the string-based API in 1.9 or make the
>>> decision
>>> in 1.10 after some feedbacks ?
>>>
>>>
>>> On Thu, 21 Mar 2019 at 21:32, Timo Walther <tw...@apache.org> wrote:
>>>
>>>> Thanks for your feedback Rong and Jark.
>>>>
>>>> @Jark: Yes, you are right that the string-based API is used quite a
>>>> lot.
>>>> On the other side, the potential user base in the future is still
>>>> bigger
>>>> than our current user base. Because the Table API will become equally
>>>> important as the DataStream API, we really need to fix some crucial
>>>> design
>>>> decisions before it is too late. I would suggest to introduce the
>>>> new DSL
>>>> in 1.9 and remove the Expression parser either in 1.10 or 1.11. From a
>>>> developement point of view, I think we can handle the overhead to
>>>> maintain
>>>> 3 APIs until then because 2 APIs will share the same code base +
>>>> expression
>>>> parser.
>>>>
>>>> Regards,
>>>> Timo
>>>>
>>>> Am 21.03.19 um 05:21 schrieb Jark Wu:
>>>>
>>>> Hi Timo,
>>>>
>>>> I'm +1 on the proposal. I like the idea to provide a Java DSL which is
>>>> more friendly than string-based approach in programming.
>>>>
>>>> My concern is if/when we can drop the string-based expression
>>>> parser. If
>>>> it takes a very long time, we have to paid more development
>>>> cost on the three Table APIs. As far as I know, the string-based
>>>> API is
>>>> used in many companies.
>>>> We should also get some feedbacks from users. So I'm CCing this
>>>> email to
>>>> user mailing list.
>>>>
>>>> Best,
>>>> Jark
>>>>
>>>>
>>>>
>>>> On Wed, 20 Mar 2019 at 08:51, Rong Rong <wa...@gmail.com> wrote:
>>>>
>>>>> Thanks for sharing the initiative of improving Java side Table
>>>>> expression
>>>>> DSL.
>>>>>
>>>>> I agree as in the doc stated that Java DSL was always a "3rd class
>>>>> citizen"
>>>>> and we've run into many hand holding scenarios with our Flink
>>>>> developers
>>>>> trying to get the Stringify syntax working.
>>>>> Overall I am a +1 on this, it also help reduce the development
>>>>> cost of
>>>>> the
>>>>> Table API so that we no longer need to maintain different DSL and
>>>>> documentations.
>>>>>
>>>>> I left a few comments in the doc. and also some features that I think
>>>>> will
>>>>> be beneficial to the final outcome. Please kindly take a look @Timo.
>>>>>
>>>>> Many thanks,
>>>>> Rong
>>>>>
>>>>> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther <tw...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> some of you might have already noticed the JIRA issue that I opened
>>>>>> recently [1] about introducing a proper Java expression DSL for the
>>>>>> Table API. Instead of using string-based expressions, we should
>>>>>> aim for
>>>>>> a unified, maintainable, programmatic Java DSL.
>>>>>>
>>>>>> Some background: The Blink merging efforts and the big
>>>>>> refactorings as
>>>>>> part of FLIP-32 have revealed many shortcomings in the current
>>>>>> Table &
>>>>>> SQL API design. Most of these legacy issues cause problems
>>>>>> nowadays in
>>>>>> making the Table API a first-class API next to the DataStream
>>>>>> API. An
>>>>>> example is the ExpressionParser class[2]. It was implemented in the
>>>>>> early days of the Table API using Scala parser combinators.
>>>>>> During the
>>>>>> last years, this parser caused many JIRA issues and user
>>>>>> confusion on
>>>>>> the mailing list. Because the exceptions and syntax might not be
>>>>>> straight forward.
>>>>>>
>>>>>> For FLINK-11908, we added a temporary bridge instead of
>>>>>> reimplementing
>>>>>> the parser in Java for FLIP-32. However, this is only a intermediate
>>>>>> solution until we made a final decision.
>>>>>>
>>>>>> I would like to propose a new, parser-free version of the Java Table
>>>>> API:
>>>>>>
>>>>>>
>>>>> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
>>>>>
>>>>>> I already implemented an early protoype that shows that such a
>>>>>> DSL is
>>>>>> not much implementation effort and integrates nicely with all
>>>>>> existing
>>>>>> API methods.
>>>>>>
>>>>>> What do you think?
>>>>>>
>>>>>> Thanks for your feedback,
>>>>>>
>>>>>> Timo
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-11890
>>>>>>
>>>>>> [2]
>>>>>>
>>>>>>
>>>>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
>>>>>
>>>>>>
>>>>
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Posted by Elias Saalmann <es...@studserv.uni-leipzig.de>.
Hi all,
I'm investigating using Table-API as an abstraction for Batch-API in our
GRADOOP project [1]. As a user of Flink I like to give you some feedback
from the user perspective.
I really like Timo's ideas - since we built another framework which is
based on Apache Flink we are just providing another API for our users.
Using java string literals (or building boolean expressions via string
builder, for instance) in our code isn't convenient. Therefore I built
little helpers for our special use cases:
- A builder [2] to build a single Expression, e.g.
ExpressionBuilder builder = new ExpressionBuilder();
Expression e = builder
.field("foo")
.as("bar")
.toExpression();
- A builder [3] to build a Seq<Expression>, e.g.
ExpressionSeqBuilder builder = new ExpressionSeqBuilder();
Seq<Expression> seq = builder
.field("foo")
.as("a")
.field("bar")
.as("b")
.buildSeq()
Using those builders we make use of those "select(Expression)" API
methods. Of course, we would appreciate a DSL provided by Flink!
Best,
Elias
[1] https://dbs.uni-leipzig.de/en/research/projects/gradoop
[2]
https://github.com/lordon/gradoop/blob/table-api/gradoop-flink/src/main/java/org/gradoop/flink/model/impl/layouts/table/util/ExpressionBuilder.java
[3]
https://github.com/lordon/gradoop/blob/table-api/gradoop-flink/src/main/java/org/gradoop/flink/model/impl/layouts/table/util/ExpressionSeqBuilder.java
Am 26.03.19 um 10:22 schrieb jincheng sun:
> Thanks for bringing up this DISCUSS Timo!
>
> Java Expression DSL is pretty useful for java user. When we have the Java
> Expression DSL, Java API will become very rich and easy to use!
>
> +1 from my side.
>
> Best,
> Jincheng
>
>
> Dawid Wysakowicz <dw...@apache.org> 于2019年3月26日周二 下午5:08写道：
>
>> Hi,
>>
>> I really like the idea of introducing Java Expression DSL. I think this
>> will solve many problems e.g. right now it's quite tricky how string
>> literals work in scala (sometimes it might go through the ExpressionParser
>> and it will end up as an UnresolvedFieldReference), another important
>> problem we could solve with this is the need for unique column names in
>> tables right now. We could at some point introduce sth like:
>>
>> Table table = ...
>>
>> table.field("fieldName")
>>
>> and etc. A common "entry point" to expressions should simplify a lot.
>>
>> Therefore I am strongly +1 for introducing this feature.
>>
>> @Jark I think we could aim to introduce the new Java DSL API in 1.9 and
>> once we do that we could deprecate the string approach.
>>
>> Best,
>>
>> Dawid
>> On 22/03/2019 03:36, Jark Wu wrote:
>>
>> Hi Timo,
>>
>> Sounds good to me.
>>
>> Do you want to deprecate the string-based API in 1.9 or make the decision
>> in 1.10 after some feedbacks ?
>>
>>
>> On Thu, 21 Mar 2019 at 21:32, Timo Walther <tw...@apache.org> wrote:
>>
>>> Thanks for your feedback Rong and Jark.
>>>
>>> @Jark: Yes, you are right that the string-based API is used quite a lot.
>>> On the other side, the potential user base in the future is still bigger
>>> than our current user base. Because the Table API will become equally
>>> important as the DataStream API, we really need to fix some crucial design
>>> decisions before it is too late. I would suggest to introduce the new DSL
>>> in 1.9 and remove the Expression parser either in 1.10 or 1.11. From a
>>> developement point of view, I think we can handle the overhead to maintain
>>> 3 APIs until then because 2 APIs will share the same code base + expression
>>> parser.
>>>
>>> Regards,
>>> Timo
>>>
>>> Am 21.03.19 um 05:21 schrieb Jark Wu:
>>>
>>> Hi Timo,
>>>
>>> I'm +1 on the proposal. I like the idea to provide a Java DSL which is
>>> more friendly than string-based approach in programming.
>>>
>>> My concern is if/when we can drop the string-based expression parser. If
>>> it takes a very long time, we have to paid more development
>>> cost on the three Table APIs. As far as I know, the string-based API is
>>> used in many companies.
>>> We should also get some feedbacks from users. So I'm CCing this email to
>>> user mailing list.
>>>
>>> Best,
>>> Jark
>>>
>>>
>>>
>>> On Wed, 20 Mar 2019 at 08:51, Rong Rong <wa...@gmail.com> wrote:
>>>
>>>> Thanks for sharing the initiative of improving Java side Table expression
>>>> DSL.
>>>>
>>>> I agree as in the doc stated that Java DSL was always a "3rd class
>>>> citizen"
>>>> and we've run into many hand holding scenarios with our Flink developers
>>>> trying to get the Stringify syntax working.
>>>> Overall I am a +1 on this, it also help reduce the development cost of
>>>> the
>>>> Table API so that we no longer need to maintain different DSL and
>>>> documentations.
>>>>
>>>> I left a few comments in the doc. and also some features that I think
>>>> will
>>>> be beneficial to the final outcome. Please kindly take a look @Timo.
>>>>
>>>> Many thanks,
>>>> Rong
>>>>
>>>> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther <tw...@apache.org> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> some of you might have already noticed the JIRA issue that I opened
>>>>> recently [1] about introducing a proper Java expression DSL for the
>>>>> Table API. Instead of using string-based expressions, we should aim for
>>>>> a unified, maintainable, programmatic Java DSL.
>>>>>
>>>>> Some background: The Blink merging efforts and the big refactorings as
>>>>> part of FLIP-32 have revealed many shortcomings in the current Table &
>>>>> SQL API design. Most of these legacy issues cause problems nowadays in
>>>>> making the Table API a first-class API next to the DataStream API. An
>>>>> example is the ExpressionParser class[2]. It was implemented in the
>>>>> early days of the Table API using Scala parser combinators. During the
>>>>> last years, this parser caused many JIRA issues and user confusion on
>>>>> the mailing list. Because the exceptions and syntax might not be
>>>>> straight forward.
>>>>>
>>>>> For FLINK-11908, we added a temporary bridge instead of reimplementing
>>>>> the parser in Java for FLIP-32. However, this is only a intermediate
>>>>> solution until we made a final decision.
>>>>>
>>>>> I would like to propose a new, parser-free version of the Java Table
>>>> API:
>>>>>
>>>>>
>>>> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
>>>>> I already implemented an early protoype that shows that such a DSL is
>>>>> not much implementation effort and integrates nicely with all existing
>>>>> API methods.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Thanks for your feedback,
>>>>>
>>>>> Timo
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-11890
>>>>>
>>>>> [2]
>>>>>
>>>>>
>>>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
>>>>>
>>>
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Posted by jincheng sun <su...@gmail.com>.
Thanks for bringing up this DISCUSS Timo!
Java Expression DSL is pretty useful for java user. When we have the Java
Expression DSL, Java API will become very rich and easy to use!
+1 from my side.
Best,
Jincheng
Dawid Wysakowicz <dw...@apache.org> 于2019年3月26日周二 下午5:08写道：
> Hi,
>
> I really like the idea of introducing Java Expression DSL. I think this
> will solve many problems e.g. right now it's quite tricky how string
> literals work in scala (sometimes it might go through the ExpressionParser
> and it will end up as an UnresolvedFieldReference), another important
> problem we could solve with this is the need for unique column names in
> tables right now. We could at some point introduce sth like:
>
> Table table = ...
>
> table.field("fieldName")
>
> and etc. A common "entry point" to expressions should simplify a lot.
>
> Therefore I am strongly +1 for introducing this feature.
>
> @Jark I think we could aim to introduce the new Java DSL API in 1.9 and
> once we do that we could deprecate the string approach.
>
> Best,
>
> Dawid
> On 22/03/2019 03:36, Jark Wu wrote:
>
> Hi Timo,
>
> Sounds good to me.
>
> Do you want to deprecate the string-based API in 1.9 or make the decision
> in 1.10 after some feedbacks ?
>
>
> On Thu, 21 Mar 2019 at 21:32, Timo Walther <tw...@apache.org> wrote:
>
>> Thanks for your feedback Rong and Jark.
>>
>> @Jark: Yes, you are right that the string-based API is used quite a lot.
>> On the other side, the potential user base in the future is still bigger
>> than our current user base. Because the Table API will become equally
>> important as the DataStream API, we really need to fix some crucial design
>> decisions before it is too late. I would suggest to introduce the new DSL
>> in 1.9 and remove the Expression parser either in 1.10 or 1.11. From a
>> developement point of view, I think we can handle the overhead to maintain
>> 3 APIs until then because 2 APIs will share the same code base + expression
>> parser.
>>
>> Regards,
>> Timo
>>
>> Am 21.03.19 um 05:21 schrieb Jark Wu:
>>
>> Hi Timo,
>>
>> I'm +1 on the proposal. I like the idea to provide a Java DSL which is
>> more friendly than string-based approach in programming.
>>
>> My concern is if/when we can drop the string-based expression parser. If
>> it takes a very long time, we have to paid more development
>> cost on the three Table APIs. As far as I know, the string-based API is
>> used in many companies.
>> We should also get some feedbacks from users. So I'm CCing this email to
>> user mailing list.
>>
>> Best,
>> Jark
>>
>>
>>
>> On Wed, 20 Mar 2019 at 08:51, Rong Rong <wa...@gmail.com> wrote:
>>
>>> Thanks for sharing the initiative of improving Java side Table expression
>>> DSL.
>>>
>>> I agree as in the doc stated that Java DSL was always a "3rd class
>>> citizen"
>>> and we've run into many hand holding scenarios with our Flink developers
>>> trying to get the Stringify syntax working.
>>> Overall I am a +1 on this, it also help reduce the development cost of
>>> the
>>> Table API so that we no longer need to maintain different DSL and
>>> documentations.
>>>
>>> I left a few comments in the doc. and also some features that I think
>>> will
>>> be beneficial to the final outcome. Please kindly take a look @Timo.
>>>
>>> Many thanks,
>>> Rong
>>>
>>> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther <tw...@apache.org> wrote:
>>>
>>> > Hi everyone,
>>> >
>>> > some of you might have already noticed the JIRA issue that I opened
>>> > recently [1] about introducing a proper Java expression DSL for the
>>> > Table API. Instead of using string-based expressions, we should aim for
>>> > a unified, maintainable, programmatic Java DSL.
>>> >
>>> > Some background: The Blink merging efforts and the big refactorings as
>>> > part of FLIP-32 have revealed many shortcomings in the current Table &
>>> > SQL API design. Most of these legacy issues cause problems nowadays in
>>> > making the Table API a first-class API next to the DataStream API. An
>>> > example is the ExpressionParser class[2]. It was implemented in the
>>> > early days of the Table API using Scala parser combinators. During the
>>> > last years, this parser caused many JIRA issues and user confusion on
>>> > the mailing list. Because the exceptions and syntax might not be
>>> > straight forward.
>>> >
>>> > For FLINK-11908, we added a temporary bridge instead of reimplementing
>>> > the parser in Java for FLIP-32. However, this is only a intermediate
>>> > solution until we made a final decision.
>>> >
>>> > I would like to propose a new, parser-free version of the Java Table
>>> API:
>>> >
>>> >
>>> >
>>> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
>>> >
>>> > I already implemented an early protoype that shows that such a DSL is
>>> > not much implementation effort and integrates nicely with all existing
>>> > API methods.
>>> >
>>> > What do you think?
>>> >
>>> > Thanks for your feedback,
>>> >
>>> > Timo
>>> >
>>> > [1] https://issues.apache.org/jira/browse/FLINK-11890
>>> >
>>> > [2]
>>> >
>>> >
>>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
>>> >
>>> >
>>>
>>
>>
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Posted by jincheng sun <su...@gmail.com>.
Thanks for bringing up this DISCUSS Timo!
Java Expression DSL is pretty useful for java user. When we have the Java
Expression DSL, Java API will become very rich and easy to use!
+1 from my side.
Best,
Jincheng
Dawid Wysakowicz <dw...@apache.org> 于2019年3月26日周二 下午5:08写道：
> Hi,
>
> I really like the idea of introducing Java Expression DSL. I think this
> will solve many problems e.g. right now it's quite tricky how string
> literals work in scala (sometimes it might go through the ExpressionParser
> and it will end up as an UnresolvedFieldReference), another important
> problem we could solve with this is the need for unique column names in
> tables right now. We could at some point introduce sth like:
>
> Table table = ...
>
> table.field("fieldName")
>
> and etc. A common "entry point" to expressions should simplify a lot.
>
> Therefore I am strongly +1 for introducing this feature.
>
> @Jark I think we could aim to introduce the new Java DSL API in 1.9 and
> once we do that we could deprecate the string approach.
>
> Best,
>
> Dawid
> On 22/03/2019 03:36, Jark Wu wrote:
>
> Hi Timo,
>
> Sounds good to me.
>
> Do you want to deprecate the string-based API in 1.9 or make the decision
> in 1.10 after some feedbacks ?
>
>
> On Thu, 21 Mar 2019 at 21:32, Timo Walther <tw...@apache.org> wrote:
>
>> Thanks for your feedback Rong and Jark.
>>
>> @Jark: Yes, you are right that the string-based API is used quite a lot.
>> On the other side, the potential user base in the future is still bigger
>> than our current user base. Because the Table API will become equally
>> important as the DataStream API, we really need to fix some crucial design
>> decisions before it is too late. I would suggest to introduce the new DSL
>> in 1.9 and remove the Expression parser either in 1.10 or 1.11. From a
>> developement point of view, I think we can handle the overhead to maintain
>> 3 APIs until then because 2 APIs will share the same code base + expression
>> parser.
>>
>> Regards,
>> Timo
>>
>> Am 21.03.19 um 05:21 schrieb Jark Wu:
>>
>> Hi Timo,
>>
>> I'm +1 on the proposal. I like the idea to provide a Java DSL which is
>> more friendly than string-based approach in programming.
>>
>> My concern is if/when we can drop the string-based expression parser. If
>> it takes a very long time, we have to paid more development
>> cost on the three Table APIs. As far as I know, the string-based API is
>> used in many companies.
>> We should also get some feedbacks from users. So I'm CCing this email to
>> user mailing list.
>>
>> Best,
>> Jark
>>
>>
>>
>> On Wed, 20 Mar 2019 at 08:51, Rong Rong <wa...@gmail.com> wrote:
>>
>>> Thanks for sharing the initiative of improving Java side Table expression
>>> DSL.
>>>
>>> I agree as in the doc stated that Java DSL was always a "3rd class
>>> citizen"
>>> and we've run into many hand holding scenarios with our Flink developers
>>> trying to get the Stringify syntax working.
>>> Overall I am a +1 on this, it also help reduce the development cost of
>>> the
>>> Table API so that we no longer need to maintain different DSL and
>>> documentations.
>>>
>>> I left a few comments in the doc. and also some features that I think
>>> will
>>> be beneficial to the final outcome. Please kindly take a look @Timo.
>>>
>>> Many thanks,
>>> Rong
>>>
>>> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther <tw...@apache.org> wrote:
>>>
>>> > Hi everyone,
>>> >
>>> > some of you might have already noticed the JIRA issue that I opened
>>> > recently [1] about introducing a proper Java expression DSL for the
>>> > Table API. Instead of using string-based expressions, we should aim for
>>> > a unified, maintainable, programmatic Java DSL.
>>> >
>>> > Some background: The Blink merging efforts and the big refactorings as
>>> > part of FLIP-32 have revealed many shortcomings in the current Table &
>>> > SQL API design. Most of these legacy issues cause problems nowadays in
>>> > making the Table API a first-class API next to the DataStream API. An
>>> > example is the ExpressionParser class[2]. It was implemented in the
>>> > early days of the Table API using Scala parser combinators. During the
>>> > last years, this parser caused many JIRA issues and user confusion on
>>> > the mailing list. Because the exceptions and syntax might not be
>>> > straight forward.
>>> >
>>> > For FLINK-11908, we added a temporary bridge instead of reimplementing
>>> > the parser in Java for FLIP-32. However, this is only a intermediate
>>> > solution until we made a final decision.
>>> >
>>> > I would like to propose a new, parser-free version of the Java Table
>>> API:
>>> >
>>> >
>>> >
>>> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
>>> >
>>> > I already implemented an early protoype that shows that such a DSL is
>>> > not much implementation effort and integrates nicely with all existing
>>> > API methods.
>>> >
>>> > What do you think?
>>> >
>>> > Thanks for your feedback,
>>> >
>>> > Timo
>>> >
>>> > [1] https://issues.apache.org/jira/browse/FLINK-11890
>>> >
>>> > [2]
>>> >
>>> >
>>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
>>> >
>>> >
>>>
>>
>>
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hi,
I really like the idea of introducing Java Expression DSL. I think this
will solve many problems e.g. right now it's quite tricky how string
literals work in scala (sometimes it might go through the
ExpressionParser and it will end up as an UnresolvedFieldReference),
another important problem we could solve with this is the need for
unique column names in tables right now. We could at some point
introduce sth like:
Table table = ...
table.field("fieldName")
and etc. A common "entry point" to expressions should simplify a lot.
Therefore I am strongly +1 for introducing this feature.
@Jark I think we could aim to introduce the new Java DSL API in 1.9 and
once we do that we could deprecate the string approach.
Best,
Dawid
On 22/03/2019 03:36, Jark Wu wrote:
> Hi Timo,
>
> Sounds good to me.
>
> Do you want to deprecate the string-based API in 1.9 or make the
> decision in 1.10 after some feedbacks ?
>
>
> On Thu, 21 Mar 2019 at 21:32, Timo Walther <twalthr@apache.org
> <ma...@apache.org>> wrote:
>
> Thanks for your feedback Rong and Jark.
>
> @Jark: Yes, you are right that the string-based API is used quite
> a lot. On the other side, the potential user base in the future is
> still bigger than our current user base. Because the Table API
> will become equally important as the DataStream API, we really
> need to fix some crucial design decisions before it is too late. I
> would suggest to introduce the new DSL in 1.9 and remove the
> Expression parser either in 1.10 or 1.11. From a developement
> point of view, I think we can handle the overhead to maintain 3
> APIs until then because 2 APIs will share the same code base +
> expression parser.
>
> Regards,
> Timo
>
> Am 21.03.19 um 05:21 schrieb Jark Wu:
>> Hi Timo,
>>
>> I'm +1 on the proposal. I like the idea to provide a Java DSL
>> which is more friendly than string-based approach in programming.
>>
>> My concern is if/when we can drop the string-based expression
>> parser. If it takes a very long time, we have to paid more
>> development
>> cost on the three Table APIs. As far as I know, the string-based
>> API is used in many companies.
>> We should also get some feedbacks from users. So I'm CCing this
>> email to user mailing list.
>>
>> Best,
>> Jark
>>
>>
>>
>> On Wed, 20 Mar 2019 at 08:51, Rong Rong <walterddr@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Thanks for sharing the initiative of improving Java side
>> Table expression
>> DSL.
>>
>> I agree as in the doc stated that Java DSL was always a "3rd
>> class citizen"
>> and we've run into many hand holding scenarios with our Flink
>> developers
>> trying to get the Stringify syntax working.
>> Overall I am a +1 on this, it also help reduce the
>> development cost of the
>> Table API so that we no longer need to maintain different DSL and
>> documentations.
>>
>> I left a few comments in the doc. and also some features that
>> I think will
>> be beneficial to the final outcome. Please kindly take a look
>> @Timo.
>>
>> Many thanks,
>> Rong
>>
>> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther
>> <twalthr@apache.org <ma...@apache.org>> wrote:
>>
>> > Hi everyone,
>> >
>> > some of you might have already noticed the JIRA issue that
>> I opened
>> > recently [1] about introducing a proper Java expression DSL
>> for the
>> > Table API. Instead of using string-based expressions, we
>> should aim for
>> > a unified, maintainable, programmatic Java DSL.
>> >
>> > Some background: The Blink merging efforts and the big
>> refactorings as
>> > part of FLIP-32 have revealed many shortcomings in the
>> current Table &
>> > SQL API design. Most of these legacy issues cause problems
>> nowadays in
>> > making the Table API a first-class API next to the
>> DataStream API. An
>> > example is the ExpressionParser class[2]. It was
>> implemented in the
>> > early days of the Table API using Scala parser combinators.
>> During the
>> > last years, this parser caused many JIRA issues and user
>> confusion on
>> > the mailing list. Because the exceptions and syntax might
>> not be
>> > straight forward.
>> >
>> > For FLINK-11908, we added a temporary bridge instead of
>> reimplementing
>> > the parser in Java for FLIP-32. However, this is only a
>> intermediate
>> > solution until we made a final decision.
>> >
>> > I would like to propose a new, parser-free version of the
>> Java Table API:
>> >
>> >
>> >
>> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
>> >
>> > I already implemented an early protoype that shows that
>> such a DSL is
>> > not much implementation effort and integrates nicely with
>> all existing
>> > API methods.
>> >
>> > What do you think?
>> >
>> > Thanks for your feedback,
>> >
>> > Timo
>> >
>> > [1] https://issues.apache.org/jira/browse/FLINK-11890
>> >
>> > [2]
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
>> >
>> >
>>
>
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hi,
I really like the idea of introducing Java Expression DSL. I think this
will solve many problems e.g. right now it's quite tricky how string
literals work in scala (sometimes it might go through the
ExpressionParser and it will end up as an UnresolvedFieldReference),
another important problem we could solve with this is the need for
unique column names in tables right now. We could at some point
introduce sth like:
Table table = ...
table.field("fieldName")
and etc. A common "entry point" to expressions should simplify a lot.
Therefore I am strongly +1 for introducing this feature.
@Jark I think we could aim to introduce the new Java DSL API in 1.9 and
once we do that we could deprecate the string approach.
Best,
Dawid
On 22/03/2019 03:36, Jark Wu wrote:
> Hi Timo,
>
> Sounds good to me.
>
> Do you want to deprecate the string-based API in 1.9 or make the
> decision in 1.10 after some feedbacks ?
>
>
> On Thu, 21 Mar 2019 at 21:32, Timo Walther <twalthr@apache.org
> <ma...@apache.org>> wrote:
>
> Thanks for your feedback Rong and Jark.
>
> @Jark: Yes, you are right that the string-based API is used quite
> a lot. On the other side, the potential user base in the future is
> still bigger than our current user base. Because the Table API
> will become equally important as the DataStream API, we really
> need to fix some crucial design decisions before it is too late. I
> would suggest to introduce the new DSL in 1.9 and remove the
> Expression parser either in 1.10 or 1.11. From a developement
> point of view, I think we can handle the overhead to maintain 3
> APIs until then because 2 APIs will share the same code base +
> expression parser.
>
> Regards,
> Timo
>
> Am 21.03.19 um 05:21 schrieb Jark Wu:
>> Hi Timo,
>>
>> I'm +1 on the proposal. I like the idea to provide a Java DSL
>> which is more friendly than string-based approach in programming.
>>
>> My concern is if/when we can drop the string-based expression
>> parser. If it takes a very long time, we have to paid more
>> development
>> cost on the three Table APIs. As far as I know, the string-based
>> API is used in many companies.
>> We should also get some feedbacks from users. So I'm CCing this
>> email to user mailing list.
>>
>> Best,
>> Jark
>>
>>
>>
>> On Wed, 20 Mar 2019 at 08:51, Rong Rong <walterddr@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Thanks for sharing the initiative of improving Java side
>> Table expression
>> DSL.
>>
>> I agree as in the doc stated that Java DSL was always a "3rd
>> class citizen"
>> and we've run into many hand holding scenarios with our Flink
>> developers
>> trying to get the Stringify syntax working.
>> Overall I am a +1 on this, it also help reduce the
>> development cost of the
>> Table API so that we no longer need to maintain different DSL and
>> documentations.
>>
>> I left a few comments in the doc. and also some features that
>> I think will
>> be beneficial to the final outcome. Please kindly take a look
>> @Timo.
>>
>> Many thanks,
>> Rong
>>
>> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther
>> <twalthr@apache.org <ma...@apache.org>> wrote:
>>
>> > Hi everyone,
>> >
>> > some of you might have already noticed the JIRA issue that
>> I opened
>> > recently [1] about introducing a proper Java expression DSL
>> for the
>> > Table API. Instead of using string-based expressions, we
>> should aim for
>> > a unified, maintainable, programmatic Java DSL.
>> >
>> > Some background: The Blink merging efforts and the big
>> refactorings as
>> > part of FLIP-32 have revealed many shortcomings in the
>> current Table &
>> > SQL API design. Most of these legacy issues cause problems
>> nowadays in
>> > making the Table API a first-class API next to the
>> DataStream API. An
>> > example is the ExpressionParser class[2]. It was
>> implemented in the
>> > early days of the Table API using Scala parser combinators.
>> During the
>> > last years, this parser caused many JIRA issues and user
>> confusion on
>> > the mailing list. Because the exceptions and syntax might
>> not be
>> > straight forward.
>> >
>> > For FLINK-11908, we added a temporary bridge instead of
>> reimplementing
>> > the parser in Java for FLIP-32. However, this is only a
>> intermediate
>> > solution until we made a final decision.
>> >
>> > I would like to propose a new, parser-free version of the
>> Java Table API:
>> >
>> >
>> >
>> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
>> >
>> > I already implemented an early protoype that shows that
>> such a DSL is
>> > not much implementation effort and integrates nicely with
>> all existing
>> > API methods.
>> >
>> > What do you think?
>> >
>> > Thanks for your feedback,
>> >
>> > Timo
>> >
>> > [1] https://issues.apache.org/jira/browse/FLINK-11890
>> >
>> > [2]
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
>> >
>> >
>>
>
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Posted by Jark Wu <im...@gmail.com>.
Hi Timo,
Sounds good to me.
Do you want to deprecate the string-based API in 1.9 or make the decision
in 1.10 after some feedbacks ?
On Thu, 21 Mar 2019 at 21:32, Timo Walther <tw...@apache.org> wrote:
> Thanks for your feedback Rong and Jark.
>
> @Jark: Yes, you are right that the string-based API is used quite a lot.
> On the other side, the potential user base in the future is still bigger
> than our current user base. Because the Table API will become equally
> important as the DataStream API, we really need to fix some crucial design
> decisions before it is too late. I would suggest to introduce the new DSL
> in 1.9 and remove the Expression parser either in 1.10 or 1.11. From a
> developement point of view, I think we can handle the overhead to maintain
> 3 APIs until then because 2 APIs will share the same code base + expression
> parser.
>
> Regards,
> Timo
>
> Am 21.03.19 um 05:21 schrieb Jark Wu:
>
> Hi Timo,
>
> I'm +1 on the proposal. I like the idea to provide a Java DSL which is
> more friendly than string-based approach in programming.
>
> My concern is if/when we can drop the string-based expression parser. If
> it takes a very long time, we have to paid more development
> cost on the three Table APIs. As far as I know, the string-based API is
> used in many companies.
> We should also get some feedbacks from users. So I'm CCing this email to
> user mailing list.
>
> Best,
> Jark
>
>
>
> On Wed, 20 Mar 2019 at 08:51, Rong Rong <wa...@gmail.com> wrote:
>
>> Thanks for sharing the initiative of improving Java side Table expression
>> DSL.
>>
>> I agree as in the doc stated that Java DSL was always a "3rd class
>> citizen"
>> and we've run into many hand holding scenarios with our Flink developers
>> trying to get the Stringify syntax working.
>> Overall I am a +1 on this, it also help reduce the development cost of the
>> Table API so that we no longer need to maintain different DSL and
>> documentations.
>>
>> I left a few comments in the doc. and also some features that I think will
>> be beneficial to the final outcome. Please kindly take a look @Timo.
>>
>> Many thanks,
>> Rong
>>
>> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther <tw...@apache.org> wrote:
>>
>> > Hi everyone,
>> >
>> > some of you might have already noticed the JIRA issue that I opened
>> > recently [1] about introducing a proper Java expression DSL for the
>> > Table API. Instead of using string-based expressions, we should aim for
>> > a unified, maintainable, programmatic Java DSL.
>> >
>> > Some background: The Blink merging efforts and the big refactorings as
>> > part of FLIP-32 have revealed many shortcomings in the current Table &
>> > SQL API design. Most of these legacy issues cause problems nowadays in
>> > making the Table API a first-class API next to the DataStream API. An
>> > example is the ExpressionParser class[2]. It was implemented in the
>> > early days of the Table API using Scala parser combinators. During the
>> > last years, this parser caused many JIRA issues and user confusion on
>> > the mailing list. Because the exceptions and syntax might not be
>> > straight forward.
>> >
>> > For FLINK-11908, we added a temporary bridge instead of reimplementing
>> > the parser in Java for FLIP-32. However, this is only a intermediate
>> > solution until we made a final decision.
>> >
>> > I would like to propose a new, parser-free version of the Java Table
>> API:
>> >
>> >
>> >
>> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
>> >
>> > I already implemented an early protoype that shows that such a DSL is
>> > not much implementation effort and integrates nicely with all existing
>> > API methods.
>> >
>> > What do you think?
>> >
>> > Thanks for your feedback,
>> >
>> > Timo
>> >
>> > [1] https://issues.apache.org/jira/browse/FLINK-11890
>> >
>> > [2]
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
>> >
>> >
>>
>
>
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Posted by Jark Wu <im...@gmail.com>.
Hi Timo,
Sounds good to me.
Do you want to deprecate the string-based API in 1.9 or make the decision
in 1.10 after some feedbacks ?
On Thu, 21 Mar 2019 at 21:32, Timo Walther <tw...@apache.org> wrote:
> Thanks for your feedback Rong and Jark.
>
> @Jark: Yes, you are right that the string-based API is used quite a lot.
> On the other side, the potential user base in the future is still bigger
> than our current user base. Because the Table API will become equally
> important as the DataStream API, we really need to fix some crucial design
> decisions before it is too late. I would suggest to introduce the new DSL
> in 1.9 and remove the Expression parser either in 1.10 or 1.11. From a
> developement point of view, I think we can handle the overhead to maintain
> 3 APIs until then because 2 APIs will share the same code base + expression
> parser.
>
> Regards,
> Timo
>
> Am 21.03.19 um 05:21 schrieb Jark Wu:
>
> Hi Timo,
>
> I'm +1 on the proposal. I like the idea to provide a Java DSL which is
> more friendly than string-based approach in programming.
>
> My concern is if/when we can drop the string-based expression parser. If
> it takes a very long time, we have to paid more development
> cost on the three Table APIs. As far as I know, the string-based API is
> used in many companies.
> We should also get some feedbacks from users. So I'm CCing this email to
> user mailing list.
>
> Best,
> Jark
>
>
>
> On Wed, 20 Mar 2019 at 08:51, Rong Rong <wa...@gmail.com> wrote:
>
>> Thanks for sharing the initiative of improving Java side Table expression
>> DSL.
>>
>> I agree as in the doc stated that Java DSL was always a "3rd class
>> citizen"
>> and we've run into many hand holding scenarios with our Flink developers
>> trying to get the Stringify syntax working.
>> Overall I am a +1 on this, it also help reduce the development cost of the
>> Table API so that we no longer need to maintain different DSL and
>> documentations.
>>
>> I left a few comments in the doc. and also some features that I think will
>> be beneficial to the final outcome. Please kindly take a look @Timo.
>>
>> Many thanks,
>> Rong
>>
>> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther <tw...@apache.org> wrote:
>>
>> > Hi everyone,
>> >
>> > some of you might have already noticed the JIRA issue that I opened
>> > recently [1] about introducing a proper Java expression DSL for the
>> > Table API. Instead of using string-based expressions, we should aim for
>> > a unified, maintainable, programmatic Java DSL.
>> >
>> > Some background: The Blink merging efforts and the big refactorings as
>> > part of FLIP-32 have revealed many shortcomings in the current Table &
>> > SQL API design. Most of these legacy issues cause problems nowadays in
>> > making the Table API a first-class API next to the DataStream API. An
>> > example is the ExpressionParser class[2]. It was implemented in the
>> > early days of the Table API using Scala parser combinators. During the
>> > last years, this parser caused many JIRA issues and user confusion on
>> > the mailing list. Because the exceptions and syntax might not be
>> > straight forward.
>> >
>> > For FLINK-11908, we added a temporary bridge instead of reimplementing
>> > the parser in Java for FLIP-32. However, this is only a intermediate
>> > solution until we made a final decision.
>> >
>> > I would like to propose a new, parser-free version of the Java Table
>> API:
>> >
>> >
>> >
>> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
>> >
>> > I already implemented an early protoype that shows that such a DSL is
>> > not much implementation effort and integrates nicely with all existing
>> > API methods.
>> >
>> > What do you think?
>> >
>> > Thanks for your feedback,
>> >
>> > Timo
>> >
>> > [1] https://issues.apache.org/jira/browse/FLINK-11890
>> >
>> > [2]
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
>> >
>> >
>>
>
>
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Posted by Timo Walther <tw...@apache.org>.
Thanks for your feedback Rong and Jark.
@Jark: Yes, you are right that the string-based API is used quite a lot.
On the other side, the potential user base in the future is still bigger
than our current user base. Because the Table API will become equally
important as the DataStream API, we really need to fix some crucial
design decisions before it is too late. I would suggest to introduce the
new DSL in 1.9 and remove the Expression parser either in 1.10 or 1.11.
From a developement point of view, I think we can handle the overhead
to maintain 3 APIs until then because 2 APIs will share the same code
base + expression parser.
Regards,
Timo
Am 21.03.19 um 05:21 schrieb Jark Wu:
> Hi Timo,
>
> I'm +1 on the proposal. I like the idea to provide a Java DSL which is
> more friendly than string-based approach in programming.
>
> My concern is if/when we can drop the string-based expression parser.
> If it takes a very long time, we have to paid more development
> cost on the three Table APIs. As far as I know, the string-based API
> is used in many companies.
> We should also get some feedbacks from users. So I'm CCing this email
> to user mailing list.
>
> Best,
> Jark
>
>
>
> On Wed, 20 Mar 2019 at 08:51, Rong Rong <walterddr@gmail.com
> <ma...@gmail.com>> wrote:
>
> Thanks for sharing the initiative of improving Java side Table
> expression
> DSL.
>
> I agree as in the doc stated that Java DSL was always a "3rd class
> citizen"
> and we've run into many hand holding scenarios with our Flink
> developers
> trying to get the Stringify syntax working.
> Overall I am a +1 on this, it also help reduce the development
> cost of the
> Table API so that we no longer need to maintain different DSL and
> documentations.
>
> I left a few comments in the doc. and also some features that I
> think will
> be beneficial to the final outcome. Please kindly take a look @Timo.
>
> Many thanks,
> Rong
>
> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther <twalthr@apache.org
> <ma...@apache.org>> wrote:
>
> > Hi everyone,
> >
> > some of you might have already noticed the JIRA issue that I opened
> > recently [1] about introducing a proper Java expression DSL for the
> > Table API. Instead of using string-based expressions, we should
> aim for
> > a unified, maintainable, programmatic Java DSL.
> >
> > Some background: The Blink merging efforts and the big
> refactorings as
> > part of FLIP-32 have revealed many shortcomings in the current
> Table &
> > SQL API design. Most of these legacy issues cause problems
> nowadays in
> > making the Table API a first-class API next to the DataStream
> API. An
> > example is the ExpressionParser class[2]. It was implemented in the
> > early days of the Table API using Scala parser combinators.
> During the
> > last years, this parser caused many JIRA issues and user
> confusion on
> > the mailing list. Because the exceptions and syntax might not be
> > straight forward.
> >
> > For FLINK-11908, we added a temporary bridge instead of
> reimplementing
> > the parser in Java for FLIP-32. However, this is only a intermediate
> > solution until we made a final decision.
> >
> > I would like to propose a new, parser-free version of the Java
> Table API:
> >
> >
> >
> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
> >
> > I already implemented an early protoype that shows that such a
> DSL is
> > not much implementation effort and integrates nicely with all
> existing
> > API methods.
> >
> > What do you think?
> >
> > Thanks for your feedback,
> >
> > Timo
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-11890
> >
> > [2]
> >
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
> >
> >
>
Re: [DISCUSS] Introduction of a Table API Java Expression DSL
Posted by Timo Walther <tw...@apache.org>.
Thanks for your feedback Rong and Jark.
@Jark: Yes, you are right that the string-based API is used quite a lot.
On the other side, the potential user base in the future is still bigger
than our current user base. Because the Table API will become equally
important as the DataStream API, we really need to fix some crucial
design decisions before it is too late. I would suggest to introduce the
new DSL in 1.9 and remove the Expression parser either in 1.10 or 1.11.
From a developement point of view, I think we can handle the overhead
to maintain 3 APIs until then because 2 APIs will share the same code
base + expression parser.
Regards,
Timo
Am 21.03.19 um 05:21 schrieb Jark Wu:
> Hi Timo,
>
> I'm +1 on the proposal. I like the idea to provide a Java DSL which is
> more friendly than string-based approach in programming.
>
> My concern is if/when we can drop the string-based expression parser.
> If it takes a very long time, we have to paid more development
> cost on the three Table APIs. As far as I know, the string-based API
> is used in many companies.
> We should also get some feedbacks from users. So I'm CCing this email
> to user mailing list.
>
> Best,
> Jark
>
>
>
> On Wed, 20 Mar 2019 at 08:51, Rong Rong <walterddr@gmail.com
> <ma...@gmail.com>> wrote:
>
> Thanks for sharing the initiative of improving Java side Table
> expression
> DSL.
>
> I agree as in the doc stated that Java DSL was always a "3rd class
> citizen"
> and we've run into many hand holding scenarios with our Flink
> developers
> trying to get the Stringify syntax working.
> Overall I am a +1 on this, it also help reduce the development
> cost of the
> Table API so that we no longer need to maintain different DSL and
> documentations.
>
> I left a few comments in the doc. and also some features that I
> think will
> be beneficial to the final outcome. Please kindly take a look @Timo.
>
> Many thanks,
> Rong
>
> On Mon, Mar 18, 2019 at 7:15 AM Timo Walther <twalthr@apache.org
> <ma...@apache.org>> wrote:
>
> > Hi everyone,
> >
> > some of you might have already noticed the JIRA issue that I opened
> > recently [1] about introducing a proper Java expression DSL for the
> > Table API. Instead of using string-based expressions, we should
> aim for
> > a unified, maintainable, programmatic Java DSL.
> >
> > Some background: The Blink merging efforts and the big
> refactorings as
> > part of FLIP-32 have revealed many shortcomings in the current
> Table &
> > SQL API design. Most of these legacy issues cause problems
> nowadays in
> > making the Table API a first-class API next to the DataStream
> API. An
> > example is the ExpressionParser class[2]. It was implemented in the
> > early days of the Table API using Scala parser combinators.
> During the
> > last years, this parser caused many JIRA issues and user
> confusion on
> > the mailing list. Because the exceptions and syntax might not be
> > straight forward.
> >
> > For FLINK-11908, we added a temporary bridge instead of
> reimplementing
> > the parser in Java for FLIP-32. However, this is only a intermediate
> > solution until we made a final decision.
> >
> > I would like to propose a new, parser-free version of the Java
> Table API:
> >
> >
> >
> https://docs.google.com/document/d/1r3bfR9R6q5Km0wXKcnhfig2XQ4aMiLG5h2MTx960Fg8/edit?usp=sharing
> >
> > I already implemented an early protoype that shows that such a
> DSL is
> > not much implementation effort and integrates nicely with all
> existing
> > API methods.
> >
> > What do you think?
> >
> > Thanks for your feedback,
> >
> > Timo
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-11890
> >
> > [2]
> >
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/expressions/PlannerExpressionParserImpl.scala
> >
> >
>