You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by burkaygur <bu...@gmail.com> on 2021/01/06 15:19:41 UTC
Roadmap for Execution Mode (Batch/Streaming) and interaction with
Table/SQL APIs
Hi Flink Community,
Really excited for the "true unification" of Batch and Stream APIs, and had
a few questions for clarification after reading the release notes, and
watching the video by aljoscha here
<https://www.youtube.com/watch?v=z9ye4jzp4DQ> .
1) How do these changes impact the Table and SQL APIs? Are they completely
orthogonal or can we get the benefits of the new Batch Mode with Flink SQL
as well?
2) What is the best ticket to follow the roadmap & track the progress of
this whole project. Specifically the parts about bootstrapping of state. I
would love to help contribute to it.
Best,
Burkay
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Roadmap for Execution Mode (Batch/Streaming) and interaction with
Table/SQL APIs
Posted by Piotr Nowojski <pn...@apache.org>.
Hi,
1. I think those changes will mostly bring new features/functionalities to
the existing Streaming APIs in order to fully support batch executions. For
example one way or another to better handle "bounded data streams" in the
DataStream API.
2. I think there is and there is not going to be one single umbrella
ticket, as this whole idea is going to take quite a bit of time to fully
design. There are a couple of the initial design proposals connected with
the batch and streaming unification:
FLIP-134 [1]
FLIP-147 [2]
But I'm pretty sure more will follow in the future.
Best,
Piotrek
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Batch+execution+for+the+DataStream+API
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-147%3A+Support+Checkpoints+After+Tasks+Finished
śr., 6 sty 2021 o 16:20 burkaygur <bu...@gmail.com> napisał(a):
> Hi Flink Community,
>
> Really excited for the "true unification" of Batch and Stream APIs, and had
> a few questions for clarification after reading the release notes, and
> watching the video by aljoscha here
> <https://www.youtube.com/watch?v=z9ye4jzp4DQ> .
>
>
> 1) How do these changes impact the Table and SQL APIs? Are they completely
> orthogonal or can we get the benefits of the new Batch Mode with Flink SQL
> as well?
> 2) What is the best ticket to follow the roadmap & track the progress of
> this whole project. Specifically the parts about bootstrapping of state. I
> would love to help contribute to it.
>
> Best,
> Burkay
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: Roadmap for Execution Mode (Batch/Streaming) and interaction
with Table/SQL APIs
Posted by Aljoscha Krettek <al...@apache.org>.
Also cc'ing dev@flink.apache.org
On 2021/01/06 09:19, burkaygur wrote:
>1) How do these changes impact the Table and SQL APIs? Are they completely
>orthogonal or can we get the benefits of the new Batch Mode with Flink SQL
>as well?
The answer here is a bit complicated. The Table API/SQL already use
similar techniques under the hood for BATCH execution. Our recent
changes just made similar functionality available for the DataStream
API. So in a way the changes are orthogonal.
However, the changes are relevant when you want to interoperate with the
DataStream and Table API. There it becomes relevant that both parts can
do STREAMING/BATCH execution well. We're not 100% there yet on this
front but we're tracking some work under FLIP-136 [1].
[1] https://cwiki.apache.org/confluence/x/0DN4CQ
>2) What is the best ticket to follow the roadmap & track the progress of
>this whole project. Specifically the parts about bootstrapping of state. I
>would love to help contribute to it.
I would say the best way to follow progress is the dev mailing list and
the the FLIP overview page [2]. That's not super intuitive and can be
hard to follow for outsiders. Sometimes, people such as myself will
write blog posts on the Flink website or private blogs that try and
shine a light on the development so it might help to follow some
relevant people from the project on Twitter, where such posts are often
announced.
Specifically about state bootstrapping, we don't have many concrete
thoughts yet. It would help if you could talk about some of the
requirements that you would have for this.
Best,
Aljoscha
[2]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
Best,
Aljoscha
Re: Roadmap for Execution Mode (Batch/Streaming) and interaction
with Table/SQL APIs
Posted by Aljoscha Krettek <al...@apache.org>.
Also cc'ing dev@flink.apache.org
On 2021/01/06 09:19, burkaygur wrote:
>1) How do these changes impact the Table and SQL APIs? Are they completely
>orthogonal or can we get the benefits of the new Batch Mode with Flink SQL
>as well?
The answer here is a bit complicated. The Table API/SQL already use
similar techniques under the hood for BATCH execution. Our recent
changes just made similar functionality available for the DataStream
API. So in a way the changes are orthogonal.
However, the changes are relevant when you want to interoperate with the
DataStream and Table API. There it becomes relevant that both parts can
do STREAMING/BATCH execution well. We're not 100% there yet on this
front but we're tracking some work under FLIP-136 [1].
[1] https://cwiki.apache.org/confluence/x/0DN4CQ
>2) What is the best ticket to follow the roadmap & track the progress of
>this whole project. Specifically the parts about bootstrapping of state. I
>would love to help contribute to it.
I would say the best way to follow progress is the dev mailing list and
the the FLIP overview page [2]. That's not super intuitive and can be
hard to follow for outsiders. Sometimes, people such as myself will
write blog posts on the Flink website or private blogs that try and
shine a light on the development so it might help to follow some
relevant people from the project on Twitter, where such posts are often
announced.
Specifically about state bootstrapping, we don't have many concrete
thoughts yet. It would help if you could talk about some of the
requirements that you would have for this.
Best,
Aljoscha
[2]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
Best,
Aljoscha