You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by John Tipper <jo...@hotmail.com> on 2019/05/30 10:29:34 UTC
Flink internals
Hi all,
Is there a guide somewhere for the internals of Flink for developers wanting to get involved in core development? I'm particularly interested in any notes on how the codebase is put together so that it's possible to learn how it works internally.
I'm particularly interested in what classes are involved in the barrier mechanism and state and where these are found, but it's a large codebase so I'm sure it would be helpful for new developers to have pointers to what does what inside the codebase or where to start looking for details of certain things.
Many thanks,
John
Re: Flink internals
Posted by Piotr Nowojski <pi...@ververica.com>.
Hi,
You can also read the FLIP proposals. Unluckily, one that is very internal [1] about credit based flow control [2] was not published as an official FLIP :( Regarding network stack and some of the other topics, there are some blogs/Flink Forward talks as well.
Piotrek
[1] https://docs.google.com/document/d/1chTOuOqe0sBsjldA_r-wXYeSIhU2zRGpUaTaik7QZ84/edit <https://docs.google.com/document/d/1chTOuOqe0sBsjldA_r-wXYeSIhU2zRGpUaTaik7QZ84/edit>
[2] https://issues.apache.org/jira/browse/FLINK-7282 <https://issues.apache.org/jira/browse/FLINK-7282>
> On 4 Jun 2019, at 11:55, John Tipper <jo...@hotmail.com> wrote:
>
> Hi Till, Fan & Rong,
>
> Thanks for your feedback - I'd seen the Flink internal page but sadly, as Rong pointed out, it's pretty limited and not maintained. I'll ask on the mailing lists, but I think it would be really helpful if there were a guide for Flink developers who want to contribute to Flink, as opposed to the current documentation which is really aimed at Flink application developers (and is really pretty good). I'd happily contribute to this, but I can't find my way round the internals...
>
> Many thanks,
>
> John
> ________________________________
> From: Till Rohrmann <tr...@apache.org>
> Sent: 04 June 2019 09:12
> To: dev
> Subject: Re: Flink internals
>
> Hi John,
>
> unfortunately, there are no really good and up to date documents about
> Flink's internals. There was some discussion about updating the internals
> [1] but the community did decide against submitting it as a season of docs
> project. I agree that we should update our documentation about Flink's
> internals, though.
>
> At the moment, the best way about learning about Flink is the code and the
> user/dev ML where you can ask about concepts and how things work.
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Apache-Flink-at-Season-of-Docs-td28133.html
>
> Cheers,
> Till
>
> On Fri, May 31, 2019 at 2:50 AM Rong Rong <wa...@gmail.com> wrote:
>
>> Hi Fan, John,
>>
>> The flink internal link[1] seems to be not updated in the last years.
>> I found out some higher level pages in the official documentation here[2]
>> however they are still very limited.
>>
>> Is there any other well maintained, internal documentations for
>> contributors?
>> I think it might be a good idea to maintain a higher-level guidance doc for
>> new contributors about Flink internals?
>>
>> Best,
>> Rong
>>
>> ----
>> [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
>> [2]
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.8/internals/components.html
>>
>> On Thu, May 30, 2019 at 4:48 AM Fan Liya <li...@gmail.com> wrote:
>>
>>> You can find some articles here:
>>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
>>>
>>> Best,
>>> Liya Fan
>>>
>>> On Thu, May 30, 2019 at 6:29 PM John Tipper <jo...@hotmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Is there a guide somewhere for the internals of Flink for developers
>>>> wanting to get involved in core development? I'm particularly
>> interested
>>>> in any notes on how the codebase is put together so that it's possible
>> to
>>>> learn how it works internally.
>>>>
>>>> I'm particularly interested in what classes are involved in the barrier
>>>> mechanism and state and where these are found, but it's a large
>> codebase
>>> so
>>>> I'm sure it would be helpful for new developers to have pointers to
>> what
>>>> does what inside the codebase or where to start looking for details of
>>>> certain things.
>>>>
>>>> Many thanks,
>>>>
>>>> John
>>>>
>>>
>>
Re: Flink internals
Posted by John Tipper <jo...@hotmail.com>.
Hi Till, Fan & Rong,
Thanks for your feedback - I'd seen the Flink internal page but sadly, as Rong pointed out, it's pretty limited and not maintained. I'll ask on the mailing lists, but I think it would be really helpful if there were a guide for Flink developers who want to contribute to Flink, as opposed to the current documentation which is really aimed at Flink application developers (and is really pretty good). I'd happily contribute to this, but I can't find my way round the internals...
Many thanks,
John
________________________________
From: Till Rohrmann <tr...@apache.org>
Sent: 04 June 2019 09:12
To: dev
Subject: Re: Flink internals
Hi John,
unfortunately, there are no really good and up to date documents about
Flink's internals. There was some discussion about updating the internals
[1] but the community did decide against submitting it as a season of docs
project. I agree that we should update our documentation about Flink's
internals, though.
At the moment, the best way about learning about Flink is the code and the
user/dev ML where you can ask about concepts and how things work.
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Apache-Flink-at-Season-of-Docs-td28133.html
Cheers,
Till
On Fri, May 31, 2019 at 2:50 AM Rong Rong <wa...@gmail.com> wrote:
> Hi Fan, John,
>
> The flink internal link[1] seems to be not updated in the last years.
> I found out some higher level pages in the official documentation here[2]
> however they are still very limited.
>
> Is there any other well maintained, internal documentations for
> contributors?
> I think it might be a good idea to maintain a higher-level guidance doc for
> new contributors about Flink internals?
>
> Best,
> Rong
>
> ----
> [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
> [2]
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/internals/components.html
>
> On Thu, May 30, 2019 at 4:48 AM Fan Liya <li...@gmail.com> wrote:
>
> > You can find some articles here:
> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
> >
> > Best,
> > Liya Fan
> >
> > On Thu, May 30, 2019 at 6:29 PM John Tipper <jo...@hotmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > Is there a guide somewhere for the internals of Flink for developers
> > > wanting to get involved in core development? I'm particularly
> interested
> > > in any notes on how the codebase is put together so that it's possible
> to
> > > learn how it works internally.
> > >
> > > I'm particularly interested in what classes are involved in the barrier
> > > mechanism and state and where these are found, but it's a large
> codebase
> > so
> > > I'm sure it would be helpful for new developers to have pointers to
> what
> > > does what inside the codebase or where to start looking for details of
> > > certain things.
> > >
> > > Many thanks,
> > >
> > > John
> > >
> >
>
Re: Flink internals
Posted by Till Rohrmann <tr...@apache.org>.
Hi John,
unfortunately, there are no really good and up to date documents about
Flink's internals. There was some discussion about updating the internals
[1] but the community did decide against submitting it as a season of docs
project. I agree that we should update our documentation about Flink's
internals, though.
At the moment, the best way about learning about Flink is the code and the
user/dev ML where you can ask about concepts and how things work.
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Apache-Flink-at-Season-of-Docs-td28133.html
Cheers,
Till
On Fri, May 31, 2019 at 2:50 AM Rong Rong <wa...@gmail.com> wrote:
> Hi Fan, John,
>
> The flink internal link[1] seems to be not updated in the last years.
> I found out some higher level pages in the official documentation here[2]
> however they are still very limited.
>
> Is there any other well maintained, internal documentations for
> contributors?
> I think it might be a good idea to maintain a higher-level guidance doc for
> new contributors about Flink internals?
>
> Best,
> Rong
>
> ----
> [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
> [2]
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/internals/components.html
>
> On Thu, May 30, 2019 at 4:48 AM Fan Liya <li...@gmail.com> wrote:
>
> > You can find some articles here:
> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
> >
> > Best,
> > Liya Fan
> >
> > On Thu, May 30, 2019 at 6:29 PM John Tipper <jo...@hotmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > Is there a guide somewhere for the internals of Flink for developers
> > > wanting to get involved in core development? I'm particularly
> interested
> > > in any notes on how the codebase is put together so that it's possible
> to
> > > learn how it works internally.
> > >
> > > I'm particularly interested in what classes are involved in the barrier
> > > mechanism and state and where these are found, but it's a large
> codebase
> > so
> > > I'm sure it would be helpful for new developers to have pointers to
> what
> > > does what inside the codebase or where to start looking for details of
> > > certain things.
> > >
> > > Many thanks,
> > >
> > > John
> > >
> >
>
Re: Flink internals
Posted by Rong Rong <wa...@gmail.com>.
Hi Fan, John,
The flink internal link[1] seems to be not updated in the last years.
I found out some higher level pages in the official documentation here[2]
however they are still very limited.
Is there any other well maintained, internal documentations for
contributors?
I think it might be a good idea to maintain a higher-level guidance doc for
new contributors about Flink internals?
Best,
Rong
----
[1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/internals/components.html
On Thu, May 30, 2019 at 4:48 AM Fan Liya <li...@gmail.com> wrote:
> You can find some articles here:
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
>
> Best,
> Liya Fan
>
> On Thu, May 30, 2019 at 6:29 PM John Tipper <jo...@hotmail.com>
> wrote:
>
> > Hi all,
> >
> > Is there a guide somewhere for the internals of Flink for developers
> > wanting to get involved in core development? I'm particularly interested
> > in any notes on how the codebase is put together so that it's possible to
> > learn how it works internally.
> >
> > I'm particularly interested in what classes are involved in the barrier
> > mechanism and state and where these are found, but it's a large codebase
> so
> > I'm sure it would be helpful for new developers to have pointers to what
> > does what inside the codebase or where to start looking for details of
> > certain things.
> >
> > Many thanks,
> >
> > John
> >
>
Re: Flink internals
Posted by Fan Liya <li...@gmail.com>.
You can find some articles here:
https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
Best,
Liya Fan
On Thu, May 30, 2019 at 6:29 PM John Tipper <jo...@hotmail.com> wrote:
> Hi all,
>
> Is there a guide somewhere for the internals of Flink for developers
> wanting to get involved in core development? I'm particularly interested
> in any notes on how the codebase is put together so that it's possible to
> learn how it works internally.
>
> I'm particularly interested in what classes are involved in the barrier
> mechanism and state and where these are found, but it's a large codebase so
> I'm sure it would be helpful for new developers to have pointers to what
> does what inside the codebase or where to start looking for details of
> certain things.
>
> Many thanks,
>
> John
>