You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Erik Erlandson <ee...@redhat.com> on 2018/10/22 16:27:16 UTC
Fwd: reaching out from k8s Big Data SIG
Hi Flink community,
I serve as chair on the Kubernetes Big Data SIG, and we've had some
interest from the SIG in learning more about what the Flink community is
working on in the kubernetes space.
I'm putting out feelers to see if anybody from the community would be
interested in giving a brief talk and/or demo of Flink integrations with
k8s at one of our regular meetings.
The Big Data SIG holds bi-weekly meetings at 10am Pacific time (next
meeting is Oct 31). Feel free to attend our meetings, or visit our slack
channel:
https://kubernetes.slack.com/messages/C0ELB338T/
More SIG info is available on our regular meeting minutes doc:
https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/
Thanks!
Erik Erlandson
Re: reaching out from k8s Big Data SIG
Posted by Till Rohrmann <tr...@apache.org>.
Hi Erik,
thanks for reaching out to the Flink community. The Flink community is
actually actively working on improving our K8s support because we see a lot
of adoption and our users would like to run Flink on K8s. Apart from the
existing Flink docker images there are currently two development threads:
1) Active K8s integration [1]
The idea of the active K8s integration is to enable Flink to talk to the
K8s master in order to actively start or terminate new Pods which run
Flink's TaskManager. That way, Flink would be able to adjust the set of
available resources dynamically based on the actual needs (coming from a
Flink job).
I know that the community is actively working on drafting a design document
for this mode which will be published very soon.
Moreover, I have a dev branch where I prototyped the K8s integration [2].
2) Reactive container mode [3]
The reactive container mode works actually independent of the underlying
container framework but K8s will be the natural deployment target for this.
The idea is that not Flink but an external system/user is responsible for
allocating/releasing resources. E.g. some auto-scaling groups could decide
that Flink is currently not keeping up wrt some metric and start some new
Pods (running Flink's TaskManager). If this happens, Flink should react to
such an event and use the newly available resources. Usually this means
that Flink scales up the parallelism of its job to deploy new tasks to the
newly started TaskManager.
There is actually a design document linked under the respective JIRA issue
for more details.
I would be happy to join your meeting to give an overview what the Flink
community is currently developing, answering questions and gathering
feedback for what we intend to do. How would I join this meeting?
[1] https://issues.apache.org/jira/browse/FLINK-9953
[2] https://github.com/tillrohrmann/flink/tree/nativeKubernetes
[3] https://issues.apache.org/jira/browse/FLINK-10407
Cheers,
Till
On Mon, Oct 22, 2018 at 6:55 PM Erik Erlandson <ee...@redhat.com> wrote:
> Hi Flink community,
>
> I serve as chair on the Kubernetes Big Data SIG, and we've had some
> interest from the SIG in learning more about what the Flink community is
> working on in the kubernetes space.
>
> I'm putting out feelers to see if anybody from the community would be
> interested in giving a brief talk and/or demo of Flink integrations with
> k8s at one of our regular meetings.
>
> The Big Data SIG holds bi-weekly meetings at 10am Pacific time (next
> meeting is Oct 31). Feel free to attend our meetings, or visit our slack
> channel:
> https://kubernetes.slack.com/messages/C0ELB338T/
>
> More SIG info is available on our regular meeting minutes doc:
>
> https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/
>
> Thanks!
> Erik Erlandson
>
Re: reaching out from k8s Big Data SIG
Posted by Jin Sun <is...@gmail.com>.
@Erik, I’m working with Till regarding together for the Flink K8s native integration, we have a draft of design document, and plan to post the document next week and collect feedbacks.
@Anand, operator is a interesting topic, look forward to see your design document.
I will keep you updated.
> On Oct 23, 2018, at 3:32 PM, Anand Swaminathan <as...@lyft.com.INVALID> wrote:
>
> Hi Erik,
>
> We @Flyte team within Lyft have been working on SparkK8Operator for quite
> some time, and now we have built a FlinkK8Operator to run Flink
> applications on Kubernetes.
> We would be happy to demo the Flink Operator during the meeting.
>
> Also I am in the process of fleshing out all the details about the Operator
> in a doc, that I would share with the Flink community in a week or so.
>
> Thanks,
> Anand
>
> On Mon, Oct 22, 2018 at 9:55 AM Erik Erlandson <ee...@redhat.com> wrote:
>
>> Hi Flink community,
>>
>> I serve as chair on the Kubernetes Big Data SIG, and we've had some
>> interest from the SIG in learning more about what the Flink community is
>> working on in the kubernetes space.
>>
>> I'm putting out feelers to see if anybody from the community would be
>> interested in giving a brief talk and/or demo of Flink integrations with
>> k8s at one of our regular meetings.
>>
>> The Big Data SIG holds bi-weekly meetings at 10am Pacific time (next
>> meeting is Oct 31). Feel free to attend our meetings, or visit our slack
>> channel:
>> https://kubernetes.slack.com/messages/C0ELB338T/
>>
>> More SIG info is available on our regular meeting minutes doc:
>>
>> https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/
>>
>> Thanks!
>> Erik Erlandson
>>
Re: reaching out from k8s Big Data SIG
Posted by Anand Swaminathan <as...@lyft.com.INVALID>.
Hi Erik,
We @Flyte team within Lyft have been working on SparkK8Operator for quite
some time, and now we have built a FlinkK8Operator to run Flink
applications on Kubernetes.
We would be happy to demo the Flink Operator during the meeting.
Also I am in the process of fleshing out all the details about the Operator
in a doc, that I would share with the Flink community in a week or so.
Thanks,
Anand
On Mon, Oct 22, 2018 at 9:55 AM Erik Erlandson <ee...@redhat.com> wrote:
> Hi Flink community,
>
> I serve as chair on the Kubernetes Big Data SIG, and we've had some
> interest from the SIG in learning more about what the Flink community is
> working on in the kubernetes space.
>
> I'm putting out feelers to see if anybody from the community would be
> interested in giving a brief talk and/or demo of Flink integrations with
> k8s at one of our regular meetings.
>
> The Big Data SIG holds bi-weekly meetings at 10am Pacific time (next
> meeting is Oct 31). Feel free to attend our meetings, or visit our slack
> channel:
> https://kubernetes.slack.com/messages/C0ELB338T/
>
> More SIG info is available on our regular meeting minutes doc:
>
> https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/
>
> Thanks!
> Erik Erlandson
>