You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Felipe Gutierrez <fe...@gmail.com> on 2018/11/28 09:21:41 UTC
Custom scheduler in Flink
Hi,
I want to develop a custom scheduler in Flink to be aware for which host
Flink must process some task. This post shows (using Apache Storm) the kind
of example I want to build (
https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
).
I saw some related question on this post from 2016 (
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html)
and I wonder if there is something more actual or better explained about a
custom scheduler.
Thanks,
Felipe
*--*
*-- Felipe Gutierrez*
*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*
Re: Custom scheduler in Flink
Posted by Felipe Gutierrez <fe...@gmail.com>.
thanks. I saw the google docs just now. I am gonna print and study it at
the weekend.
*--*
*-- Felipe Gutierrez*
*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*
On Fri, Nov 30, 2018 at 4:42 PM Felipe Gutierrez <
felipe.o.gutierrez@gmail.com> wrote:
> thanks. I will check it out
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>
>
> On Fri, Nov 30, 2018 at 3:51 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Felipe,
>>
>> https://issues.apache.org/jira/browse/FLINK-10429 might also be
>> interesting. The community is currently working on making the Scheduler
>> pluggable to make it easier to extend this component.
>>
>> Cheers,
>> Till
>>
>> On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez <
>> felipe.o.gutierrez@gmail.com> wrote:
>>
>>> Thanks, I'll check it out.
>>> *--*
>>> *-- Felipe Gutierrez*
>>>
>>> *-- skype: felipe.o.gutierrez*
>>> *--* *https://felipeogutierrez.blogspot.com
>>> <https://felipeogutierrez.blogspot.com>*
>>>
>>>
>>> On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <ch...@apache.org>
>>> wrote:
>>>
>>> > There's no *reasonable *way to implement a custom Scheduler, i.e.,
>>> > something where can just plug in your scheduler in a nice way. For this
>>> > you'll have to directly modify the source of Flink.
>>> >
>>> > The work in https://issues.apache.org/jira/browse/FLINK-8886 may also
>>> be
>>> > of interest, but is still in the design phase.
>>> >
>>> > On 28.11.2018 10:21, Felipe Gutierrez wrote:
>>> >
>>> > Hi,
>>> >
>>> > I want to develop a custom scheduler in Flink to be aware for which
>>> host
>>> > Flink must process some task. This post shows (using Apache Storm) the
>>> kind
>>> > of example I want to build (
>>> https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
>>> > ).
>>> >
>>> > I saw some related question on this post from 2016 (
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html
>>> )
>>> > and I wonder if there is something more actual or better explained
>>> about a
>>> > custom scheduler.
>>> >
>>> > Thanks,
>>> > Felipe
>>> > *--*
>>> > *-- Felipe Gutierrez*
>>> >
>>> > *-- skype: felipe.o.gutierrez*
>>> > *--* *https://felipeogutierrez.blogspot.com<
>>> https://felipeogutierrez.blogspot.com> <
>>> https://felipeogutierrez.blogspot.com>*
>>> >
>>> >
>>> >
>>> >
>>>
>>
Re: Custom scheduler in Flink
Posted by Felipe Gutierrez <fe...@gmail.com>.
thanks. I saw the google docs just now. I am gonna print and study it at
the weekend.
*--*
*-- Felipe Gutierrez*
*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*
On Fri, Nov 30, 2018 at 4:42 PM Felipe Gutierrez <
felipe.o.gutierrez@gmail.com> wrote:
> thanks. I will check it out
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>
>
> On Fri, Nov 30, 2018 at 3:51 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Felipe,
>>
>> https://issues.apache.org/jira/browse/FLINK-10429 might also be
>> interesting. The community is currently working on making the Scheduler
>> pluggable to make it easier to extend this component.
>>
>> Cheers,
>> Till
>>
>> On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez <
>> felipe.o.gutierrez@gmail.com> wrote:
>>
>>> Thanks, I'll check it out.
>>> *--*
>>> *-- Felipe Gutierrez*
>>>
>>> *-- skype: felipe.o.gutierrez*
>>> *--* *https://felipeogutierrez.blogspot.com
>>> <https://felipeogutierrez.blogspot.com>*
>>>
>>>
>>> On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <ch...@apache.org>
>>> wrote:
>>>
>>> > There's no *reasonable *way to implement a custom Scheduler, i.e.,
>>> > something where can just plug in your scheduler in a nice way. For this
>>> > you'll have to directly modify the source of Flink.
>>> >
>>> > The work in https://issues.apache.org/jira/browse/FLINK-8886 may also
>>> be
>>> > of interest, but is still in the design phase.
>>> >
>>> > On 28.11.2018 10:21, Felipe Gutierrez wrote:
>>> >
>>> > Hi,
>>> >
>>> > I want to develop a custom scheduler in Flink to be aware for which
>>> host
>>> > Flink must process some task. This post shows (using Apache Storm) the
>>> kind
>>> > of example I want to build (
>>> https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
>>> > ).
>>> >
>>> > I saw some related question on this post from 2016 (
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html
>>> )
>>> > and I wonder if there is something more actual or better explained
>>> about a
>>> > custom scheduler.
>>> >
>>> > Thanks,
>>> > Felipe
>>> > *--*
>>> > *-- Felipe Gutierrez*
>>> >
>>> > *-- skype: felipe.o.gutierrez*
>>> > *--* *https://felipeogutierrez.blogspot.com<
>>> https://felipeogutierrez.blogspot.com> <
>>> https://felipeogutierrez.blogspot.com>*
>>> >
>>> >
>>> >
>>> >
>>>
>>
Re: Custom scheduler in Flink
Posted by Felipe Gutierrez <fe...@gmail.com>.
thanks. I will check it out
*--*
*-- Felipe Gutierrez*
*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*
On Fri, Nov 30, 2018 at 3:51 PM Till Rohrmann <tr...@apache.org> wrote:
> Hi Felipe,
>
> https://issues.apache.org/jira/browse/FLINK-10429 might also be
> interesting. The community is currently working on making the Scheduler
> pluggable to make it easier to extend this component.
>
> Cheers,
> Till
>
> On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez <
> felipe.o.gutierrez@gmail.com> wrote:
>
>> Thanks, I'll check it out.
>> *--*
>> *-- Felipe Gutierrez*
>>
>> *-- skype: felipe.o.gutierrez*
>> *--* *https://felipeogutierrez.blogspot.com
>> <https://felipeogutierrez.blogspot.com>*
>>
>>
>> On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>> > There's no *reasonable *way to implement a custom Scheduler, i.e.,
>> > something where can just plug in your scheduler in a nice way. For this
>> > you'll have to directly modify the source of Flink.
>> >
>> > The work in https://issues.apache.org/jira/browse/FLINK-8886 may also
>> be
>> > of interest, but is still in the design phase.
>> >
>> > On 28.11.2018 10:21, Felipe Gutierrez wrote:
>> >
>> > Hi,
>> >
>> > I want to develop a custom scheduler in Flink to be aware for which host
>> > Flink must process some task. This post shows (using Apache Storm) the
>> kind
>> > of example I want to build (
>> https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
>> > ).
>> >
>> > I saw some related question on this post from 2016 (
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html
>> )
>> > and I wonder if there is something more actual or better explained
>> about a
>> > custom scheduler.
>> >
>> > Thanks,
>> > Felipe
>> > *--*
>> > *-- Felipe Gutierrez*
>> >
>> > *-- skype: felipe.o.gutierrez*
>> > *--* *https://felipeogutierrez.blogspot.com<
>> https://felipeogutierrez.blogspot.com> <
>> https://felipeogutierrez.blogspot.com>*
>> >
>> >
>> >
>> >
>>
>
Re: Custom scheduler in Flink
Posted by Felipe Gutierrez <fe...@gmail.com>.
thanks. I will check it out
*--*
*-- Felipe Gutierrez*
*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*
On Fri, Nov 30, 2018 at 3:51 PM Till Rohrmann <tr...@apache.org> wrote:
> Hi Felipe,
>
> https://issues.apache.org/jira/browse/FLINK-10429 might also be
> interesting. The community is currently working on making the Scheduler
> pluggable to make it easier to extend this component.
>
> Cheers,
> Till
>
> On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez <
> felipe.o.gutierrez@gmail.com> wrote:
>
>> Thanks, I'll check it out.
>> *--*
>> *-- Felipe Gutierrez*
>>
>> *-- skype: felipe.o.gutierrez*
>> *--* *https://felipeogutierrez.blogspot.com
>> <https://felipeogutierrez.blogspot.com>*
>>
>>
>> On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>> > There's no *reasonable *way to implement a custom Scheduler, i.e.,
>> > something where can just plug in your scheduler in a nice way. For this
>> > you'll have to directly modify the source of Flink.
>> >
>> > The work in https://issues.apache.org/jira/browse/FLINK-8886 may also
>> be
>> > of interest, but is still in the design phase.
>> >
>> > On 28.11.2018 10:21, Felipe Gutierrez wrote:
>> >
>> > Hi,
>> >
>> > I want to develop a custom scheduler in Flink to be aware for which host
>> > Flink must process some task. This post shows (using Apache Storm) the
>> kind
>> > of example I want to build (
>> https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
>> > ).
>> >
>> > I saw some related question on this post from 2016 (
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html
>> )
>> > and I wonder if there is something more actual or better explained
>> about a
>> > custom scheduler.
>> >
>> > Thanks,
>> > Felipe
>> > *--*
>> > *-- Felipe Gutierrez*
>> >
>> > *-- skype: felipe.o.gutierrez*
>> > *--* *https://felipeogutierrez.blogspot.com<
>> https://felipeogutierrez.blogspot.com> <
>> https://felipeogutierrez.blogspot.com>*
>> >
>> >
>> >
>> >
>>
>
Re: Custom scheduler in Flink
Posted by Till Rohrmann <tr...@apache.org>.
Hi Felipe,
https://issues.apache.org/jira/browse/FLINK-10429 might also be
interesting. The community is currently working on making the Scheduler
pluggable to make it easier to extend this component.
Cheers,
Till
On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez <
felipe.o.gutierrez@gmail.com> wrote:
> Thanks, I'll check it out.
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>
>
> On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > There's no *reasonable *way to implement a custom Scheduler, i.e.,
> > something where can just plug in your scheduler in a nice way. For this
> > you'll have to directly modify the source of Flink.
> >
> > The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be
> > of interest, but is still in the design phase.
> >
> > On 28.11.2018 10:21, Felipe Gutierrez wrote:
> >
> > Hi,
> >
> > I want to develop a custom scheduler in Flink to be aware for which host
> > Flink must process some task. This post shows (using Apache Storm) the
> kind
> > of example I want to build (
> https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
> > ).
> >
> > I saw some related question on this post from 2016 (
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html
> )
> > and I wonder if there is something more actual or better explained about
> a
> > custom scheduler.
> >
> > Thanks,
> > Felipe
> > *--*
> > *-- Felipe Gutierrez*
> >
> > *-- skype: felipe.o.gutierrez*
> > *--* *https://felipeogutierrez.blogspot.com<
> https://felipeogutierrez.blogspot.com> <
> https://felipeogutierrez.blogspot.com>*
> >
> >
> >
> >
>
Re: Custom scheduler in Flink
Posted by Till Rohrmann <tr...@apache.org>.
Hi Felipe,
https://issues.apache.org/jira/browse/FLINK-10429 might also be
interesting. The community is currently working on making the Scheduler
pluggable to make it easier to extend this component.
Cheers,
Till
On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez <
felipe.o.gutierrez@gmail.com> wrote:
> Thanks, I'll check it out.
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>
>
> On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > There's no *reasonable *way to implement a custom Scheduler, i.e.,
> > something where can just plug in your scheduler in a nice way. For this
> > you'll have to directly modify the source of Flink.
> >
> > The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be
> > of interest, but is still in the design phase.
> >
> > On 28.11.2018 10:21, Felipe Gutierrez wrote:
> >
> > Hi,
> >
> > I want to develop a custom scheduler in Flink to be aware for which host
> > Flink must process some task. This post shows (using Apache Storm) the
> kind
> > of example I want to build (
> https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
> > ).
> >
> > I saw some related question on this post from 2016 (
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html
> )
> > and I wonder if there is something more actual or better explained about
> a
> > custom scheduler.
> >
> > Thanks,
> > Felipe
> > *--*
> > *-- Felipe Gutierrez*
> >
> > *-- skype: felipe.o.gutierrez*
> > *--* *https://felipeogutierrez.blogspot.com<
> https://felipeogutierrez.blogspot.com> <
> https://felipeogutierrez.blogspot.com>*
> >
> >
> >
> >
>
Re: Custom scheduler in Flink
Posted by Felipe Gutierrez <fe...@gmail.com>.
Thanks, I'll check it out.
*--*
*-- Felipe Gutierrez*
*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*
On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <ch...@apache.org> wrote:
> There's no *reasonable *way to implement a custom Scheduler, i.e.,
> something where can just plug in your scheduler in a nice way. For this
> you'll have to directly modify the source of Flink.
>
> The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be
> of interest, but is still in the design phase.
>
> On 28.11.2018 10:21, Felipe Gutierrez wrote:
>
> Hi,
>
> I want to develop a custom scheduler in Flink to be aware for which host
> Flink must process some task. This post shows (using Apache Storm) the kind
> of example I want to build (https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
> ).
>
> I saw some related question on this post from 2016 (http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html)
> and I wonder if there is something more actual or better explained about a
> custom scheduler.
>
> Thanks,
> Felipe
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com<https://felipeogutierrez.blogspot.com> <https://felipeogutierrez.blogspot.com>*
>
>
>
>
Re: Custom scheduler in Flink
Posted by Felipe Gutierrez <fe...@gmail.com>.
Thanks, I'll check it out.
*--*
*-- Felipe Gutierrez*
*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*
On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler <ch...@apache.org> wrote:
> There's no *reasonable *way to implement a custom Scheduler, i.e.,
> something where can just plug in your scheduler in a nice way. For this
> you'll have to directly modify the source of Flink.
>
> The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be
> of interest, but is still in the design phase.
>
> On 28.11.2018 10:21, Felipe Gutierrez wrote:
>
> Hi,
>
> I want to develop a custom scheduler in Flink to be aware for which host
> Flink must process some task. This post shows (using Apache Storm) the kind
> of example I want to build (https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
> ).
>
> I saw some related question on this post from 2016 (http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html)
> and I wonder if there is something more actual or better explained about a
> custom scheduler.
>
> Thanks,
> Felipe
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com<https://felipeogutierrez.blogspot.com> <https://felipeogutierrez.blogspot.com>*
>
>
>
>
Re: Custom scheduler in Flink
Posted by Chesnay Schepler <ch...@apache.org>.
There's no /reasonable /way to implement a custom Scheduler, i.e.,
something where can just plug in your scheduler in a nice way. For this
you'll have to directly modify the source of Flink.
The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be
of interest, but is still in the design phase.
On 28.11.2018 10:21, Felipe Gutierrez wrote:
> Hi,
>
> I want to develop a custom scheduler in Flink to be aware for which host
> Flink must process some task. This post shows (using Apache Storm) the kind
> of example I want to build (
> https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
> ).
>
> I saw some related question on this post from 2016 (
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html)
> and I wonder if there is something more actual or better explained about a
> custom scheduler.
>
> Thanks,
> Felipe
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>
Re: Custom scheduler in Flink
Posted by Chesnay Schepler <ch...@apache.org>.
There's no /reasonable /way to implement a custom Scheduler, i.e.,
something where can just plug in your scheduler in a nice way. For this
you'll have to directly modify the source of Flink.
The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be
of interest, but is still in the design phase.
On 28.11.2018 10:21, Felipe Gutierrez wrote:
> Hi,
>
> I want to develop a custom scheduler in Flink to be aware for which host
> Flink must process some task. This post shows (using Apache Storm) the kind
> of example I want to build (
> https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e
> ).
>
> I saw some related question on this post from 2016 (
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html)
> and I wonder if there is something more actual or better explained about a
> custom scheduler.
>
> Thanks,
> Felipe
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>