You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Stephan Ewen <se...@apache.org> on 2016/01/08 16:43:31 UTC
Dripping the Flink-on-Tez code for Flink 1.0
Hi all!
Currently, Flink has a module to run batch program code on Tez rather than
Flink's own distributed execution engine.
I would suggest that we drop this code for the next release (1.0) as part
of a code consolidation:
- There seems little in both the Flink and the Tez community to use and
expand this functionality.
- The original motivation (better exploit resource elasticity in YARN)
will no longer be valid in the near future. I am re-working the YARN
integration currently to make it more elastic and make it possible to run
Flink on Mesos.
- The Flink-on-Tez code is rather POC status. Large scale testing it and
making adding all missing features will take more effort than making
Flink's own YARN integration resource elastic.
Please let me know what you think!
Especially @Kostas, since you wrote the initial POC, I'd be interested in
your opinion.
Greetings,
Stephan
Re: Dripping the Flink-on-Tez code for Flink 1.0
Posted by Ufuk Celebi <uc...@apache.org>.
I’ve created an issue [1] and closed all other Tez issues.
[1] https://issues.apache.org/jira/browse/FLINK-3235
> On 11 Jan 2016, at 02:52, Henry Saputra <he...@gmail.com> wrote:
>
> +1
>
> I am always for simplifying our code base when possible.
>
>
>
> On Sunday, January 10, 2016, Stephan Ewen <se...@apache.org> wrote:
>
>> I was typing with fat fingers again. Meant "dropping code" of course, not
>> "dripping" ;-)
>>
>> On Sun, Jan 10, 2016 at 11:55 AM, Fabian Hueske <fhueske@gmail.com
>> <javascript:;>> wrote:
>>
>>> +1
>>>
>>> 2016-01-08 17:35 GMT+01:00 Till Rohrmann <trohrmann@apache.org
>> <javascript:;>>:
>>>
>>>> +1 since it increase maintainability of the code base if it is not
>> really
>>>> used and thus removed.
>>>>
>>>> On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi <uce@apache.org
>> <javascript:;>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> I wanted to make a similar proposal.
>>>>>
>>>>> – Ufuk
>>>>>
>>>>>> On 08 Jan 2016, at 17:03, Kostas Tzoumas <ktzoumas@apache.org
>> <javascript:;>>
>>> wrote:
>>>>>>
>>>>>> for clarification, I was talking about dropping the code, I am
>> unsure
>>>>> about
>>>>>> the consequences of dripping code :-)
>>>>>>
>>>>>> On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <
>> ktzoumas@apache.org <javascript:;>>
>>>>> wrote:
>>>>>>
>>>>>>> +1 from my side
>>>>>>>
>>>>>>> Flink on Tez never got a lot of user traction. It served well as a
>>>>>>> prototype of "this is possible", but since the core functionality
>>> will
>>>>> be
>>>>>>> subsumed by making Flink on YARN resource elastic, I don't see any
>>>>> reason
>>>>>>> we should have it as part of the Flink codebase.
>>>>>>>
>>>>>>> Best,
>>>>>>> Kostas
>>>>>>>
>>>>>>> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <sewen@apache.org
>> <javascript:;>>
>>>> wrote:
>>>>>>>
>>>>>>>> Hi all!
>>>>>>>>
>>>>>>>> Currently, Flink has a module to run batch program code on Tez
>>> rather
>>>>> than
>>>>>>>> Flink's own distributed execution engine.
>>>>>>>>
>>>>>>>> I would suggest that we drop this code for the next release (1.0)
>>> as
>>>>> part
>>>>>>>> of a code consolidation:
>>>>>>>>
>>>>>>>> - There seems little in both the Flink and the Tez community to
>>> use
>>>>> and
>>>>>>>> expand this functionality.
>>>>>>>>
>>>>>>>> - The original motivation (better exploit resource elasticity in
>>>> YARN)
>>>>>>>> will no longer be valid in the near future. I am re-working the
>>> YARN
>>>>>>>> integration currently to make it more elastic and make it
>> possible
>>> to
>>>>> run
>>>>>>>> Flink on Mesos.
>>>>>>>>
>>>>>>>> - The Flink-on-Tez code is rather POC status. Large scale
>> testing
>>> it
>>>>> and
>>>>>>>> making adding all missing features will take more effort than
>>> making
>>>>>>>> Flink's own YARN integration resource elastic.
>>>>>>>>
>>>>>>>>
>>>>>>>> Please let me know what you think!
>>>>>>>> Especially @Kostas, since you wrote the initial POC, I'd be
>>>> interested
>>>>> in
>>>>>>>> your opinion.
>>>>>>>>
>>>>>>>> Greetings,
>>>>>>>> Stephan
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: Dripping the Flink-on-Tez code for Flink 1.0
Posted by Henry Saputra <he...@gmail.com>.
+1
I am always for simplifying our code base when possible.
On Sunday, January 10, 2016, Stephan Ewen <se...@apache.org> wrote:
> I was typing with fat fingers again. Meant "dropping code" of course, not
> "dripping" ;-)
>
> On Sun, Jan 10, 2016 at 11:55 AM, Fabian Hueske <fhueske@gmail.com
> <javascript:;>> wrote:
>
> > +1
> >
> > 2016-01-08 17:35 GMT+01:00 Till Rohrmann <trohrmann@apache.org
> <javascript:;>>:
> >
> > > +1 since it increase maintainability of the code base if it is not
> really
> > > used and thus removed.
> > >
> > > On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi <uce@apache.org
> <javascript:;>> wrote:
> > >
> > > > +1
> > > >
> > > > I wanted to make a similar proposal.
> > > >
> > > > – Ufuk
> > > >
> > > > > On 08 Jan 2016, at 17:03, Kostas Tzoumas <ktzoumas@apache.org
> <javascript:;>>
> > wrote:
> > > > >
> > > > > for clarification, I was talking about dropping the code, I am
> unsure
> > > > about
> > > > > the consequences of dripping code :-)
> > > > >
> > > > > On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <
> ktzoumas@apache.org <javascript:;>>
> > > > wrote:
> > > > >
> > > > >> +1 from my side
> > > > >>
> > > > >> Flink on Tez never got a lot of user traction. It served well as a
> > > > >> prototype of "this is possible", but since the core functionality
> > will
> > > > be
> > > > >> subsumed by making Flink on YARN resource elastic, I don't see any
> > > > reason
> > > > >> we should have it as part of the Flink codebase.
> > > > >>
> > > > >> Best,
> > > > >> Kostas
> > > > >>
> > > > >> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <sewen@apache.org
> <javascript:;>>
> > > wrote:
> > > > >>
> > > > >>> Hi all!
> > > > >>>
> > > > >>> Currently, Flink has a module to run batch program code on Tez
> > rather
> > > > than
> > > > >>> Flink's own distributed execution engine.
> > > > >>>
> > > > >>> I would suggest that we drop this code for the next release (1.0)
> > as
> > > > part
> > > > >>> of a code consolidation:
> > > > >>>
> > > > >>> - There seems little in both the Flink and the Tez community to
> > use
> > > > and
> > > > >>> expand this functionality.
> > > > >>>
> > > > >>> - The original motivation (better exploit resource elasticity in
> > > YARN)
> > > > >>> will no longer be valid in the near future. I am re-working the
> > YARN
> > > > >>> integration currently to make it more elastic and make it
> possible
> > to
> > > > run
> > > > >>> Flink on Mesos.
> > > > >>>
> > > > >>> - The Flink-on-Tez code is rather POC status. Large scale
> testing
> > it
> > > > and
> > > > >>> making adding all missing features will take more effort than
> > making
> > > > >>> Flink's own YARN integration resource elastic.
> > > > >>>
> > > > >>>
> > > > >>> Please let me know what you think!
> > > > >>> Especially @Kostas, since you wrote the initial POC, I'd be
> > > interested
> > > > in
> > > > >>> your opinion.
> > > > >>>
> > > > >>> Greetings,
> > > > >>> Stephan
> > > > >>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>
Re: Dripping the Flink-on-Tez code for Flink 1.0
Posted by Stephan Ewen <se...@apache.org>.
I was typing with fat fingers again. Meant "dropping code" of course, not
"dripping" ;-)
On Sun, Jan 10, 2016 at 11:55 AM, Fabian Hueske <fh...@gmail.com> wrote:
> +1
>
> 2016-01-08 17:35 GMT+01:00 Till Rohrmann <tr...@apache.org>:
>
> > +1 since it increase maintainability of the code base if it is not really
> > used and thus removed.
> >
> > On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > > +1
> > >
> > > I wanted to make a similar proposal.
> > >
> > > – Ufuk
> > >
> > > > On 08 Jan 2016, at 17:03, Kostas Tzoumas <kt...@apache.org>
> wrote:
> > > >
> > > > for clarification, I was talking about dropping the code, I am unsure
> > > about
> > > > the consequences of dripping code :-)
> > > >
> > > > On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <kt...@apache.org>
> > > wrote:
> > > >
> > > >> +1 from my side
> > > >>
> > > >> Flink on Tez never got a lot of user traction. It served well as a
> > > >> prototype of "this is possible", but since the core functionality
> will
> > > be
> > > >> subsumed by making Flink on YARN resource elastic, I don't see any
> > > reason
> > > >> we should have it as part of the Flink codebase.
> > > >>
> > > >> Best,
> > > >> Kostas
> > > >>
> > > >> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <se...@apache.org>
> > wrote:
> > > >>
> > > >>> Hi all!
> > > >>>
> > > >>> Currently, Flink has a module to run batch program code on Tez
> rather
> > > than
> > > >>> Flink's own distributed execution engine.
> > > >>>
> > > >>> I would suggest that we drop this code for the next release (1.0)
> as
> > > part
> > > >>> of a code consolidation:
> > > >>>
> > > >>> - There seems little in both the Flink and the Tez community to
> use
> > > and
> > > >>> expand this functionality.
> > > >>>
> > > >>> - The original motivation (better exploit resource elasticity in
> > YARN)
> > > >>> will no longer be valid in the near future. I am re-working the
> YARN
> > > >>> integration currently to make it more elastic and make it possible
> to
> > > run
> > > >>> Flink on Mesos.
> > > >>>
> > > >>> - The Flink-on-Tez code is rather POC status. Large scale testing
> it
> > > and
> > > >>> making adding all missing features will take more effort than
> making
> > > >>> Flink's own YARN integration resource elastic.
> > > >>>
> > > >>>
> > > >>> Please let me know what you think!
> > > >>> Especially @Kostas, since you wrote the initial POC, I'd be
> > interested
> > > in
> > > >>> your opinion.
> > > >>>
> > > >>> Greetings,
> > > >>> Stephan
> > > >>>
> > > >>
> > > >>
> > >
> > >
> >
>
Re: Dripping the Flink-on-Tez code for Flink 1.0
Posted by Fabian Hueske <fh...@gmail.com>.
+1
2016-01-08 17:35 GMT+01:00 Till Rohrmann <tr...@apache.org>:
> +1 since it increase maintainability of the code base if it is not really
> used and thus removed.
>
> On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > +1
> >
> > I wanted to make a similar proposal.
> >
> > – Ufuk
> >
> > > On 08 Jan 2016, at 17:03, Kostas Tzoumas <kt...@apache.org> wrote:
> > >
> > > for clarification, I was talking about dropping the code, I am unsure
> > about
> > > the consequences of dripping code :-)
> > >
> > > On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <kt...@apache.org>
> > wrote:
> > >
> > >> +1 from my side
> > >>
> > >> Flink on Tez never got a lot of user traction. It served well as a
> > >> prototype of "this is possible", but since the core functionality will
> > be
> > >> subsumed by making Flink on YARN resource elastic, I don't see any
> > reason
> > >> we should have it as part of the Flink codebase.
> > >>
> > >> Best,
> > >> Kostas
> > >>
> > >> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <se...@apache.org>
> wrote:
> > >>
> > >>> Hi all!
> > >>>
> > >>> Currently, Flink has a module to run batch program code on Tez rather
> > than
> > >>> Flink's own distributed execution engine.
> > >>>
> > >>> I would suggest that we drop this code for the next release (1.0) as
> > part
> > >>> of a code consolidation:
> > >>>
> > >>> - There seems little in both the Flink and the Tez community to use
> > and
> > >>> expand this functionality.
> > >>>
> > >>> - The original motivation (better exploit resource elasticity in
> YARN)
> > >>> will no longer be valid in the near future. I am re-working the YARN
> > >>> integration currently to make it more elastic and make it possible to
> > run
> > >>> Flink on Mesos.
> > >>>
> > >>> - The Flink-on-Tez code is rather POC status. Large scale testing it
> > and
> > >>> making adding all missing features will take more effort than making
> > >>> Flink's own YARN integration resource elastic.
> > >>>
> > >>>
> > >>> Please let me know what you think!
> > >>> Especially @Kostas, since you wrote the initial POC, I'd be
> interested
> > in
> > >>> your opinion.
> > >>>
> > >>> Greetings,
> > >>> Stephan
> > >>>
> > >>
> > >>
> >
> >
>
Re: Dripping the Flink-on-Tez code for Flink 1.0
Posted by Till Rohrmann <tr...@apache.org>.
+1 since it increase maintainability of the code base if it is not really
used and thus removed.
On Fri, Jan 8, 2016 at 5:33 PM, Ufuk Celebi <uc...@apache.org> wrote:
> +1
>
> I wanted to make a similar proposal.
>
> – Ufuk
>
> > On 08 Jan 2016, at 17:03, Kostas Tzoumas <kt...@apache.org> wrote:
> >
> > for clarification, I was talking about dropping the code, I am unsure
> about
> > the consequences of dripping code :-)
> >
> > On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <kt...@apache.org>
> wrote:
> >
> >> +1 from my side
> >>
> >> Flink on Tez never got a lot of user traction. It served well as a
> >> prototype of "this is possible", but since the core functionality will
> be
> >> subsumed by making Flink on YARN resource elastic, I don't see any
> reason
> >> we should have it as part of the Flink codebase.
> >>
> >> Best,
> >> Kostas
> >>
> >> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
> >>
> >>> Hi all!
> >>>
> >>> Currently, Flink has a module to run batch program code on Tez rather
> than
> >>> Flink's own distributed execution engine.
> >>>
> >>> I would suggest that we drop this code for the next release (1.0) as
> part
> >>> of a code consolidation:
> >>>
> >>> - There seems little in both the Flink and the Tez community to use
> and
> >>> expand this functionality.
> >>>
> >>> - The original motivation (better exploit resource elasticity in YARN)
> >>> will no longer be valid in the near future. I am re-working the YARN
> >>> integration currently to make it more elastic and make it possible to
> run
> >>> Flink on Mesos.
> >>>
> >>> - The Flink-on-Tez code is rather POC status. Large scale testing it
> and
> >>> making adding all missing features will take more effort than making
> >>> Flink's own YARN integration resource elastic.
> >>>
> >>>
> >>> Please let me know what you think!
> >>> Especially @Kostas, since you wrote the initial POC, I'd be interested
> in
> >>> your opinion.
> >>>
> >>> Greetings,
> >>> Stephan
> >>>
> >>
> >>
>
>
Re: Dripping the Flink-on-Tez code for Flink 1.0
Posted by Ufuk Celebi <uc...@apache.org>.
+1
I wanted to make a similar proposal.
– Ufuk
> On 08 Jan 2016, at 17:03, Kostas Tzoumas <kt...@apache.org> wrote:
>
> for clarification, I was talking about dropping the code, I am unsure about
> the consequences of dripping code :-)
>
> On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <kt...@apache.org> wrote:
>
>> +1 from my side
>>
>> Flink on Tez never got a lot of user traction. It served well as a
>> prototype of "this is possible", but since the core functionality will be
>> subsumed by making Flink on YARN resource elastic, I don't see any reason
>> we should have it as part of the Flink codebase.
>>
>> Best,
>> Kostas
>>
>> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> Hi all!
>>>
>>> Currently, Flink has a module to run batch program code on Tez rather than
>>> Flink's own distributed execution engine.
>>>
>>> I would suggest that we drop this code for the next release (1.0) as part
>>> of a code consolidation:
>>>
>>> - There seems little in both the Flink and the Tez community to use and
>>> expand this functionality.
>>>
>>> - The original motivation (better exploit resource elasticity in YARN)
>>> will no longer be valid in the near future. I am re-working the YARN
>>> integration currently to make it more elastic and make it possible to run
>>> Flink on Mesos.
>>>
>>> - The Flink-on-Tez code is rather POC status. Large scale testing it and
>>> making adding all missing features will take more effort than making
>>> Flink's own YARN integration resource elastic.
>>>
>>>
>>> Please let me know what you think!
>>> Especially @Kostas, since you wrote the initial POC, I'd be interested in
>>> your opinion.
>>>
>>> Greetings,
>>> Stephan
>>>
>>
>>
Re: Dripping the Flink-on-Tez code for Flink 1.0
Posted by Kostas Tzoumas <kt...@apache.org>.
for clarification, I was talking about dropping the code, I am unsure about
the consequences of dripping code :-)
On Fri, Jan 8, 2016 at 4:57 PM, Kostas Tzoumas <kt...@apache.org> wrote:
> +1 from my side
>
> Flink on Tez never got a lot of user traction. It served well as a
> prototype of "this is possible", but since the core functionality will be
> subsumed by making Flink on YARN resource elastic, I don't see any reason
> we should have it as part of the Flink codebase.
>
> Best,
> Kostas
>
> On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi all!
>>
>> Currently, Flink has a module to run batch program code on Tez rather than
>> Flink's own distributed execution engine.
>>
>> I would suggest that we drop this code for the next release (1.0) as part
>> of a code consolidation:
>>
>> - There seems little in both the Flink and the Tez community to use and
>> expand this functionality.
>>
>> - The original motivation (better exploit resource elasticity in YARN)
>> will no longer be valid in the near future. I am re-working the YARN
>> integration currently to make it more elastic and make it possible to run
>> Flink on Mesos.
>>
>> - The Flink-on-Tez code is rather POC status. Large scale testing it and
>> making adding all missing features will take more effort than making
>> Flink's own YARN integration resource elastic.
>>
>>
>> Please let me know what you think!
>> Especially @Kostas, since you wrote the initial POC, I'd be interested in
>> your opinion.
>>
>> Greetings,
>> Stephan
>>
>
>
Re: Dripping the Flink-on-Tez code for Flink 1.0
Posted by Kostas Tzoumas <kt...@apache.org>.
+1 from my side
Flink on Tez never got a lot of user traction. It served well as a
prototype of "this is possible", but since the core functionality will be
subsumed by making Flink on YARN resource elastic, I don't see any reason
we should have it as part of the Flink codebase.
Best,
Kostas
On Fri, Jan 8, 2016 at 4:43 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi all!
>
> Currently, Flink has a module to run batch program code on Tez rather than
> Flink's own distributed execution engine.
>
> I would suggest that we drop this code for the next release (1.0) as part
> of a code consolidation:
>
> - There seems little in both the Flink and the Tez community to use and
> expand this functionality.
>
> - The original motivation (better exploit resource elasticity in YARN)
> will no longer be valid in the near future. I am re-working the YARN
> integration currently to make it more elastic and make it possible to run
> Flink on Mesos.
>
> - The Flink-on-Tez code is rather POC status. Large scale testing it and
> making adding all missing features will take more effort than making
> Flink's own YARN integration resource elastic.
>
>
> Please let me know what you think!
> Especially @Kostas, since you wrote the initial POC, I'd be interested in
> your opinion.
>
> Greetings,
> Stephan
>