You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Matthias Pohl <ma...@ververica.com> on 2021/03/25 11:22:34 UTC

Re: [BULK]Re: [SURVEY] Remove Mesos support

Hi everyone,
considering the upcoming release of Flink 1.13, I wanted to revive the
discussion about the Mesos support ones more. Mesos is also already listed
as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align the
documentation accordingly to make it more explicit?

What do you think?

Best,
Matthias

[1] https://flink.apache.org/roadmap.html#feature-radar

On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org> wrote:

> Hi Oleksandr,
>
> yes you are right. The biggest problem is at the moment the lack of test
> coverage and thereby confidence to make changes. We have some e2e tests
> which you can find here [1]. These tests are, however, quite coarse grained
> and are missing a lot of cases. One idea would be to add a Mesos e2e test
> based on Flink's end-to-end test framework [2]. I think what needs to be
> done there is to add a Mesos resource and a way to submit jobs to a Mesos
> cluster to write e2e tests.
>
> [1] https://github.com/apache/flink/tree/master/flink-jepsen
> [2]
> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
>
> Cheers,
> Till
>
> On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
> o.nitavskyi@criteo.com> wrote:
>
>> Hello Xintong,
>>
>> Thanks for the insights and support.
>>
>> Browsing the Mesos backlog and didn't identify anything critical, which
>> is left there.
>>
>> I see that there are were quite a lot of contributions to the Flink Mesos
>> in the recent version:
>> https://github.com/apache/flink/commits/master/flink-mesos.
>> We plan to validate the current Flink master (or release 1.12 branch) our
>> Mesos setup. In case of any issues, we will try to propose changes.
>> My feeling is that our test results shouldn't affect the Flink 1.12
>> release cycle. And if any potential commits will land into the 1.12.1 it
>> should be totally fine.
>>
>> In the future, we would be glad to help you guys with any
>> maintenance-related questions. One of the highest priorities around this
>> component seems to be the development of the full e2e test.
>>
>> Kind Regards
>> Oleksandr Nitavskyi
>> ________________________________
>> From: Xintong Song <to...@gmail.com>
>> Sent: Tuesday, October 27, 2020 7:14 AM
>> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
>> Cc: Piyush Narang <p....@criteo.com>
>> Subject: [BULK]Re: [SURVEY] Remove Mesos support
>>
>> Hi Piyush,
>>
>> Thanks a lot for sharing the information. It would be a great relief that
>> you are good with Flink on Mesos as is.
>>
>> As for the jira issues, I believe the most essential ones should have
>> already been resolved. You may find some remaining open issues here [1],
>> but not all of them are necessary if we decide to keep Flink on Mesos as is.
>>
>> At the moment and in the short future, I think helps are mostly needed on
>> testing the upcoming release 1.12 with Mesos use cases. The community is
>> currently actively preparing the new release, and hopefully we could come
>> up with a release candidate early next month. It would be greatly
>> appreciated if you fork as experienced Flink on Mesos users can help with
>> verifying the release candidates.
>>
>>
>> Thank you~
>>
>> Xintong Song
>>
>> [1]
>> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
>> <
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
>> >
>>
>> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
>> <ma...@criteo.com>> wrote:
>>
>> Hi Xintong,
>>
>>
>>
>> Do you have any jiras that cover any of the items on 1 or 2? I can reach
>> out to folks internally and see if I can get some folks to commit to
>> helping out.
>>
>>
>>
>> To cover the other qs:
>>
>>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We use
>> Yarn for some our Flink workloads when we can. Mesos is only used when we
>> need streaming capabilities in our WW dcs (as our Yarn is centralized in
>> one DC)
>>   *   We’re currently on Flink 1.9 (old planner). We have a plan to bump
>> to 1.11 / 1.12 this quarter.
>>   *   We typically upgrade once every 6 months to a year (not every
>> release). We’d like to speed up the cadence but we’re not there yet.
>>   *   We’d largely be good with keeping Flink on Mesos as-is and
>> functional while missing out on some of the newer features. We understand
>> the pain on the communities side and we can take on the work if we see some
>> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put in
>> the request to port it over.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> -- Piyush
>>
>>
>>
>>
>>
>> From: Xintong Song <to...@gmail.com>>
>> Date: Sunday, October 25, 2020 at 10:57 PM
>> To: dev <de...@flink.apache.org>>, user <
>> user@flink.apache.org<ma...@flink.apache.org>>
>> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
>> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
>> p.narang@criteo.com>>
>> Subject: Re: [SURVEY] Remove Mesos support
>>
>>
>>
>> Thanks for sharing the information with us, Piyush an Lasse.
>>
>>
>>
>> @Piyush
>>
>>
>>
>> Thanks for offering the help. IMO, there are currently several problems
>> that make supporting Flink on Mesos challenging for us.
>>
>>   1.  Lack of Mesos experts. AFAIK, there are very few people (if not
>> none) among the active contributors in this community that are familiar
>> with Mesos and can help with development on this component.
>>   2.  Absence of tests. Mesos does not provide a testing cluster, like
>> `MiniYARNCluster`, making it hard to test interactions between Flink and
>> Mesos. We have only a few very simple e2e tests running on Mesos deployed
>> in a docker, covering the most fundamental workflows. We are not sure how
>> well those tests work, especially against some potential corner cases.
>>   3.  Divergence from other deployment. Because of 1 and 2, the new
>> efforts (features, maintenance, refactors) tend to exclude Mesos if
>> possible. When the new efforts have to touch the Mesos related components
>> (e.g., changes to the common resource manager interfaces), we have to be
>> very careful and make as few changes as possible, to avoid accidentally
>> breaking anything that we are not familiar with. As a result, the component
>> diverges a lot from other deployment components (K8s/Yarn), which makes it
>> harder to maintain.
>>
>> It would be greatly appreciated if you can help with either of the above
>> issues.
>>
>>
>>
>> Additionally, I have a few questions concerning your use cases at Criteo.
>> IIUC, you are going to stay on Mesos in the foreseeable future, while
>> keeping the Flink version up-to-date? What Flink version are you currently
>> using? How often do you upgrade (e.g., every release)? Would you be good
>> with keeping the Flink on Mesos component as it is (means that deployment
>> and resource management improvements may not be ported to Mesos), while
>> keeping other components up-to-date (e.g., improvements from programming
>> APIs, operators, state backens, etc.)?
>>
>>
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>>
>>
>> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
>> lassenedergaardflink@gmail.com<ma...@gmail.com>>
>> wrote:
>>
>> Hi
>>
>>
>>
>> At Trackunit We have been using Mesos for long time but have now moved to
>> k8s.
>>
>> Med venlig hilsen / Best regards
>>
>> Lasse Nedergaard
>>
>>
>>
>>
>>
>> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetzger@apache.org
>> <ma...@apache.org>>:
>>
>> 
>>
>> Hey Piyush,
>>
>> thanks a lot for raising this concern. I believe we should keep Mesos in
>> Flink then in the foreseeable future.
>>
>> Your offer to help is much appreciated. We'll let you know once there is
>> something.
>>
>>
>>
>> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
>> <ma...@criteo.com>> wrote:
>>
>> Thanks Kostas. If there's items we can help with, I'm sure we'd be able
>> to find folks who would be excited to contribute / help in any way.
>>
>> -- Piyush
>>
>>
>> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com<mailto:
>> kkloudas@gmail.com>> wrote:
>>
>>     Thanks Piyush for the message.
>>     After this, I revoke my +1. I agree with the previous opinions that we
>>     cannot drop code that is actively used by users, especially if it
>>     something that deep in the stack as support for cluster management
>>     framework.
>>
>>     Cheers,
>>     Kostas
>>
>>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <p.narang@criteo.com
>> <ma...@criteo.com>> wrote:
>>     >
>>     > Hi folks,
>>     >
>>     >
>>     >
>>     > We at Criteo are active users of the Flink on Mesos resource
>> management component. We are pretty heavy users of Mesos for scheduling
>> workloads on our edge datacenters and we do want to continue to be able to
>> run some of our Flink topologies (to compute machine learning short term
>> features) on those DCs. If possible our vote would be not to drop Mesos
>> support as that will tie us to an old release / have to maintain a fork as
>> we’re not planning to migrate off Mesos anytime soon. Is the burden
>> something that can be helped with by the community? (Or are you referring
>> to having to ensure PRs handle the Mesos piece as well when they touch the
>> resource managers?)
>>     >
>>     >
>>     >
>>     > Thanks,
>>     >
>>     >
>>     >
>>     > -- Piyush
>>     >
>>     >
>>     >
>>     >
>>     >
>>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
>> trohrmann@apache.org>>
>>     > Date: Friday, October 23, 2020 at 8:19 AM
>>     > To: Xintong Song <tonysong820@gmail.com<mailto:
>> tonysong820@gmail.com>>
>>     > Cc: dev <de...@flink.apache.org>>, user <
>> user@flink.apache.org<ma...@flink.apache.org>>
>>     > Subject: Re: [SURVEY] Remove Mesos support
>>     >
>>     >
>>     >
>>     > Thanks for starting this survey Robert! I second Konstantin and
>> Xintong in the sense that our Mesos user's opinions should matter most
>> here. If our community is no longer using the Mesos integration, then I
>> would be +1 for removing it in order to decrease the maintenance burden.
>>     >
>>     >
>>     >
>>     > Cheers,
>>     >
>>     > Till
>>     >
>>     >
>>     >
>>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <tonysong820@gmail.com
>> <ma...@gmail.com>> wrote:
>>     >
>>     > +1 for adding a warning in 1.12 about planning to remove Mesos
>> support.
>>     >
>>     >
>>     >
>>     > With my developer hat on, removing the Mesos support would
>> definitely reduce the maintaining overhead for the deployment and resource
>> management related components. On the other hand, the Flink on Mesos users'
>> voices definitely matter a lot for this community. Either way, it would be
>> good to draw users attention to this discussion early.
>>     >
>>     >
>>     >
>>     > Thank you~
>>     >
>>     > Xintong Song
>>     >
>>     >
>>     >
>>     >
>>     >
>>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <knaufk@apache.org
>> <ma...@apache.org>> wrote:
>>     >
>>     > Hi Robert,
>>     >
>>     > +1 to the plan you outlined. If we were to drop support in Flink
>> 1.13+, we
>>     > would still support it in Flink 1.12- with bug fixes for some time
>> so that
>>     > users have time to move on.
>>     >
>>     > It would certainly be very interesting to hear from current Flink
>> on Mesos
>>     > users, on how they see the evolution of this part of the ecosystem.
>>     >
>>     > Best,
>>     >
>>     > Konstantin
>>
>

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Matthias Pohl <ma...@ververica.com>.
Thanks for everyone's feedback. I'm gonna initiate a vote in a separate
thread.

On Mon, Mar 29, 2021 at 9:18 AM Robert Metzger <rm...@apache.org> wrote:

> +1
>
>
>
> On Mon, Mar 29, 2021 at 5:44 AM Yangze Guo <ka...@gmail.com> wrote:
>
> > +1
> >
> > Best,
> > Yangze Guo
> >
> > On Mon, Mar 29, 2021 at 11:31 AM Xintong Song <to...@gmail.com>
> > wrote:
> > >
> > > +1
> > > It's already a matter of fact for a while that we no longer port new
> > features to the Mesos deployment.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann <tr...@apache.org>
> > wrote:
> > >>
> > >> +1 for officially deprecating this component for the 1.13 release.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kn...@apache.org>
> > wrote:
> > >>>
> > >>> Hi Matthias,
> > >>>
> > >>> Thank you for following up on this. +1 to officially deprecate Mesos
> > in the code and documentation, too. It will be confusing for users if
> this
> > diverges from the roadmap.
> > >>>
> > >>> Cheers,
> > >>>
> > >>> Konstantin
> > >>>
> > >>> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <
> matthias@ververica.com>
> > wrote:
> > >>>>
> > >>>> Hi everyone,
> > >>>> considering the upcoming release of Flink 1.13, I wanted to revive
> the
> > >>>> discussion about the Mesos support ones more. Mesos is also already
> > listed
> > >>>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to
> > align the
> > >>>> documentation accordingly to make it more explicit?
> > >>>>
> > >>>> What do you think?
> > >>>>
> > >>>> Best,
> > >>>> Matthias
> > >>>>
> > >>>> [1] https://flink.apache.org/roadmap.html#feature-radar
> > >>>>
> > >>>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <trohrmann@apache.org
> >
> > wrote:
> > >>>>
> > >>>> > Hi Oleksandr,
> > >>>> >
> > >>>> > yes you are right. The biggest problem is at the moment the lack
> of
> > test
> > >>>> > coverage and thereby confidence to make changes. We have some e2e
> > tests
> > >>>> > which you can find here [1]. These tests are, however, quite
> coarse
> > grained
> > >>>> > and are missing a lot of cases. One idea would be to add a Mesos
> > e2e test
> > >>>> > based on Flink's end-to-end test framework [2]. I think what needs
> > to be
> > >>>> > done there is to add a Mesos resource and a way to submit jobs to
> a
> > Mesos
> > >>>> > cluster to write e2e tests.
> > >>>> >
> > >>>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
> > >>>> > [2]
> > >>>> >
> >
> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
> > >>>> >
> > >>>> > Cheers,
> > >>>> > Till
> > >>>> >
> > >>>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
> > >>>> > o.nitavskyi@criteo.com> wrote:
> > >>>> >
> > >>>> >> Hello Xintong,
> > >>>> >>
> > >>>> >> Thanks for the insights and support.
> > >>>> >>
> > >>>> >> Browsing the Mesos backlog and didn't identify anything critical,
> > which
> > >>>> >> is left there.
> > >>>> >>
> > >>>> >> I see that there are were quite a lot of contributions to the
> > Flink Mesos
> > >>>> >> in the recent version:
> > >>>> >> https://github.com/apache/flink/commits/master/flink-mesos.
> > >>>> >> We plan to validate the current Flink master (or release 1.12
> > branch) our
> > >>>> >> Mesos setup. In case of any issues, we will try to propose
> changes.
> > >>>> >> My feeling is that our test results shouldn't affect the Flink
> 1.12
> > >>>> >> release cycle. And if any potential commits will land into the
> > 1.12.1 it
> > >>>> >> should be totally fine.
> > >>>> >>
> > >>>> >> In the future, we would be glad to help you guys with any
> > >>>> >> maintenance-related questions. One of the highest priorities
> > around this
> > >>>> >> component seems to be the development of the full e2e test.
> > >>>> >>
> > >>>> >> Kind Regards
> > >>>> >> Oleksandr Nitavskyi
> > >>>> >> ________________________________
> > >>>> >> From: Xintong Song <to...@gmail.com>
> > >>>> >> Sent: Tuesday, October 27, 2020 7:14 AM
> > >>>> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
> > >>>> >> Cc: Piyush Narang <p....@criteo.com>
> > >>>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
> > >>>> >>
> > >>>> >> Hi Piyush,
> > >>>> >>
> > >>>> >> Thanks a lot for sharing the information. It would be a great
> > relief that
> > >>>> >> you are good with Flink on Mesos as is.
> > >>>> >>
> > >>>> >> As for the jira issues, I believe the most essential ones should
> > have
> > >>>> >> already been resolved. You may find some remaining open issues
> > here [1],
> > >>>> >> but not all of them are necessary if we decide to keep Flink on
> > Mesos as is.
> > >>>> >>
> > >>>> >> At the moment and in the short future, I think helps are mostly
> > needed on
> > >>>> >> testing the upcoming release 1.12 with Mesos use cases. The
> > community is
> > >>>> >> currently actively preparing the new release, and hopefully we
> > could come
> > >>>> >> up with a release candidate early next month. It would be greatly
> > >>>> >> appreciated if you fork as experienced Flink on Mesos users can
> > help with
> > >>>> >> verifying the release candidates.
> > >>>> >>
> > >>>> >>
> > >>>> >> Thank you~
> > >>>> >>
> > >>>> >> Xintong Song
> > >>>> >>
> > >>>> >> [1]
> > >>>> >>
> >
> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
> > >>>> >> <
> > >>>> >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
> > >>>> >> >
> > >>>> >>
> > >>>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <
> p.narang@criteo.com
> > >>>> >> <ma...@criteo.com>> wrote:
> > >>>> >>
> > >>>> >> Hi Xintong,
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Do you have any jiras that cover any of the items on 1 or 2? I
> can
> > reach
> > >>>> >> out to folks internally and see if I can get some folks to commit
> > to
> > >>>> >> helping out.
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> To cover the other qs:
> > >>>> >>
> > >>>> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos.
> We
> > use
> > >>>> >> Yarn for some our Flink workloads when we can. Mesos is only used
> > when we
> > >>>> >> need streaming capabilities in our WW dcs (as our Yarn is
> > centralized in
> > >>>> >> one DC)
> > >>>> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan
> > to bump
> > >>>> >> to 1.11 / 1.12 this quarter.
> > >>>> >>   *   We typically upgrade once every 6 months to a year (not
> every
> > >>>> >> release). We’d like to speed up the cadence but we’re not there
> > yet.
> > >>>> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
> > >>>> >> functional while missing out on some of the newer features. We
> > understand
> > >>>> >> the pain on the communities side and we can take on the work if
> we
> > see some
> > >>>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to
> > put in
> > >>>> >> the request to port it over.
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Thanks,
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> -- Piyush
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> From: Xintong Song <tonysong820@gmail.com<mailto:
> > tonysong820@gmail.com>>
> > >>>> >> Date: Sunday, October 25, 2020 at 10:57 PM
> > >>>> >> To: dev <de...@flink.apache.org>>,
> user
> > <
> > >>>> >> user@flink.apache.org<ma...@flink.apache.org>>
> > >>>> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
> > >>>> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
> > >>>> >> p.narang@criteo.com>>
> > >>>> >> Subject: Re: [SURVEY] Remove Mesos support
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Thanks for sharing the information with us, Piyush an Lasse.
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> @Piyush
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Thanks for offering the help. IMO, there are currently several
> > problems
> > >>>> >> that make supporting Flink on Mesos challenging for us.
> > >>>> >>
> > >>>> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if
> > not
> > >>>> >> none) among the active contributors in this community that are
> > familiar
> > >>>> >> with Mesos and can help with development on this component.
> > >>>> >>   2.  Absence of tests. Mesos does not provide a testing cluster,
> > like
> > >>>> >> `MiniYARNCluster`, making it hard to test interactions between
> > Flink and
> > >>>> >> Mesos. We have only a few very simple e2e tests running on Mesos
> > deployed
> > >>>> >> in a docker, covering the most fundamental workflows. We are not
> > sure how
> > >>>> >> well those tests work, especially against some potential corner
> > cases.
> > >>>> >>   3.  Divergence from other deployment. Because of 1 and 2, the
> new
> > >>>> >> efforts (features, maintenance, refactors) tend to exclude Mesos
> if
> > >>>> >> possible. When the new efforts have to touch the Mesos related
> > components
> > >>>> >> (e.g., changes to the common resource manager interfaces), we
> have
> > to be
> > >>>> >> very careful and make as few changes as possible, to avoid
> > accidentally
> > >>>> >> breaking anything that we are not familiar with. As a result, the
> > component
> > >>>> >> diverges a lot from other deployment components (K8s/Yarn), which
> > makes it
> > >>>> >> harder to maintain.
> > >>>> >>
> > >>>> >> It would be greatly appreciated if you can help with either of
> the
> > above
> > >>>> >> issues.
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Additionally, I have a few questions concerning your use cases at
> > Criteo.
> > >>>> >> IIUC, you are going to stay on Mesos in the foreseeable future,
> > while
> > >>>> >> keeping the Flink version up-to-date? What Flink version are you
> > currently
> > >>>> >> using? How often do you upgrade (e.g., every release)? Would you
> > be good
> > >>>> >> with keeping the Flink on Mesos component as it is (means that
> > deployment
> > >>>> >> and resource management improvements may not be ported to Mesos),
> > while
> > >>>> >> keeping other components up-to-date (e.g., improvements from
> > programming
> > >>>> >> APIs, operators, state backens, etc.)?
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Thank you~
> > >>>> >>
> > >>>> >> Xintong Song
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
> > >>>> >> lassenedergaardflink@gmail.com<mailto:
> > lassenedergaardflink@gmail.com>>
> > >>>> >> wrote:
> > >>>> >>
> > >>>> >> Hi
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> At Trackunit We have been using Mesos for long time but have now
> > moved to
> > >>>> >> k8s.
> > >>>> >>
> > >>>> >> Med venlig hilsen / Best regards
> > >>>> >>
> > >>>> >> Lasse Nedergaard
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <
> > rmetzger@apache.org
> > >>>> >> <ma...@apache.org>>:
> > >>>> >>
> > >>>> >> 
> > >>>> >>
> > >>>> >> Hey Piyush,
> > >>>> >>
> > >>>> >> thanks a lot for raising this concern. I believe we should keep
> > Mesos in
> > >>>> >> Flink then in the foreseeable future.
> > >>>> >>
> > >>>> >> Your offer to help is much appreciated. We'll let you know once
> > there is
> > >>>> >> something.
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <
> p.narang@criteo.com
> > >>>> >> <ma...@criteo.com>> wrote:
> > >>>> >>
> > >>>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd
> be
> > able
> > >>>> >> to find folks who would be excited to contribute / help in any
> way.
> > >>>> >>
> > >>>> >> -- Piyush
> > >>>> >>
> > >>>> >>
> > >>>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com
> > <mailto:
> > >>>> >> kkloudas@gmail.com>> wrote:
> > >>>> >>
> > >>>> >>     Thanks Piyush for the message.
> > >>>> >>     After this, I revoke my +1. I agree with the previous
> opinions
> > that we
> > >>>> >>     cannot drop code that is actively used by users, especially
> if
> > it
> > >>>> >>     something that deep in the stack as support for cluster
> > management
> > >>>> >>     framework.
> > >>>> >>
> > >>>> >>     Cheers,
> > >>>> >>     Kostas
> > >>>> >>
> > >>>> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <
> > p.narang@criteo.com
> > >>>> >> <ma...@criteo.com>> wrote:
> > >>>> >>     >
> > >>>> >>     > Hi folks,
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > We at Criteo are active users of the Flink on Mesos
> resource
> > >>>> >> management component. We are pretty heavy users of Mesos for
> > scheduling
> > >>>> >> workloads on our edge datacenters and we do want to continue to
> be
> > able to
> > >>>> >> run some of our Flink topologies (to compute machine learning
> > short term
> > >>>> >> features) on those DCs. If possible our vote would be not to drop
> > Mesos
> > >>>> >> support as that will tie us to an old release / have to maintain
> a
> > fork as
> > >>>> >> we’re not planning to migrate off Mesos anytime soon. Is the
> burden
> > >>>> >> something that can be helped with by the community? (Or are you
> > referring
> > >>>> >> to having to ensure PRs handle the Mesos piece as well when they
> > touch the
> > >>>> >> resource managers?)
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > Thanks,
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > -- Piyush
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
> > >>>> >> trohrmann@apache.org>>
> > >>>> >>     > Date: Friday, October 23, 2020 at 8:19 AM
> > >>>> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
> > >>>> >> tonysong820@gmail.com>>
> > >>>> >>     > Cc: dev <dev@flink.apache.org<mailto:dev@flink.apache.org
> >>,
> > user <
> > >>>> >> user@flink.apache.org<ma...@flink.apache.org>>
> > >>>> >>     > Subject: Re: [SURVEY] Remove Mesos support
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > Thanks for starting this survey Robert! I second Konstantin
> > and
> > >>>> >> Xintong in the sense that our Mesos user's opinions should matter
> > most
> > >>>> >> here. If our community is no longer using the Mesos integration,
> > then I
> > >>>> >> would be +1 for removing it in order to decrease the maintenance
> > burden.
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > Cheers,
> > >>>> >>     >
> > >>>> >>     > Till
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <
> > tonysong820@gmail.com
> > >>>> >> <ma...@gmail.com>> wrote:
> > >>>> >>     >
> > >>>> >>     > +1 for adding a warning in 1.12 about planning to remove
> > Mesos
> > >>>> >> support.
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > With my developer hat on, removing the Mesos support would
> > >>>> >> definitely reduce the maintaining overhead for the deployment and
> > resource
> > >>>> >> management related components. On the other hand, the Flink on
> > Mesos users'
> > >>>> >> voices definitely matter a lot for this community. Either way, it
> > would be
> > >>>> >> good to draw users attention to this discussion early.
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > Thank you~
> > >>>> >>     >
> > >>>> >>     > Xintong Song
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <
> > knaufk@apache.org
> > >>>> >> <ma...@apache.org>> wrote:
> > >>>> >>     >
> > >>>> >>     > Hi Robert,
> > >>>> >>     >
> > >>>> >>     > +1 to the plan you outlined. If we were to drop support in
> > Flink
> > >>>> >> 1.13+, we
> > >>>> >>     > would still support it in Flink 1.12- with bug fixes for
> > some time
> > >>>> >> so that
> > >>>> >>     > users have time to move on.
> > >>>> >>     >
> > >>>> >>     > It would certainly be very interesting to hear from current
> > Flink
> > >>>> >> on Mesos
> > >>>> >>     > users, on how they see the evolution of this part of the
> > ecosystem.
> > >>>> >>     >
> > >>>> >>     > Best,
> > >>>> >>     >
> > >>>> >>     > Konstantin
> > >>>> >>
> > >>>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> Konstantin Knauf
> > >>>
> > >>> https://twitter.com/snntrable
> > >>>
> > >>> https://github.com/knaufk
> >

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Matthias Pohl <ma...@ververica.com>.
Thanks for everyone's feedback. I'm gonna initiate a vote in a separate
thread.

On Mon, Mar 29, 2021 at 9:18 AM Robert Metzger <rm...@apache.org> wrote:

> +1
>
>
>
> On Mon, Mar 29, 2021 at 5:44 AM Yangze Guo <ka...@gmail.com> wrote:
>
> > +1
> >
> > Best,
> > Yangze Guo
> >
> > On Mon, Mar 29, 2021 at 11:31 AM Xintong Song <to...@gmail.com>
> > wrote:
> > >
> > > +1
> > > It's already a matter of fact for a while that we no longer port new
> > features to the Mesos deployment.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann <tr...@apache.org>
> > wrote:
> > >>
> > >> +1 for officially deprecating this component for the 1.13 release.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kn...@apache.org>
> > wrote:
> > >>>
> > >>> Hi Matthias,
> > >>>
> > >>> Thank you for following up on this. +1 to officially deprecate Mesos
> > in the code and documentation, too. It will be confusing for users if
> this
> > diverges from the roadmap.
> > >>>
> > >>> Cheers,
> > >>>
> > >>> Konstantin
> > >>>
> > >>> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <
> matthias@ververica.com>
> > wrote:
> > >>>>
> > >>>> Hi everyone,
> > >>>> considering the upcoming release of Flink 1.13, I wanted to revive
> the
> > >>>> discussion about the Mesos support ones more. Mesos is also already
> > listed
> > >>>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to
> > align the
> > >>>> documentation accordingly to make it more explicit?
> > >>>>
> > >>>> What do you think?
> > >>>>
> > >>>> Best,
> > >>>> Matthias
> > >>>>
> > >>>> [1] https://flink.apache.org/roadmap.html#feature-radar
> > >>>>
> > >>>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <trohrmann@apache.org
> >
> > wrote:
> > >>>>
> > >>>> > Hi Oleksandr,
> > >>>> >
> > >>>> > yes you are right. The biggest problem is at the moment the lack
> of
> > test
> > >>>> > coverage and thereby confidence to make changes. We have some e2e
> > tests
> > >>>> > which you can find here [1]. These tests are, however, quite
> coarse
> > grained
> > >>>> > and are missing a lot of cases. One idea would be to add a Mesos
> > e2e test
> > >>>> > based on Flink's end-to-end test framework [2]. I think what needs
> > to be
> > >>>> > done there is to add a Mesos resource and a way to submit jobs to
> a
> > Mesos
> > >>>> > cluster to write e2e tests.
> > >>>> >
> > >>>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
> > >>>> > [2]
> > >>>> >
> >
> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
> > >>>> >
> > >>>> > Cheers,
> > >>>> > Till
> > >>>> >
> > >>>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
> > >>>> > o.nitavskyi@criteo.com> wrote:
> > >>>> >
> > >>>> >> Hello Xintong,
> > >>>> >>
> > >>>> >> Thanks for the insights and support.
> > >>>> >>
> > >>>> >> Browsing the Mesos backlog and didn't identify anything critical,
> > which
> > >>>> >> is left there.
> > >>>> >>
> > >>>> >> I see that there are were quite a lot of contributions to the
> > Flink Mesos
> > >>>> >> in the recent version:
> > >>>> >> https://github.com/apache/flink/commits/master/flink-mesos.
> > >>>> >> We plan to validate the current Flink master (or release 1.12
> > branch) our
> > >>>> >> Mesos setup. In case of any issues, we will try to propose
> changes.
> > >>>> >> My feeling is that our test results shouldn't affect the Flink
> 1.12
> > >>>> >> release cycle. And if any potential commits will land into the
> > 1.12.1 it
> > >>>> >> should be totally fine.
> > >>>> >>
> > >>>> >> In the future, we would be glad to help you guys with any
> > >>>> >> maintenance-related questions. One of the highest priorities
> > around this
> > >>>> >> component seems to be the development of the full e2e test.
> > >>>> >>
> > >>>> >> Kind Regards
> > >>>> >> Oleksandr Nitavskyi
> > >>>> >> ________________________________
> > >>>> >> From: Xintong Song <to...@gmail.com>
> > >>>> >> Sent: Tuesday, October 27, 2020 7:14 AM
> > >>>> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
> > >>>> >> Cc: Piyush Narang <p....@criteo.com>
> > >>>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
> > >>>> >>
> > >>>> >> Hi Piyush,
> > >>>> >>
> > >>>> >> Thanks a lot for sharing the information. It would be a great
> > relief that
> > >>>> >> you are good with Flink on Mesos as is.
> > >>>> >>
> > >>>> >> As for the jira issues, I believe the most essential ones should
> > have
> > >>>> >> already been resolved. You may find some remaining open issues
> > here [1],
> > >>>> >> but not all of them are necessary if we decide to keep Flink on
> > Mesos as is.
> > >>>> >>
> > >>>> >> At the moment and in the short future, I think helps are mostly
> > needed on
> > >>>> >> testing the upcoming release 1.12 with Mesos use cases. The
> > community is
> > >>>> >> currently actively preparing the new release, and hopefully we
> > could come
> > >>>> >> up with a release candidate early next month. It would be greatly
> > >>>> >> appreciated if you fork as experienced Flink on Mesos users can
> > help with
> > >>>> >> verifying the release candidates.
> > >>>> >>
> > >>>> >>
> > >>>> >> Thank you~
> > >>>> >>
> > >>>> >> Xintong Song
> > >>>> >>
> > >>>> >> [1]
> > >>>> >>
> >
> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
> > >>>> >> <
> > >>>> >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
> > >>>> >> >
> > >>>> >>
> > >>>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <
> p.narang@criteo.com
> > >>>> >> <ma...@criteo.com>> wrote:
> > >>>> >>
> > >>>> >> Hi Xintong,
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Do you have any jiras that cover any of the items on 1 or 2? I
> can
> > reach
> > >>>> >> out to folks internally and see if I can get some folks to commit
> > to
> > >>>> >> helping out.
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> To cover the other qs:
> > >>>> >>
> > >>>> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos.
> We
> > use
> > >>>> >> Yarn for some our Flink workloads when we can. Mesos is only used
> > when we
> > >>>> >> need streaming capabilities in our WW dcs (as our Yarn is
> > centralized in
> > >>>> >> one DC)
> > >>>> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan
> > to bump
> > >>>> >> to 1.11 / 1.12 this quarter.
> > >>>> >>   *   We typically upgrade once every 6 months to a year (not
> every
> > >>>> >> release). We’d like to speed up the cadence but we’re not there
> > yet.
> > >>>> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
> > >>>> >> functional while missing out on some of the newer features. We
> > understand
> > >>>> >> the pain on the communities side and we can take on the work if
> we
> > see some
> > >>>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to
> > put in
> > >>>> >> the request to port it over.
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Thanks,
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> -- Piyush
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> From: Xintong Song <tonysong820@gmail.com<mailto:
> > tonysong820@gmail.com>>
> > >>>> >> Date: Sunday, October 25, 2020 at 10:57 PM
> > >>>> >> To: dev <de...@flink.apache.org>>,
> user
> > <
> > >>>> >> user@flink.apache.org<ma...@flink.apache.org>>
> > >>>> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
> > >>>> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
> > >>>> >> p.narang@criteo.com>>
> > >>>> >> Subject: Re: [SURVEY] Remove Mesos support
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Thanks for sharing the information with us, Piyush an Lasse.
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> @Piyush
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Thanks for offering the help. IMO, there are currently several
> > problems
> > >>>> >> that make supporting Flink on Mesos challenging for us.
> > >>>> >>
> > >>>> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if
> > not
> > >>>> >> none) among the active contributors in this community that are
> > familiar
> > >>>> >> with Mesos and can help with development on this component.
> > >>>> >>   2.  Absence of tests. Mesos does not provide a testing cluster,
> > like
> > >>>> >> `MiniYARNCluster`, making it hard to test interactions between
> > Flink and
> > >>>> >> Mesos. We have only a few very simple e2e tests running on Mesos
> > deployed
> > >>>> >> in a docker, covering the most fundamental workflows. We are not
> > sure how
> > >>>> >> well those tests work, especially against some potential corner
> > cases.
> > >>>> >>   3.  Divergence from other deployment. Because of 1 and 2, the
> new
> > >>>> >> efforts (features, maintenance, refactors) tend to exclude Mesos
> if
> > >>>> >> possible. When the new efforts have to touch the Mesos related
> > components
> > >>>> >> (e.g., changes to the common resource manager interfaces), we
> have
> > to be
> > >>>> >> very careful and make as few changes as possible, to avoid
> > accidentally
> > >>>> >> breaking anything that we are not familiar with. As a result, the
> > component
> > >>>> >> diverges a lot from other deployment components (K8s/Yarn), which
> > makes it
> > >>>> >> harder to maintain.
> > >>>> >>
> > >>>> >> It would be greatly appreciated if you can help with either of
> the
> > above
> > >>>> >> issues.
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Additionally, I have a few questions concerning your use cases at
> > Criteo.
> > >>>> >> IIUC, you are going to stay on Mesos in the foreseeable future,
> > while
> > >>>> >> keeping the Flink version up-to-date? What Flink version are you
> > currently
> > >>>> >> using? How often do you upgrade (e.g., every release)? Would you
> > be good
> > >>>> >> with keeping the Flink on Mesos component as it is (means that
> > deployment
> > >>>> >> and resource management improvements may not be ported to Mesos),
> > while
> > >>>> >> keeping other components up-to-date (e.g., improvements from
> > programming
> > >>>> >> APIs, operators, state backens, etc.)?
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Thank you~
> > >>>> >>
> > >>>> >> Xintong Song
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
> > >>>> >> lassenedergaardflink@gmail.com<mailto:
> > lassenedergaardflink@gmail.com>>
> > >>>> >> wrote:
> > >>>> >>
> > >>>> >> Hi
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> At Trackunit We have been using Mesos for long time but have now
> > moved to
> > >>>> >> k8s.
> > >>>> >>
> > >>>> >> Med venlig hilsen / Best regards
> > >>>> >>
> > >>>> >> Lasse Nedergaard
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <
> > rmetzger@apache.org
> > >>>> >> <ma...@apache.org>>:
> > >>>> >>
> > >>>> >> 
> > >>>> >>
> > >>>> >> Hey Piyush,
> > >>>> >>
> > >>>> >> thanks a lot for raising this concern. I believe we should keep
> > Mesos in
> > >>>> >> Flink then in the foreseeable future.
> > >>>> >>
> > >>>> >> Your offer to help is much appreciated. We'll let you know once
> > there is
> > >>>> >> something.
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <
> p.narang@criteo.com
> > >>>> >> <ma...@criteo.com>> wrote:
> > >>>> >>
> > >>>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd
> be
> > able
> > >>>> >> to find folks who would be excited to contribute / help in any
> way.
> > >>>> >>
> > >>>> >> -- Piyush
> > >>>> >>
> > >>>> >>
> > >>>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com
> > <mailto:
> > >>>> >> kkloudas@gmail.com>> wrote:
> > >>>> >>
> > >>>> >>     Thanks Piyush for the message.
> > >>>> >>     After this, I revoke my +1. I agree with the previous
> opinions
> > that we
> > >>>> >>     cannot drop code that is actively used by users, especially
> if
> > it
> > >>>> >>     something that deep in the stack as support for cluster
> > management
> > >>>> >>     framework.
> > >>>> >>
> > >>>> >>     Cheers,
> > >>>> >>     Kostas
> > >>>> >>
> > >>>> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <
> > p.narang@criteo.com
> > >>>> >> <ma...@criteo.com>> wrote:
> > >>>> >>     >
> > >>>> >>     > Hi folks,
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > We at Criteo are active users of the Flink on Mesos
> resource
> > >>>> >> management component. We are pretty heavy users of Mesos for
> > scheduling
> > >>>> >> workloads on our edge datacenters and we do want to continue to
> be
> > able to
> > >>>> >> run some of our Flink topologies (to compute machine learning
> > short term
> > >>>> >> features) on those DCs. If possible our vote would be not to drop
> > Mesos
> > >>>> >> support as that will tie us to an old release / have to maintain
> a
> > fork as
> > >>>> >> we’re not planning to migrate off Mesos anytime soon. Is the
> burden
> > >>>> >> something that can be helped with by the community? (Or are you
> > referring
> > >>>> >> to having to ensure PRs handle the Mesos piece as well when they
> > touch the
> > >>>> >> resource managers?)
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > Thanks,
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > -- Piyush
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
> > >>>> >> trohrmann@apache.org>>
> > >>>> >>     > Date: Friday, October 23, 2020 at 8:19 AM
> > >>>> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
> > >>>> >> tonysong820@gmail.com>>
> > >>>> >>     > Cc: dev <dev@flink.apache.org<mailto:dev@flink.apache.org
> >>,
> > user <
> > >>>> >> user@flink.apache.org<ma...@flink.apache.org>>
> > >>>> >>     > Subject: Re: [SURVEY] Remove Mesos support
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > Thanks for starting this survey Robert! I second Konstantin
> > and
> > >>>> >> Xintong in the sense that our Mesos user's opinions should matter
> > most
> > >>>> >> here. If our community is no longer using the Mesos integration,
> > then I
> > >>>> >> would be +1 for removing it in order to decrease the maintenance
> > burden.
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > Cheers,
> > >>>> >>     >
> > >>>> >>     > Till
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <
> > tonysong820@gmail.com
> > >>>> >> <ma...@gmail.com>> wrote:
> > >>>> >>     >
> > >>>> >>     > +1 for adding a warning in 1.12 about planning to remove
> > Mesos
> > >>>> >> support.
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > With my developer hat on, removing the Mesos support would
> > >>>> >> definitely reduce the maintaining overhead for the deployment and
> > resource
> > >>>> >> management related components. On the other hand, the Flink on
> > Mesos users'
> > >>>> >> voices definitely matter a lot for this community. Either way, it
> > would be
> > >>>> >> good to draw users attention to this discussion early.
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > Thank you~
> > >>>> >>     >
> > >>>> >>     > Xintong Song
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     >
> > >>>> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <
> > knaufk@apache.org
> > >>>> >> <ma...@apache.org>> wrote:
> > >>>> >>     >
> > >>>> >>     > Hi Robert,
> > >>>> >>     >
> > >>>> >>     > +1 to the plan you outlined. If we were to drop support in
> > Flink
> > >>>> >> 1.13+, we
> > >>>> >>     > would still support it in Flink 1.12- with bug fixes for
> > some time
> > >>>> >> so that
> > >>>> >>     > users have time to move on.
> > >>>> >>     >
> > >>>> >>     > It would certainly be very interesting to hear from current
> > Flink
> > >>>> >> on Mesos
> > >>>> >>     > users, on how they see the evolution of this part of the
> > ecosystem.
> > >>>> >>     >
> > >>>> >>     > Best,
> > >>>> >>     >
> > >>>> >>     > Konstantin
> > >>>> >>
> > >>>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> Konstantin Knauf
> > >>>
> > >>> https://twitter.com/snntrable
> > >>>
> > >>> https://github.com/knaufk
> >

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Robert Metzger <rm...@apache.org>.
+1



On Mon, Mar 29, 2021 at 5:44 AM Yangze Guo <ka...@gmail.com> wrote:

> +1
>
> Best,
> Yangze Guo
>
> On Mon, Mar 29, 2021 at 11:31 AM Xintong Song <to...@gmail.com>
> wrote:
> >
> > +1
> > It's already a matter of fact for a while that we no longer port new
> features to the Mesos deployment.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann <tr...@apache.org>
> wrote:
> >>
> >> +1 for officially deprecating this component for the 1.13 release.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kn...@apache.org>
> wrote:
> >>>
> >>> Hi Matthias,
> >>>
> >>> Thank you for following up on this. +1 to officially deprecate Mesos
> in the code and documentation, too. It will be confusing for users if this
> diverges from the roadmap.
> >>>
> >>> Cheers,
> >>>
> >>> Konstantin
> >>>
> >>> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <ma...@ververica.com>
> wrote:
> >>>>
> >>>> Hi everyone,
> >>>> considering the upcoming release of Flink 1.13, I wanted to revive the
> >>>> discussion about the Mesos support ones more. Mesos is also already
> listed
> >>>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to
> align the
> >>>> documentation accordingly to make it more explicit?
> >>>>
> >>>> What do you think?
> >>>>
> >>>> Best,
> >>>> Matthias
> >>>>
> >>>> [1] https://flink.apache.org/roadmap.html#feature-radar
> >>>>
> >>>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org>
> wrote:
> >>>>
> >>>> > Hi Oleksandr,
> >>>> >
> >>>> > yes you are right. The biggest problem is at the moment the lack of
> test
> >>>> > coverage and thereby confidence to make changes. We have some e2e
> tests
> >>>> > which you can find here [1]. These tests are, however, quite coarse
> grained
> >>>> > and are missing a lot of cases. One idea would be to add a Mesos
> e2e test
> >>>> > based on Flink's end-to-end test framework [2]. I think what needs
> to be
> >>>> > done there is to add a Mesos resource and a way to submit jobs to a
> Mesos
> >>>> > cluster to write e2e tests.
> >>>> >
> >>>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
> >>>> > [2]
> >>>> >
> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
> >>>> >
> >>>> > Cheers,
> >>>> > Till
> >>>> >
> >>>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
> >>>> > o.nitavskyi@criteo.com> wrote:
> >>>> >
> >>>> >> Hello Xintong,
> >>>> >>
> >>>> >> Thanks for the insights and support.
> >>>> >>
> >>>> >> Browsing the Mesos backlog and didn't identify anything critical,
> which
> >>>> >> is left there.
> >>>> >>
> >>>> >> I see that there are were quite a lot of contributions to the
> Flink Mesos
> >>>> >> in the recent version:
> >>>> >> https://github.com/apache/flink/commits/master/flink-mesos.
> >>>> >> We plan to validate the current Flink master (or release 1.12
> branch) our
> >>>> >> Mesos setup. In case of any issues, we will try to propose changes.
> >>>> >> My feeling is that our test results shouldn't affect the Flink 1.12
> >>>> >> release cycle. And if any potential commits will land into the
> 1.12.1 it
> >>>> >> should be totally fine.
> >>>> >>
> >>>> >> In the future, we would be glad to help you guys with any
> >>>> >> maintenance-related questions. One of the highest priorities
> around this
> >>>> >> component seems to be the development of the full e2e test.
> >>>> >>
> >>>> >> Kind Regards
> >>>> >> Oleksandr Nitavskyi
> >>>> >> ________________________________
> >>>> >> From: Xintong Song <to...@gmail.com>
> >>>> >> Sent: Tuesday, October 27, 2020 7:14 AM
> >>>> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
> >>>> >> Cc: Piyush Narang <p....@criteo.com>
> >>>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
> >>>> >>
> >>>> >> Hi Piyush,
> >>>> >>
> >>>> >> Thanks a lot for sharing the information. It would be a great
> relief that
> >>>> >> you are good with Flink on Mesos as is.
> >>>> >>
> >>>> >> As for the jira issues, I believe the most essential ones should
> have
> >>>> >> already been resolved. You may find some remaining open issues
> here [1],
> >>>> >> but not all of them are necessary if we decide to keep Flink on
> Mesos as is.
> >>>> >>
> >>>> >> At the moment and in the short future, I think helps are mostly
> needed on
> >>>> >> testing the upcoming release 1.12 with Mesos use cases. The
> community is
> >>>> >> currently actively preparing the new release, and hopefully we
> could come
> >>>> >> up with a release candidate early next month. It would be greatly
> >>>> >> appreciated if you fork as experienced Flink on Mesos users can
> help with
> >>>> >> verifying the release candidates.
> >>>> >>
> >>>> >>
> >>>> >> Thank you~
> >>>> >>
> >>>> >> Xintong Song
> >>>> >>
> >>>> >> [1]
> >>>> >>
> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
> >>>> >> <
> >>>> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
> >>>> >> >
> >>>> >>
> >>>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
> >>>> >> <ma...@criteo.com>> wrote:
> >>>> >>
> >>>> >> Hi Xintong,
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Do you have any jiras that cover any of the items on 1 or 2? I can
> reach
> >>>> >> out to folks internally and see if I can get some folks to commit
> to
> >>>> >> helping out.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> To cover the other qs:
> >>>> >>
> >>>> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We
> use
> >>>> >> Yarn for some our Flink workloads when we can. Mesos is only used
> when we
> >>>> >> need streaming capabilities in our WW dcs (as our Yarn is
> centralized in
> >>>> >> one DC)
> >>>> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan
> to bump
> >>>> >> to 1.11 / 1.12 this quarter.
> >>>> >>   *   We typically upgrade once every 6 months to a year (not every
> >>>> >> release). We’d like to speed up the cadence but we’re not there
> yet.
> >>>> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
> >>>> >> functional while missing out on some of the newer features. We
> understand
> >>>> >> the pain on the communities side and we can take on the work if we
> see some
> >>>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to
> put in
> >>>> >> the request to port it over.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Thanks,
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> -- Piyush
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> From: Xintong Song <tonysong820@gmail.com<mailto:
> tonysong820@gmail.com>>
> >>>> >> Date: Sunday, October 25, 2020 at 10:57 PM
> >>>> >> To: dev <de...@flink.apache.org>>, user
> <
> >>>> >> user@flink.apache.org<ma...@flink.apache.org>>
> >>>> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
> >>>> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
> >>>> >> p.narang@criteo.com>>
> >>>> >> Subject: Re: [SURVEY] Remove Mesos support
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Thanks for sharing the information with us, Piyush an Lasse.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> @Piyush
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Thanks for offering the help. IMO, there are currently several
> problems
> >>>> >> that make supporting Flink on Mesos challenging for us.
> >>>> >>
> >>>> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if
> not
> >>>> >> none) among the active contributors in this community that are
> familiar
> >>>> >> with Mesos and can help with development on this component.
> >>>> >>   2.  Absence of tests. Mesos does not provide a testing cluster,
> like
> >>>> >> `MiniYARNCluster`, making it hard to test interactions between
> Flink and
> >>>> >> Mesos. We have only a few very simple e2e tests running on Mesos
> deployed
> >>>> >> in a docker, covering the most fundamental workflows. We are not
> sure how
> >>>> >> well those tests work, especially against some potential corner
> cases.
> >>>> >>   3.  Divergence from other deployment. Because of 1 and 2, the new
> >>>> >> efforts (features, maintenance, refactors) tend to exclude Mesos if
> >>>> >> possible. When the new efforts have to touch the Mesos related
> components
> >>>> >> (e.g., changes to the common resource manager interfaces), we have
> to be
> >>>> >> very careful and make as few changes as possible, to avoid
> accidentally
> >>>> >> breaking anything that we are not familiar with. As a result, the
> component
> >>>> >> diverges a lot from other deployment components (K8s/Yarn), which
> makes it
> >>>> >> harder to maintain.
> >>>> >>
> >>>> >> It would be greatly appreciated if you can help with either of the
> above
> >>>> >> issues.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Additionally, I have a few questions concerning your use cases at
> Criteo.
> >>>> >> IIUC, you are going to stay on Mesos in the foreseeable future,
> while
> >>>> >> keeping the Flink version up-to-date? What Flink version are you
> currently
> >>>> >> using? How often do you upgrade (e.g., every release)? Would you
> be good
> >>>> >> with keeping the Flink on Mesos component as it is (means that
> deployment
> >>>> >> and resource management improvements may not be ported to Mesos),
> while
> >>>> >> keeping other components up-to-date (e.g., improvements from
> programming
> >>>> >> APIs, operators, state backens, etc.)?
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Thank you~
> >>>> >>
> >>>> >> Xintong Song
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
> >>>> >> lassenedergaardflink@gmail.com<mailto:
> lassenedergaardflink@gmail.com>>
> >>>> >> wrote:
> >>>> >>
> >>>> >> Hi
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> At Trackunit We have been using Mesos for long time but have now
> moved to
> >>>> >> k8s.
> >>>> >>
> >>>> >> Med venlig hilsen / Best regards
> >>>> >>
> >>>> >> Lasse Nedergaard
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <
> rmetzger@apache.org
> >>>> >> <ma...@apache.org>>:
> >>>> >>
> >>>> >> 
> >>>> >>
> >>>> >> Hey Piyush,
> >>>> >>
> >>>> >> thanks a lot for raising this concern. I believe we should keep
> Mesos in
> >>>> >> Flink then in the foreseeable future.
> >>>> >>
> >>>> >> Your offer to help is much appreciated. We'll let you know once
> there is
> >>>> >> something.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
> >>>> >> <ma...@criteo.com>> wrote:
> >>>> >>
> >>>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be
> able
> >>>> >> to find folks who would be excited to contribute / help in any way.
> >>>> >>
> >>>> >> -- Piyush
> >>>> >>
> >>>> >>
> >>>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com
> <mailto:
> >>>> >> kkloudas@gmail.com>> wrote:
> >>>> >>
> >>>> >>     Thanks Piyush for the message.
> >>>> >>     After this, I revoke my +1. I agree with the previous opinions
> that we
> >>>> >>     cannot drop code that is actively used by users, especially if
> it
> >>>> >>     something that deep in the stack as support for cluster
> management
> >>>> >>     framework.
> >>>> >>
> >>>> >>     Cheers,
> >>>> >>     Kostas
> >>>> >>
> >>>> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <
> p.narang@criteo.com
> >>>> >> <ma...@criteo.com>> wrote:
> >>>> >>     >
> >>>> >>     > Hi folks,
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > We at Criteo are active users of the Flink on Mesos resource
> >>>> >> management component. We are pretty heavy users of Mesos for
> scheduling
> >>>> >> workloads on our edge datacenters and we do want to continue to be
> able to
> >>>> >> run some of our Flink topologies (to compute machine learning
> short term
> >>>> >> features) on those DCs. If possible our vote would be not to drop
> Mesos
> >>>> >> support as that will tie us to an old release / have to maintain a
> fork as
> >>>> >> we’re not planning to migrate off Mesos anytime soon. Is the burden
> >>>> >> something that can be helped with by the community? (Or are you
> referring
> >>>> >> to having to ensure PRs handle the Mesos piece as well when they
> touch the
> >>>> >> resource managers?)
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > Thanks,
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > -- Piyush
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
> >>>> >> trohrmann@apache.org>>
> >>>> >>     > Date: Friday, October 23, 2020 at 8:19 AM
> >>>> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
> >>>> >> tonysong820@gmail.com>>
> >>>> >>     > Cc: dev <de...@flink.apache.org>>,
> user <
> >>>> >> user@flink.apache.org<ma...@flink.apache.org>>
> >>>> >>     > Subject: Re: [SURVEY] Remove Mesos support
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > Thanks for starting this survey Robert! I second Konstantin
> and
> >>>> >> Xintong in the sense that our Mesos user's opinions should matter
> most
> >>>> >> here. If our community is no longer using the Mesos integration,
> then I
> >>>> >> would be +1 for removing it in order to decrease the maintenance
> burden.
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > Cheers,
> >>>> >>     >
> >>>> >>     > Till
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <
> tonysong820@gmail.com
> >>>> >> <ma...@gmail.com>> wrote:
> >>>> >>     >
> >>>> >>     > +1 for adding a warning in 1.12 about planning to remove
> Mesos
> >>>> >> support.
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > With my developer hat on, removing the Mesos support would
> >>>> >> definitely reduce the maintaining overhead for the deployment and
> resource
> >>>> >> management related components. On the other hand, the Flink on
> Mesos users'
> >>>> >> voices definitely matter a lot for this community. Either way, it
> would be
> >>>> >> good to draw users attention to this discussion early.
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > Thank you~
> >>>> >>     >
> >>>> >>     > Xintong Song
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <
> knaufk@apache.org
> >>>> >> <ma...@apache.org>> wrote:
> >>>> >>     >
> >>>> >>     > Hi Robert,
> >>>> >>     >
> >>>> >>     > +1 to the plan you outlined. If we were to drop support in
> Flink
> >>>> >> 1.13+, we
> >>>> >>     > would still support it in Flink 1.12- with bug fixes for
> some time
> >>>> >> so that
> >>>> >>     > users have time to move on.
> >>>> >>     >
> >>>> >>     > It would certainly be very interesting to hear from current
> Flink
> >>>> >> on Mesos
> >>>> >>     > users, on how they see the evolution of this part of the
> ecosystem.
> >>>> >>     >
> >>>> >>     > Best,
> >>>> >>     >
> >>>> >>     > Konstantin
> >>>> >>
> >>>> >
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Konstantin Knauf
> >>>
> >>> https://twitter.com/snntrable
> >>>
> >>> https://github.com/knaufk
>

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Robert Metzger <rm...@apache.org>.
+1



On Mon, Mar 29, 2021 at 5:44 AM Yangze Guo <ka...@gmail.com> wrote:

> +1
>
> Best,
> Yangze Guo
>
> On Mon, Mar 29, 2021 at 11:31 AM Xintong Song <to...@gmail.com>
> wrote:
> >
> > +1
> > It's already a matter of fact for a while that we no longer port new
> features to the Mesos deployment.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann <tr...@apache.org>
> wrote:
> >>
> >> +1 for officially deprecating this component for the 1.13 release.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kn...@apache.org>
> wrote:
> >>>
> >>> Hi Matthias,
> >>>
> >>> Thank you for following up on this. +1 to officially deprecate Mesos
> in the code and documentation, too. It will be confusing for users if this
> diverges from the roadmap.
> >>>
> >>> Cheers,
> >>>
> >>> Konstantin
> >>>
> >>> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <ma...@ververica.com>
> wrote:
> >>>>
> >>>> Hi everyone,
> >>>> considering the upcoming release of Flink 1.13, I wanted to revive the
> >>>> discussion about the Mesos support ones more. Mesos is also already
> listed
> >>>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to
> align the
> >>>> documentation accordingly to make it more explicit?
> >>>>
> >>>> What do you think?
> >>>>
> >>>> Best,
> >>>> Matthias
> >>>>
> >>>> [1] https://flink.apache.org/roadmap.html#feature-radar
> >>>>
> >>>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org>
> wrote:
> >>>>
> >>>> > Hi Oleksandr,
> >>>> >
> >>>> > yes you are right. The biggest problem is at the moment the lack of
> test
> >>>> > coverage and thereby confidence to make changes. We have some e2e
> tests
> >>>> > which you can find here [1]. These tests are, however, quite coarse
> grained
> >>>> > and are missing a lot of cases. One idea would be to add a Mesos
> e2e test
> >>>> > based on Flink's end-to-end test framework [2]. I think what needs
> to be
> >>>> > done there is to add a Mesos resource and a way to submit jobs to a
> Mesos
> >>>> > cluster to write e2e tests.
> >>>> >
> >>>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
> >>>> > [2]
> >>>> >
> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
> >>>> >
> >>>> > Cheers,
> >>>> > Till
> >>>> >
> >>>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
> >>>> > o.nitavskyi@criteo.com> wrote:
> >>>> >
> >>>> >> Hello Xintong,
> >>>> >>
> >>>> >> Thanks for the insights and support.
> >>>> >>
> >>>> >> Browsing the Mesos backlog and didn't identify anything critical,
> which
> >>>> >> is left there.
> >>>> >>
> >>>> >> I see that there are were quite a lot of contributions to the
> Flink Mesos
> >>>> >> in the recent version:
> >>>> >> https://github.com/apache/flink/commits/master/flink-mesos.
> >>>> >> We plan to validate the current Flink master (or release 1.12
> branch) our
> >>>> >> Mesos setup. In case of any issues, we will try to propose changes.
> >>>> >> My feeling is that our test results shouldn't affect the Flink 1.12
> >>>> >> release cycle. And if any potential commits will land into the
> 1.12.1 it
> >>>> >> should be totally fine.
> >>>> >>
> >>>> >> In the future, we would be glad to help you guys with any
> >>>> >> maintenance-related questions. One of the highest priorities
> around this
> >>>> >> component seems to be the development of the full e2e test.
> >>>> >>
> >>>> >> Kind Regards
> >>>> >> Oleksandr Nitavskyi
> >>>> >> ________________________________
> >>>> >> From: Xintong Song <to...@gmail.com>
> >>>> >> Sent: Tuesday, October 27, 2020 7:14 AM
> >>>> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
> >>>> >> Cc: Piyush Narang <p....@criteo.com>
> >>>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
> >>>> >>
> >>>> >> Hi Piyush,
> >>>> >>
> >>>> >> Thanks a lot for sharing the information. It would be a great
> relief that
> >>>> >> you are good with Flink on Mesos as is.
> >>>> >>
> >>>> >> As for the jira issues, I believe the most essential ones should
> have
> >>>> >> already been resolved. You may find some remaining open issues
> here [1],
> >>>> >> but not all of them are necessary if we decide to keep Flink on
> Mesos as is.
> >>>> >>
> >>>> >> At the moment and in the short future, I think helps are mostly
> needed on
> >>>> >> testing the upcoming release 1.12 with Mesos use cases. The
> community is
> >>>> >> currently actively preparing the new release, and hopefully we
> could come
> >>>> >> up with a release candidate early next month. It would be greatly
> >>>> >> appreciated if you fork as experienced Flink on Mesos users can
> help with
> >>>> >> verifying the release candidates.
> >>>> >>
> >>>> >>
> >>>> >> Thank you~
> >>>> >>
> >>>> >> Xintong Song
> >>>> >>
> >>>> >> [1]
> >>>> >>
> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
> >>>> >> <
> >>>> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
> >>>> >> >
> >>>> >>
> >>>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
> >>>> >> <ma...@criteo.com>> wrote:
> >>>> >>
> >>>> >> Hi Xintong,
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Do you have any jiras that cover any of the items on 1 or 2? I can
> reach
> >>>> >> out to folks internally and see if I can get some folks to commit
> to
> >>>> >> helping out.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> To cover the other qs:
> >>>> >>
> >>>> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We
> use
> >>>> >> Yarn for some our Flink workloads when we can. Mesos is only used
> when we
> >>>> >> need streaming capabilities in our WW dcs (as our Yarn is
> centralized in
> >>>> >> one DC)
> >>>> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan
> to bump
> >>>> >> to 1.11 / 1.12 this quarter.
> >>>> >>   *   We typically upgrade once every 6 months to a year (not every
> >>>> >> release). We’d like to speed up the cadence but we’re not there
> yet.
> >>>> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
> >>>> >> functional while missing out on some of the newer features. We
> understand
> >>>> >> the pain on the communities side and we can take on the work if we
> see some
> >>>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to
> put in
> >>>> >> the request to port it over.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Thanks,
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> -- Piyush
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> From: Xintong Song <tonysong820@gmail.com<mailto:
> tonysong820@gmail.com>>
> >>>> >> Date: Sunday, October 25, 2020 at 10:57 PM
> >>>> >> To: dev <de...@flink.apache.org>>, user
> <
> >>>> >> user@flink.apache.org<ma...@flink.apache.org>>
> >>>> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
> >>>> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
> >>>> >> p.narang@criteo.com>>
> >>>> >> Subject: Re: [SURVEY] Remove Mesos support
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Thanks for sharing the information with us, Piyush an Lasse.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> @Piyush
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Thanks for offering the help. IMO, there are currently several
> problems
> >>>> >> that make supporting Flink on Mesos challenging for us.
> >>>> >>
> >>>> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if
> not
> >>>> >> none) among the active contributors in this community that are
> familiar
> >>>> >> with Mesos and can help with development on this component.
> >>>> >>   2.  Absence of tests. Mesos does not provide a testing cluster,
> like
> >>>> >> `MiniYARNCluster`, making it hard to test interactions between
> Flink and
> >>>> >> Mesos. We have only a few very simple e2e tests running on Mesos
> deployed
> >>>> >> in a docker, covering the most fundamental workflows. We are not
> sure how
> >>>> >> well those tests work, especially against some potential corner
> cases.
> >>>> >>   3.  Divergence from other deployment. Because of 1 and 2, the new
> >>>> >> efforts (features, maintenance, refactors) tend to exclude Mesos if
> >>>> >> possible. When the new efforts have to touch the Mesos related
> components
> >>>> >> (e.g., changes to the common resource manager interfaces), we have
> to be
> >>>> >> very careful and make as few changes as possible, to avoid
> accidentally
> >>>> >> breaking anything that we are not familiar with. As a result, the
> component
> >>>> >> diverges a lot from other deployment components (K8s/Yarn), which
> makes it
> >>>> >> harder to maintain.
> >>>> >>
> >>>> >> It would be greatly appreciated if you can help with either of the
> above
> >>>> >> issues.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Additionally, I have a few questions concerning your use cases at
> Criteo.
> >>>> >> IIUC, you are going to stay on Mesos in the foreseeable future,
> while
> >>>> >> keeping the Flink version up-to-date? What Flink version are you
> currently
> >>>> >> using? How often do you upgrade (e.g., every release)? Would you
> be good
> >>>> >> with keeping the Flink on Mesos component as it is (means that
> deployment
> >>>> >> and resource management improvements may not be ported to Mesos),
> while
> >>>> >> keeping other components up-to-date (e.g., improvements from
> programming
> >>>> >> APIs, operators, state backens, etc.)?
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Thank you~
> >>>> >>
> >>>> >> Xintong Song
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
> >>>> >> lassenedergaardflink@gmail.com<mailto:
> lassenedergaardflink@gmail.com>>
> >>>> >> wrote:
> >>>> >>
> >>>> >> Hi
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> At Trackunit We have been using Mesos for long time but have now
> moved to
> >>>> >> k8s.
> >>>> >>
> >>>> >> Med venlig hilsen / Best regards
> >>>> >>
> >>>> >> Lasse Nedergaard
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <
> rmetzger@apache.org
> >>>> >> <ma...@apache.org>>:
> >>>> >>
> >>>> >> 
> >>>> >>
> >>>> >> Hey Piyush,
> >>>> >>
> >>>> >> thanks a lot for raising this concern. I believe we should keep
> Mesos in
> >>>> >> Flink then in the foreseeable future.
> >>>> >>
> >>>> >> Your offer to help is much appreciated. We'll let you know once
> there is
> >>>> >> something.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
> >>>> >> <ma...@criteo.com>> wrote:
> >>>> >>
> >>>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be
> able
> >>>> >> to find folks who would be excited to contribute / help in any way.
> >>>> >>
> >>>> >> -- Piyush
> >>>> >>
> >>>> >>
> >>>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com
> <mailto:
> >>>> >> kkloudas@gmail.com>> wrote:
> >>>> >>
> >>>> >>     Thanks Piyush for the message.
> >>>> >>     After this, I revoke my +1. I agree with the previous opinions
> that we
> >>>> >>     cannot drop code that is actively used by users, especially if
> it
> >>>> >>     something that deep in the stack as support for cluster
> management
> >>>> >>     framework.
> >>>> >>
> >>>> >>     Cheers,
> >>>> >>     Kostas
> >>>> >>
> >>>> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <
> p.narang@criteo.com
> >>>> >> <ma...@criteo.com>> wrote:
> >>>> >>     >
> >>>> >>     > Hi folks,
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > We at Criteo are active users of the Flink on Mesos resource
> >>>> >> management component. We are pretty heavy users of Mesos for
> scheduling
> >>>> >> workloads on our edge datacenters and we do want to continue to be
> able to
> >>>> >> run some of our Flink topologies (to compute machine learning
> short term
> >>>> >> features) on those DCs. If possible our vote would be not to drop
> Mesos
> >>>> >> support as that will tie us to an old release / have to maintain a
> fork as
> >>>> >> we’re not planning to migrate off Mesos anytime soon. Is the burden
> >>>> >> something that can be helped with by the community? (Or are you
> referring
> >>>> >> to having to ensure PRs handle the Mesos piece as well when they
> touch the
> >>>> >> resource managers?)
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > Thanks,
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > -- Piyush
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
> >>>> >> trohrmann@apache.org>>
> >>>> >>     > Date: Friday, October 23, 2020 at 8:19 AM
> >>>> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
> >>>> >> tonysong820@gmail.com>>
> >>>> >>     > Cc: dev <de...@flink.apache.org>>,
> user <
> >>>> >> user@flink.apache.org<ma...@flink.apache.org>>
> >>>> >>     > Subject: Re: [SURVEY] Remove Mesos support
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > Thanks for starting this survey Robert! I second Konstantin
> and
> >>>> >> Xintong in the sense that our Mesos user's opinions should matter
> most
> >>>> >> here. If our community is no longer using the Mesos integration,
> then I
> >>>> >> would be +1 for removing it in order to decrease the maintenance
> burden.
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > Cheers,
> >>>> >>     >
> >>>> >>     > Till
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <
> tonysong820@gmail.com
> >>>> >> <ma...@gmail.com>> wrote:
> >>>> >>     >
> >>>> >>     > +1 for adding a warning in 1.12 about planning to remove
> Mesos
> >>>> >> support.
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > With my developer hat on, removing the Mesos support would
> >>>> >> definitely reduce the maintaining overhead for the deployment and
> resource
> >>>> >> management related components. On the other hand, the Flink on
> Mesos users'
> >>>> >> voices definitely matter a lot for this community. Either way, it
> would be
> >>>> >> good to draw users attention to this discussion early.
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > Thank you~
> >>>> >>     >
> >>>> >>     > Xintong Song
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     >
> >>>> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <
> knaufk@apache.org
> >>>> >> <ma...@apache.org>> wrote:
> >>>> >>     >
> >>>> >>     > Hi Robert,
> >>>> >>     >
> >>>> >>     > +1 to the plan you outlined. If we were to drop support in
> Flink
> >>>> >> 1.13+, we
> >>>> >>     > would still support it in Flink 1.12- with bug fixes for
> some time
> >>>> >> so that
> >>>> >>     > users have time to move on.
> >>>> >>     >
> >>>> >>     > It would certainly be very interesting to hear from current
> Flink
> >>>> >> on Mesos
> >>>> >>     > users, on how they see the evolution of this part of the
> ecosystem.
> >>>> >>     >
> >>>> >>     > Best,
> >>>> >>     >
> >>>> >>     > Konstantin
> >>>> >>
> >>>> >
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Konstantin Knauf
> >>>
> >>> https://twitter.com/snntrable
> >>>
> >>> https://github.com/knaufk
>

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Yangze Guo <ka...@gmail.com>.
+1

Best,
Yangze Guo

On Mon, Mar 29, 2021 at 11:31 AM Xintong Song <to...@gmail.com> wrote:
>
> +1
> It's already a matter of fact for a while that we no longer port new features to the Mesos deployment.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann <tr...@apache.org> wrote:
>>
>> +1 for officially deprecating this component for the 1.13 release.
>>
>> Cheers,
>> Till
>>
>> On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kn...@apache.org> wrote:
>>>
>>> Hi Matthias,
>>>
>>> Thank you for following up on this. +1 to officially deprecate Mesos in the code and documentation, too. It will be confusing for users if this diverges from the roadmap.
>>>
>>> Cheers,
>>>
>>> Konstantin
>>>
>>> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <ma...@ververica.com> wrote:
>>>>
>>>> Hi everyone,
>>>> considering the upcoming release of Flink 1.13, I wanted to revive the
>>>> discussion about the Mesos support ones more. Mesos is also already listed
>>>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align the
>>>> documentation accordingly to make it more explicit?
>>>>
>>>> What do you think?
>>>>
>>>> Best,
>>>> Matthias
>>>>
>>>> [1] https://flink.apache.org/roadmap.html#feature-radar
>>>>
>>>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org> wrote:
>>>>
>>>> > Hi Oleksandr,
>>>> >
>>>> > yes you are right. The biggest problem is at the moment the lack of test
>>>> > coverage and thereby confidence to make changes. We have some e2e tests
>>>> > which you can find here [1]. These tests are, however, quite coarse grained
>>>> > and are missing a lot of cases. One idea would be to add a Mesos e2e test
>>>> > based on Flink's end-to-end test framework [2]. I think what needs to be
>>>> > done there is to add a Mesos resource and a way to submit jobs to a Mesos
>>>> > cluster to write e2e tests.
>>>> >
>>>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
>>>> > [2]
>>>> > https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
>>>> >
>>>> > Cheers,
>>>> > Till
>>>> >
>>>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
>>>> > o.nitavskyi@criteo.com> wrote:
>>>> >
>>>> >> Hello Xintong,
>>>> >>
>>>> >> Thanks for the insights and support.
>>>> >>
>>>> >> Browsing the Mesos backlog and didn't identify anything critical, which
>>>> >> is left there.
>>>> >>
>>>> >> I see that there are were quite a lot of contributions to the Flink Mesos
>>>> >> in the recent version:
>>>> >> https://github.com/apache/flink/commits/master/flink-mesos.
>>>> >> We plan to validate the current Flink master (or release 1.12 branch) our
>>>> >> Mesos setup. In case of any issues, we will try to propose changes.
>>>> >> My feeling is that our test results shouldn't affect the Flink 1.12
>>>> >> release cycle. And if any potential commits will land into the 1.12.1 it
>>>> >> should be totally fine.
>>>> >>
>>>> >> In the future, we would be glad to help you guys with any
>>>> >> maintenance-related questions. One of the highest priorities around this
>>>> >> component seems to be the development of the full e2e test.
>>>> >>
>>>> >> Kind Regards
>>>> >> Oleksandr Nitavskyi
>>>> >> ________________________________
>>>> >> From: Xintong Song <to...@gmail.com>
>>>> >> Sent: Tuesday, October 27, 2020 7:14 AM
>>>> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
>>>> >> Cc: Piyush Narang <p....@criteo.com>
>>>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
>>>> >>
>>>> >> Hi Piyush,
>>>> >>
>>>> >> Thanks a lot for sharing the information. It would be a great relief that
>>>> >> you are good with Flink on Mesos as is.
>>>> >>
>>>> >> As for the jira issues, I believe the most essential ones should have
>>>> >> already been resolved. You may find some remaining open issues here [1],
>>>> >> but not all of them are necessary if we decide to keep Flink on Mesos as is.
>>>> >>
>>>> >> At the moment and in the short future, I think helps are mostly needed on
>>>> >> testing the upcoming release 1.12 with Mesos use cases. The community is
>>>> >> currently actively preparing the new release, and hopefully we could come
>>>> >> up with a release candidate early next month. It would be greatly
>>>> >> appreciated if you fork as experienced Flink on Mesos users can help with
>>>> >> verifying the release candidates.
>>>> >>
>>>> >>
>>>> >> Thank you~
>>>> >>
>>>> >> Xintong Song
>>>> >>
>>>> >> [1]
>>>> >> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
>>>> >> <
>>>> >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
>>>> >> >
>>>> >>
>>>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
>>>> >> <ma...@criteo.com>> wrote:
>>>> >>
>>>> >> Hi Xintong,
>>>> >>
>>>> >>
>>>> >>
>>>> >> Do you have any jiras that cover any of the items on 1 or 2? I can reach
>>>> >> out to folks internally and see if I can get some folks to commit to
>>>> >> helping out.
>>>> >>
>>>> >>
>>>> >>
>>>> >> To cover the other qs:
>>>> >>
>>>> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We use
>>>> >> Yarn for some our Flink workloads when we can. Mesos is only used when we
>>>> >> need streaming capabilities in our WW dcs (as our Yarn is centralized in
>>>> >> one DC)
>>>> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan to bump
>>>> >> to 1.11 / 1.12 this quarter.
>>>> >>   *   We typically upgrade once every 6 months to a year (not every
>>>> >> release). We’d like to speed up the cadence but we’re not there yet.
>>>> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
>>>> >> functional while missing out on some of the newer features. We understand
>>>> >> the pain on the communities side and we can take on the work if we see some
>>>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put in
>>>> >> the request to port it over.
>>>> >>
>>>> >>
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >>
>>>> >>
>>>> >> -- Piyush
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> From: Xintong Song <to...@gmail.com>>
>>>> >> Date: Sunday, October 25, 2020 at 10:57 PM
>>>> >> To: dev <de...@flink.apache.org>>, user <
>>>> >> user@flink.apache.org<ma...@flink.apache.org>>
>>>> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
>>>> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
>>>> >> p.narang@criteo.com>>
>>>> >> Subject: Re: [SURVEY] Remove Mesos support
>>>> >>
>>>> >>
>>>> >>
>>>> >> Thanks for sharing the information with us, Piyush an Lasse.
>>>> >>
>>>> >>
>>>> >>
>>>> >> @Piyush
>>>> >>
>>>> >>
>>>> >>
>>>> >> Thanks for offering the help. IMO, there are currently several problems
>>>> >> that make supporting Flink on Mesos challenging for us.
>>>> >>
>>>> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if not
>>>> >> none) among the active contributors in this community that are familiar
>>>> >> with Mesos and can help with development on this component.
>>>> >>   2.  Absence of tests. Mesos does not provide a testing cluster, like
>>>> >> `MiniYARNCluster`, making it hard to test interactions between Flink and
>>>> >> Mesos. We have only a few very simple e2e tests running on Mesos deployed
>>>> >> in a docker, covering the most fundamental workflows. We are not sure how
>>>> >> well those tests work, especially against some potential corner cases.
>>>> >>   3.  Divergence from other deployment. Because of 1 and 2, the new
>>>> >> efforts (features, maintenance, refactors) tend to exclude Mesos if
>>>> >> possible. When the new efforts have to touch the Mesos related components
>>>> >> (e.g., changes to the common resource manager interfaces), we have to be
>>>> >> very careful and make as few changes as possible, to avoid accidentally
>>>> >> breaking anything that we are not familiar with. As a result, the component
>>>> >> diverges a lot from other deployment components (K8s/Yarn), which makes it
>>>> >> harder to maintain.
>>>> >>
>>>> >> It would be greatly appreciated if you can help with either of the above
>>>> >> issues.
>>>> >>
>>>> >>
>>>> >>
>>>> >> Additionally, I have a few questions concerning your use cases at Criteo.
>>>> >> IIUC, you are going to stay on Mesos in the foreseeable future, while
>>>> >> keeping the Flink version up-to-date? What Flink version are you currently
>>>> >> using? How often do you upgrade (e.g., every release)? Would you be good
>>>> >> with keeping the Flink on Mesos component as it is (means that deployment
>>>> >> and resource management improvements may not be ported to Mesos), while
>>>> >> keeping other components up-to-date (e.g., improvements from programming
>>>> >> APIs, operators, state backens, etc.)?
>>>> >>
>>>> >>
>>>> >>
>>>> >> Thank you~
>>>> >>
>>>> >> Xintong Song
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
>>>> >> lassenedergaardflink@gmail.com<ma...@gmail.com>>
>>>> >> wrote:
>>>> >>
>>>> >> Hi
>>>> >>
>>>> >>
>>>> >>
>>>> >> At Trackunit We have been using Mesos for long time but have now moved to
>>>> >> k8s.
>>>> >>
>>>> >> Med venlig hilsen / Best regards
>>>> >>
>>>> >> Lasse Nedergaard
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetzger@apache.org
>>>> >> <ma...@apache.org>>:
>>>> >>
>>>> >> 
>>>> >>
>>>> >> Hey Piyush,
>>>> >>
>>>> >> thanks a lot for raising this concern. I believe we should keep Mesos in
>>>> >> Flink then in the foreseeable future.
>>>> >>
>>>> >> Your offer to help is much appreciated. We'll let you know once there is
>>>> >> something.
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
>>>> >> <ma...@criteo.com>> wrote:
>>>> >>
>>>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be able
>>>> >> to find folks who would be excited to contribute / help in any way.
>>>> >>
>>>> >> -- Piyush
>>>> >>
>>>> >>
>>>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com<mailto:
>>>> >> kkloudas@gmail.com>> wrote:
>>>> >>
>>>> >>     Thanks Piyush for the message.
>>>> >>     After this, I revoke my +1. I agree with the previous opinions that we
>>>> >>     cannot drop code that is actively used by users, especially if it
>>>> >>     something that deep in the stack as support for cluster management
>>>> >>     framework.
>>>> >>
>>>> >>     Cheers,
>>>> >>     Kostas
>>>> >>
>>>> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <p.narang@criteo.com
>>>> >> <ma...@criteo.com>> wrote:
>>>> >>     >
>>>> >>     > Hi folks,
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > We at Criteo are active users of the Flink on Mesos resource
>>>> >> management component. We are pretty heavy users of Mesos for scheduling
>>>> >> workloads on our edge datacenters and we do want to continue to be able to
>>>> >> run some of our Flink topologies (to compute machine learning short term
>>>> >> features) on those DCs. If possible our vote would be not to drop Mesos
>>>> >> support as that will tie us to an old release / have to maintain a fork as
>>>> >> we’re not planning to migrate off Mesos anytime soon. Is the burden
>>>> >> something that can be helped with by the community? (Or are you referring
>>>> >> to having to ensure PRs handle the Mesos piece as well when they touch the
>>>> >> resource managers?)
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > Thanks,
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > -- Piyush
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
>>>> >> trohrmann@apache.org>>
>>>> >>     > Date: Friday, October 23, 2020 at 8:19 AM
>>>> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
>>>> >> tonysong820@gmail.com>>
>>>> >>     > Cc: dev <de...@flink.apache.org>>, user <
>>>> >> user@flink.apache.org<ma...@flink.apache.org>>
>>>> >>     > Subject: Re: [SURVEY] Remove Mesos support
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > Thanks for starting this survey Robert! I second Konstantin and
>>>> >> Xintong in the sense that our Mesos user's opinions should matter most
>>>> >> here. If our community is no longer using the Mesos integration, then I
>>>> >> would be +1 for removing it in order to decrease the maintenance burden.
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > Cheers,
>>>> >>     >
>>>> >>     > Till
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <tonysong820@gmail.com
>>>> >> <ma...@gmail.com>> wrote:
>>>> >>     >
>>>> >>     > +1 for adding a warning in 1.12 about planning to remove Mesos
>>>> >> support.
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > With my developer hat on, removing the Mesos support would
>>>> >> definitely reduce the maintaining overhead for the deployment and resource
>>>> >> management related components. On the other hand, the Flink on Mesos users'
>>>> >> voices definitely matter a lot for this community. Either way, it would be
>>>> >> good to draw users attention to this discussion early.
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > Thank you~
>>>> >>     >
>>>> >>     > Xintong Song
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <knaufk@apache.org
>>>> >> <ma...@apache.org>> wrote:
>>>> >>     >
>>>> >>     > Hi Robert,
>>>> >>     >
>>>> >>     > +1 to the plan you outlined. If we were to drop support in Flink
>>>> >> 1.13+, we
>>>> >>     > would still support it in Flink 1.12- with bug fixes for some time
>>>> >> so that
>>>> >>     > users have time to move on.
>>>> >>     >
>>>> >>     > It would certainly be very interesting to hear from current Flink
>>>> >> on Mesos
>>>> >>     > users, on how they see the evolution of this part of the ecosystem.
>>>> >>     >
>>>> >>     > Best,
>>>> >>     >
>>>> >>     > Konstantin
>>>> >>
>>>> >
>>>
>>>
>>>
>>> --
>>>
>>> Konstantin Knauf
>>>
>>> https://twitter.com/snntrable
>>>
>>> https://github.com/knaufk

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Yangze Guo <ka...@gmail.com>.
+1

Best,
Yangze Guo

On Mon, Mar 29, 2021 at 11:31 AM Xintong Song <to...@gmail.com> wrote:
>
> +1
> It's already a matter of fact for a while that we no longer port new features to the Mesos deployment.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann <tr...@apache.org> wrote:
>>
>> +1 for officially deprecating this component for the 1.13 release.
>>
>> Cheers,
>> Till
>>
>> On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kn...@apache.org> wrote:
>>>
>>> Hi Matthias,
>>>
>>> Thank you for following up on this. +1 to officially deprecate Mesos in the code and documentation, too. It will be confusing for users if this diverges from the roadmap.
>>>
>>> Cheers,
>>>
>>> Konstantin
>>>
>>> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <ma...@ververica.com> wrote:
>>>>
>>>> Hi everyone,
>>>> considering the upcoming release of Flink 1.13, I wanted to revive the
>>>> discussion about the Mesos support ones more. Mesos is also already listed
>>>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align the
>>>> documentation accordingly to make it more explicit?
>>>>
>>>> What do you think?
>>>>
>>>> Best,
>>>> Matthias
>>>>
>>>> [1] https://flink.apache.org/roadmap.html#feature-radar
>>>>
>>>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org> wrote:
>>>>
>>>> > Hi Oleksandr,
>>>> >
>>>> > yes you are right. The biggest problem is at the moment the lack of test
>>>> > coverage and thereby confidence to make changes. We have some e2e tests
>>>> > which you can find here [1]. These tests are, however, quite coarse grained
>>>> > and are missing a lot of cases. One idea would be to add a Mesos e2e test
>>>> > based on Flink's end-to-end test framework [2]. I think what needs to be
>>>> > done there is to add a Mesos resource and a way to submit jobs to a Mesos
>>>> > cluster to write e2e tests.
>>>> >
>>>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
>>>> > [2]
>>>> > https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
>>>> >
>>>> > Cheers,
>>>> > Till
>>>> >
>>>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
>>>> > o.nitavskyi@criteo.com> wrote:
>>>> >
>>>> >> Hello Xintong,
>>>> >>
>>>> >> Thanks for the insights and support.
>>>> >>
>>>> >> Browsing the Mesos backlog and didn't identify anything critical, which
>>>> >> is left there.
>>>> >>
>>>> >> I see that there are were quite a lot of contributions to the Flink Mesos
>>>> >> in the recent version:
>>>> >> https://github.com/apache/flink/commits/master/flink-mesos.
>>>> >> We plan to validate the current Flink master (or release 1.12 branch) our
>>>> >> Mesos setup. In case of any issues, we will try to propose changes.
>>>> >> My feeling is that our test results shouldn't affect the Flink 1.12
>>>> >> release cycle. And if any potential commits will land into the 1.12.1 it
>>>> >> should be totally fine.
>>>> >>
>>>> >> In the future, we would be glad to help you guys with any
>>>> >> maintenance-related questions. One of the highest priorities around this
>>>> >> component seems to be the development of the full e2e test.
>>>> >>
>>>> >> Kind Regards
>>>> >> Oleksandr Nitavskyi
>>>> >> ________________________________
>>>> >> From: Xintong Song <to...@gmail.com>
>>>> >> Sent: Tuesday, October 27, 2020 7:14 AM
>>>> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
>>>> >> Cc: Piyush Narang <p....@criteo.com>
>>>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
>>>> >>
>>>> >> Hi Piyush,
>>>> >>
>>>> >> Thanks a lot for sharing the information. It would be a great relief that
>>>> >> you are good with Flink on Mesos as is.
>>>> >>
>>>> >> As for the jira issues, I believe the most essential ones should have
>>>> >> already been resolved. You may find some remaining open issues here [1],
>>>> >> but not all of them are necessary if we decide to keep Flink on Mesos as is.
>>>> >>
>>>> >> At the moment and in the short future, I think helps are mostly needed on
>>>> >> testing the upcoming release 1.12 with Mesos use cases. The community is
>>>> >> currently actively preparing the new release, and hopefully we could come
>>>> >> up with a release candidate early next month. It would be greatly
>>>> >> appreciated if you fork as experienced Flink on Mesos users can help with
>>>> >> verifying the release candidates.
>>>> >>
>>>> >>
>>>> >> Thank you~
>>>> >>
>>>> >> Xintong Song
>>>> >>
>>>> >> [1]
>>>> >> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
>>>> >> <
>>>> >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
>>>> >> >
>>>> >>
>>>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
>>>> >> <ma...@criteo.com>> wrote:
>>>> >>
>>>> >> Hi Xintong,
>>>> >>
>>>> >>
>>>> >>
>>>> >> Do you have any jiras that cover any of the items on 1 or 2? I can reach
>>>> >> out to folks internally and see if I can get some folks to commit to
>>>> >> helping out.
>>>> >>
>>>> >>
>>>> >>
>>>> >> To cover the other qs:
>>>> >>
>>>> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We use
>>>> >> Yarn for some our Flink workloads when we can. Mesos is only used when we
>>>> >> need streaming capabilities in our WW dcs (as our Yarn is centralized in
>>>> >> one DC)
>>>> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan to bump
>>>> >> to 1.11 / 1.12 this quarter.
>>>> >>   *   We typically upgrade once every 6 months to a year (not every
>>>> >> release). We’d like to speed up the cadence but we’re not there yet.
>>>> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
>>>> >> functional while missing out on some of the newer features. We understand
>>>> >> the pain on the communities side and we can take on the work if we see some
>>>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put in
>>>> >> the request to port it over.
>>>> >>
>>>> >>
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >>
>>>> >>
>>>> >> -- Piyush
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> From: Xintong Song <to...@gmail.com>>
>>>> >> Date: Sunday, October 25, 2020 at 10:57 PM
>>>> >> To: dev <de...@flink.apache.org>>, user <
>>>> >> user@flink.apache.org<ma...@flink.apache.org>>
>>>> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
>>>> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
>>>> >> p.narang@criteo.com>>
>>>> >> Subject: Re: [SURVEY] Remove Mesos support
>>>> >>
>>>> >>
>>>> >>
>>>> >> Thanks for sharing the information with us, Piyush an Lasse.
>>>> >>
>>>> >>
>>>> >>
>>>> >> @Piyush
>>>> >>
>>>> >>
>>>> >>
>>>> >> Thanks for offering the help. IMO, there are currently several problems
>>>> >> that make supporting Flink on Mesos challenging for us.
>>>> >>
>>>> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if not
>>>> >> none) among the active contributors in this community that are familiar
>>>> >> with Mesos and can help with development on this component.
>>>> >>   2.  Absence of tests. Mesos does not provide a testing cluster, like
>>>> >> `MiniYARNCluster`, making it hard to test interactions between Flink and
>>>> >> Mesos. We have only a few very simple e2e tests running on Mesos deployed
>>>> >> in a docker, covering the most fundamental workflows. We are not sure how
>>>> >> well those tests work, especially against some potential corner cases.
>>>> >>   3.  Divergence from other deployment. Because of 1 and 2, the new
>>>> >> efforts (features, maintenance, refactors) tend to exclude Mesos if
>>>> >> possible. When the new efforts have to touch the Mesos related components
>>>> >> (e.g., changes to the common resource manager interfaces), we have to be
>>>> >> very careful and make as few changes as possible, to avoid accidentally
>>>> >> breaking anything that we are not familiar with. As a result, the component
>>>> >> diverges a lot from other deployment components (K8s/Yarn), which makes it
>>>> >> harder to maintain.
>>>> >>
>>>> >> It would be greatly appreciated if you can help with either of the above
>>>> >> issues.
>>>> >>
>>>> >>
>>>> >>
>>>> >> Additionally, I have a few questions concerning your use cases at Criteo.
>>>> >> IIUC, you are going to stay on Mesos in the foreseeable future, while
>>>> >> keeping the Flink version up-to-date? What Flink version are you currently
>>>> >> using? How often do you upgrade (e.g., every release)? Would you be good
>>>> >> with keeping the Flink on Mesos component as it is (means that deployment
>>>> >> and resource management improvements may not be ported to Mesos), while
>>>> >> keeping other components up-to-date (e.g., improvements from programming
>>>> >> APIs, operators, state backens, etc.)?
>>>> >>
>>>> >>
>>>> >>
>>>> >> Thank you~
>>>> >>
>>>> >> Xintong Song
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
>>>> >> lassenedergaardflink@gmail.com<ma...@gmail.com>>
>>>> >> wrote:
>>>> >>
>>>> >> Hi
>>>> >>
>>>> >>
>>>> >>
>>>> >> At Trackunit We have been using Mesos for long time but have now moved to
>>>> >> k8s.
>>>> >>
>>>> >> Med venlig hilsen / Best regards
>>>> >>
>>>> >> Lasse Nedergaard
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetzger@apache.org
>>>> >> <ma...@apache.org>>:
>>>> >>
>>>> >> 
>>>> >>
>>>> >> Hey Piyush,
>>>> >>
>>>> >> thanks a lot for raising this concern. I believe we should keep Mesos in
>>>> >> Flink then in the foreseeable future.
>>>> >>
>>>> >> Your offer to help is much appreciated. We'll let you know once there is
>>>> >> something.
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
>>>> >> <ma...@criteo.com>> wrote:
>>>> >>
>>>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be able
>>>> >> to find folks who would be excited to contribute / help in any way.
>>>> >>
>>>> >> -- Piyush
>>>> >>
>>>> >>
>>>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com<mailto:
>>>> >> kkloudas@gmail.com>> wrote:
>>>> >>
>>>> >>     Thanks Piyush for the message.
>>>> >>     After this, I revoke my +1. I agree with the previous opinions that we
>>>> >>     cannot drop code that is actively used by users, especially if it
>>>> >>     something that deep in the stack as support for cluster management
>>>> >>     framework.
>>>> >>
>>>> >>     Cheers,
>>>> >>     Kostas
>>>> >>
>>>> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <p.narang@criteo.com
>>>> >> <ma...@criteo.com>> wrote:
>>>> >>     >
>>>> >>     > Hi folks,
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > We at Criteo are active users of the Flink on Mesos resource
>>>> >> management component. We are pretty heavy users of Mesos for scheduling
>>>> >> workloads on our edge datacenters and we do want to continue to be able to
>>>> >> run some of our Flink topologies (to compute machine learning short term
>>>> >> features) on those DCs. If possible our vote would be not to drop Mesos
>>>> >> support as that will tie us to an old release / have to maintain a fork as
>>>> >> we’re not planning to migrate off Mesos anytime soon. Is the burden
>>>> >> something that can be helped with by the community? (Or are you referring
>>>> >> to having to ensure PRs handle the Mesos piece as well when they touch the
>>>> >> resource managers?)
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > Thanks,
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > -- Piyush
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
>>>> >> trohrmann@apache.org>>
>>>> >>     > Date: Friday, October 23, 2020 at 8:19 AM
>>>> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
>>>> >> tonysong820@gmail.com>>
>>>> >>     > Cc: dev <de...@flink.apache.org>>, user <
>>>> >> user@flink.apache.org<ma...@flink.apache.org>>
>>>> >>     > Subject: Re: [SURVEY] Remove Mesos support
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > Thanks for starting this survey Robert! I second Konstantin and
>>>> >> Xintong in the sense that our Mesos user's opinions should matter most
>>>> >> here. If our community is no longer using the Mesos integration, then I
>>>> >> would be +1 for removing it in order to decrease the maintenance burden.
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > Cheers,
>>>> >>     >
>>>> >>     > Till
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <tonysong820@gmail.com
>>>> >> <ma...@gmail.com>> wrote:
>>>> >>     >
>>>> >>     > +1 for adding a warning in 1.12 about planning to remove Mesos
>>>> >> support.
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > With my developer hat on, removing the Mesos support would
>>>> >> definitely reduce the maintaining overhead for the deployment and resource
>>>> >> management related components. On the other hand, the Flink on Mesos users'
>>>> >> voices definitely matter a lot for this community. Either way, it would be
>>>> >> good to draw users attention to this discussion early.
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > Thank you~
>>>> >>     >
>>>> >>     > Xintong Song
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     >
>>>> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <knaufk@apache.org
>>>> >> <ma...@apache.org>> wrote:
>>>> >>     >
>>>> >>     > Hi Robert,
>>>> >>     >
>>>> >>     > +1 to the plan you outlined. If we were to drop support in Flink
>>>> >> 1.13+, we
>>>> >>     > would still support it in Flink 1.12- with bug fixes for some time
>>>> >> so that
>>>> >>     > users have time to move on.
>>>> >>     >
>>>> >>     > It would certainly be very interesting to hear from current Flink
>>>> >> on Mesos
>>>> >>     > users, on how they see the evolution of this part of the ecosystem.
>>>> >>     >
>>>> >>     > Best,
>>>> >>     >
>>>> >>     > Konstantin
>>>> >>
>>>> >
>>>
>>>
>>>
>>> --
>>>
>>> Konstantin Knauf
>>>
>>> https://twitter.com/snntrable
>>>
>>> https://github.com/knaufk

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Xintong Song <to...@gmail.com>.
+1
It's already a matter of fact for a while that we no longer port new
features to the Mesos deployment.

Thank you~

Xintong Song



On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann <tr...@apache.org> wrote:

> +1 for officially deprecating this component for the 1.13 release.
>
> Cheers,
> Till
>
> On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kn...@apache.org>
> wrote:
>
>> Hi Matthias,
>>
>> Thank you for following up on this. +1 to officially deprecate Mesos in
>> the code and documentation, too. It will be confusing for users if this
>> diverges from the roadmap.
>>
>> Cheers,
>>
>> Konstantin
>>
>> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <ma...@ververica.com>
>> wrote:
>>
>>> Hi everyone,
>>> considering the upcoming release of Flink 1.13, I wanted to revive the
>>> discussion about the Mesos support ones more. Mesos is also already
>>> listed
>>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align
>>> the
>>> documentation accordingly to make it more explicit?
>>>
>>> What do you think?
>>>
>>> Best,
>>> Matthias
>>>
>>> [1] https://flink.apache.org/roadmap.html#feature-radar
>>>
>>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>> > Hi Oleksandr,
>>> >
>>> > yes you are right. The biggest problem is at the moment the lack of
>>> test
>>> > coverage and thereby confidence to make changes. We have some e2e tests
>>> > which you can find here [1]. These tests are, however, quite coarse
>>> grained
>>> > and are missing a lot of cases. One idea would be to add a Mesos e2e
>>> test
>>> > based on Flink's end-to-end test framework [2]. I think what needs to
>>> be
>>> > done there is to add a Mesos resource and a way to submit jobs to a
>>> Mesos
>>> > cluster to write e2e tests.
>>> >
>>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
>>> > [2]
>>> >
>>> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
>>> >
>>> > Cheers,
>>> > Till
>>> >
>>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
>>> > o.nitavskyi@criteo.com> wrote:
>>> >
>>> >> Hello Xintong,
>>> >>
>>> >> Thanks for the insights and support.
>>> >>
>>> >> Browsing the Mesos backlog and didn't identify anything critical,
>>> which
>>> >> is left there.
>>> >>
>>> >> I see that there are were quite a lot of contributions to the Flink
>>> Mesos
>>> >> in the recent version:
>>> >> https://github.com/apache/flink/commits/master/flink-mesos.
>>> >> We plan to validate the current Flink master (or release 1.12 branch)
>>> our
>>> >> Mesos setup. In case of any issues, we will try to propose changes.
>>> >> My feeling is that our test results shouldn't affect the Flink 1.12
>>> >> release cycle. And if any potential commits will land into the 1.12.1
>>> it
>>> >> should be totally fine.
>>> >>
>>> >> In the future, we would be glad to help you guys with any
>>> >> maintenance-related questions. One of the highest priorities around
>>> this
>>> >> component seems to be the development of the full e2e test.
>>> >>
>>> >> Kind Regards
>>> >> Oleksandr Nitavskyi
>>> >> ________________________________
>>> >> From: Xintong Song <to...@gmail.com>
>>> >> Sent: Tuesday, October 27, 2020 7:14 AM
>>> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
>>> >> Cc: Piyush Narang <p....@criteo.com>
>>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
>>> >>
>>> >> Hi Piyush,
>>> >>
>>> >> Thanks a lot for sharing the information. It would be a great relief
>>> that
>>> >> you are good with Flink on Mesos as is.
>>> >>
>>> >> As for the jira issues, I believe the most essential ones should have
>>> >> already been resolved. You may find some remaining open issues here
>>> [1],
>>> >> but not all of them are necessary if we decide to keep Flink on Mesos
>>> as is.
>>> >>
>>> >> At the moment and in the short future, I think helps are mostly
>>> needed on
>>> >> testing the upcoming release 1.12 with Mesos use cases. The community
>>> is
>>> >> currently actively preparing the new release, and hopefully we could
>>> come
>>> >> up with a release candidate early next month. It would be greatly
>>> >> appreciated if you fork as experienced Flink on Mesos users can help
>>> with
>>> >> verifying the release candidates.
>>> >>
>>> >>
>>> >> Thank you~
>>> >>
>>> >> Xintong Song
>>> >>
>>> >> [1]
>>> >>
>>> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
>>> >> <
>>> >>
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
>>> >> >
>>> >>
>>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
>>> >> <ma...@criteo.com>> wrote:
>>> >>
>>> >> Hi Xintong,
>>> >>
>>> >>
>>> >>
>>> >> Do you have any jiras that cover any of the items on 1 or 2? I can
>>> reach
>>> >> out to folks internally and see if I can get some folks to commit to
>>> >> helping out.
>>> >>
>>> >>
>>> >>
>>> >> To cover the other qs:
>>> >>
>>> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We use
>>> >> Yarn for some our Flink workloads when we can. Mesos is only used
>>> when we
>>> >> need streaming capabilities in our WW dcs (as our Yarn is centralized
>>> in
>>> >> one DC)
>>> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan to
>>> bump
>>> >> to 1.11 / 1.12 this quarter.
>>> >>   *   We typically upgrade once every 6 months to a year (not every
>>> >> release). We’d like to speed up the cadence but we’re not there yet.
>>> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
>>> >> functional while missing out on some of the newer features. We
>>> understand
>>> >> the pain on the communities side and we can take on the work if we
>>> see some
>>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put
>>> in
>>> >> the request to port it over.
>>> >>
>>> >>
>>> >>
>>> >> Thanks,
>>> >>
>>> >>
>>> >>
>>> >> -- Piyush
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> From: Xintong Song <tonysong820@gmail.com<mailto:
>>> tonysong820@gmail.com>>
>>> >> Date: Sunday, October 25, 2020 at 10:57 PM
>>> >> To: dev <de...@flink.apache.org>>, user <
>>> >> user@flink.apache.org<ma...@flink.apache.org>>
>>> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
>>> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
>>> >> p.narang@criteo.com>>
>>> >> Subject: Re: [SURVEY] Remove Mesos support
>>> >>
>>> >>
>>> >>
>>> >> Thanks for sharing the information with us, Piyush an Lasse.
>>> >>
>>> >>
>>> >>
>>> >> @Piyush
>>> >>
>>> >>
>>> >>
>>> >> Thanks for offering the help. IMO, there are currently several
>>> problems
>>> >> that make supporting Flink on Mesos challenging for us.
>>> >>
>>> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if not
>>> >> none) among the active contributors in this community that are
>>> familiar
>>> >> with Mesos and can help with development on this component.
>>> >>   2.  Absence of tests. Mesos does not provide a testing cluster, like
>>> >> `MiniYARNCluster`, making it hard to test interactions between Flink
>>> and
>>> >> Mesos. We have only a few very simple e2e tests running on Mesos
>>> deployed
>>> >> in a docker, covering the most fundamental workflows. We are not sure
>>> how
>>> >> well those tests work, especially against some potential corner cases.
>>> >>   3.  Divergence from other deployment. Because of 1 and 2, the new
>>> >> efforts (features, maintenance, refactors) tend to exclude Mesos if
>>> >> possible. When the new efforts have to touch the Mesos related
>>> components
>>> >> (e.g., changes to the common resource manager interfaces), we have to
>>> be
>>> >> very careful and make as few changes as possible, to avoid
>>> accidentally
>>> >> breaking anything that we are not familiar with. As a result, the
>>> component
>>> >> diverges a lot from other deployment components (K8s/Yarn), which
>>> makes it
>>> >> harder to maintain.
>>> >>
>>> >> It would be greatly appreciated if you can help with either of the
>>> above
>>> >> issues.
>>> >>
>>> >>
>>> >>
>>> >> Additionally, I have a few questions concerning your use cases at
>>> Criteo.
>>> >> IIUC, you are going to stay on Mesos in the foreseeable future, while
>>> >> keeping the Flink version up-to-date? What Flink version are you
>>> currently
>>> >> using? How often do you upgrade (e.g., every release)? Would you be
>>> good
>>> >> with keeping the Flink on Mesos component as it is (means that
>>> deployment
>>> >> and resource management improvements may not be ported to Mesos),
>>> while
>>> >> keeping other components up-to-date (e.g., improvements from
>>> programming
>>> >> APIs, operators, state backens, etc.)?
>>> >>
>>> >>
>>> >>
>>> >> Thank you~
>>> >>
>>> >> Xintong Song
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
>>> >> lassenedergaardflink@gmail.com<mailto:lassenedergaardflink@gmail.com
>>> >>
>>> >> wrote:
>>> >>
>>> >> Hi
>>> >>
>>> >>
>>> >>
>>> >> At Trackunit We have been using Mesos for long time but have now
>>> moved to
>>> >> k8s.
>>> >>
>>> >> Med venlig hilsen / Best regards
>>> >>
>>> >> Lasse Nedergaard
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetzger@apache.org
>>> >> <ma...@apache.org>>:
>>> >>
>>> >> 
>>> >>
>>> >> Hey Piyush,
>>> >>
>>> >> thanks a lot for raising this concern. I believe we should keep Mesos
>>> in
>>> >> Flink then in the foreseeable future.
>>> >>
>>> >> Your offer to help is much appreciated. We'll let you know once there
>>> is
>>> >> something.
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
>>> >> <ma...@criteo.com>> wrote:
>>> >>
>>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be
>>> able
>>> >> to find folks who would be excited to contribute / help in any way.
>>> >>
>>> >> -- Piyush
>>> >>
>>> >>
>>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com<mailto:
>>> >> kkloudas@gmail.com>> wrote:
>>> >>
>>> >>     Thanks Piyush for the message.
>>> >>     After this, I revoke my +1. I agree with the previous opinions
>>> that we
>>> >>     cannot drop code that is actively used by users, especially if it
>>> >>     something that deep in the stack as support for cluster management
>>> >>     framework.
>>> >>
>>> >>     Cheers,
>>> >>     Kostas
>>> >>
>>> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <
>>> p.narang@criteo.com
>>> >> <ma...@criteo.com>> wrote:
>>> >>     >
>>> >>     > Hi folks,
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > We at Criteo are active users of the Flink on Mesos resource
>>> >> management component. We are pretty heavy users of Mesos for
>>> scheduling
>>> >> workloads on our edge datacenters and we do want to continue to be
>>> able to
>>> >> run some of our Flink topologies (to compute machine learning short
>>> term
>>> >> features) on those DCs. If possible our vote would be not to drop
>>> Mesos
>>> >> support as that will tie us to an old release / have to maintain a
>>> fork as
>>> >> we’re not planning to migrate off Mesos anytime soon. Is the burden
>>> >> something that can be helped with by the community? (Or are you
>>> referring
>>> >> to having to ensure PRs handle the Mesos piece as well when they
>>> touch the
>>> >> resource managers?)
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > Thanks,
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > -- Piyush
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
>>> >> trohrmann@apache.org>>
>>> >>     > Date: Friday, October 23, 2020 at 8:19 AM
>>> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
>>> >> tonysong820@gmail.com>>
>>> >>     > Cc: dev <de...@flink.apache.org>>,
>>> user <
>>> >> user@flink.apache.org<ma...@flink.apache.org>>
>>> >>     > Subject: Re: [SURVEY] Remove Mesos support
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > Thanks for starting this survey Robert! I second Konstantin and
>>> >> Xintong in the sense that our Mesos user's opinions should matter most
>>> >> here. If our community is no longer using the Mesos integration, then
>>> I
>>> >> would be +1 for removing it in order to decrease the maintenance
>>> burden.
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > Cheers,
>>> >>     >
>>> >>     > Till
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <
>>> tonysong820@gmail.com
>>> >> <ma...@gmail.com>> wrote:
>>> >>     >
>>> >>     > +1 for adding a warning in 1.12 about planning to remove Mesos
>>> >> support.
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > With my developer hat on, removing the Mesos support would
>>> >> definitely reduce the maintaining overhead for the deployment and
>>> resource
>>> >> management related components. On the other hand, the Flink on Mesos
>>> users'
>>> >> voices definitely matter a lot for this community. Either way, it
>>> would be
>>> >> good to draw users attention to this discussion early.
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > Thank you~
>>> >>     >
>>> >>     > Xintong Song
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <
>>> knaufk@apache.org
>>> >> <ma...@apache.org>> wrote:
>>> >>     >
>>> >>     > Hi Robert,
>>> >>     >
>>> >>     > +1 to the plan you outlined. If we were to drop support in Flink
>>> >> 1.13+, we
>>> >>     > would still support it in Flink 1.12- with bug fixes for some
>>> time
>>> >> so that
>>> >>     > users have time to move on.
>>> >>     >
>>> >>     > It would certainly be very interesting to hear from current
>>> Flink
>>> >> on Mesos
>>> >>     > users, on how they see the evolution of this part of the
>>> ecosystem.
>>> >>     >
>>> >>     > Best,
>>> >>     >
>>> >>     > Konstantin
>>> >>
>>> >
>>>
>>
>>
>> --
>>
>> Konstantin Knauf
>>
>> https://twitter.com/snntrable
>>
>> https://github.com/knaufk
>>
>

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Xintong Song <to...@gmail.com>.
+1
It's already a matter of fact for a while that we no longer port new
features to the Mesos deployment.

Thank you~

Xintong Song



On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann <tr...@apache.org> wrote:

> +1 for officially deprecating this component for the 1.13 release.
>
> Cheers,
> Till
>
> On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kn...@apache.org>
> wrote:
>
>> Hi Matthias,
>>
>> Thank you for following up on this. +1 to officially deprecate Mesos in
>> the code and documentation, too. It will be confusing for users if this
>> diverges from the roadmap.
>>
>> Cheers,
>>
>> Konstantin
>>
>> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <ma...@ververica.com>
>> wrote:
>>
>>> Hi everyone,
>>> considering the upcoming release of Flink 1.13, I wanted to revive the
>>> discussion about the Mesos support ones more. Mesos is also already
>>> listed
>>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align
>>> the
>>> documentation accordingly to make it more explicit?
>>>
>>> What do you think?
>>>
>>> Best,
>>> Matthias
>>>
>>> [1] https://flink.apache.org/roadmap.html#feature-radar
>>>
>>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>> > Hi Oleksandr,
>>> >
>>> > yes you are right. The biggest problem is at the moment the lack of
>>> test
>>> > coverage and thereby confidence to make changes. We have some e2e tests
>>> > which you can find here [1]. These tests are, however, quite coarse
>>> grained
>>> > and are missing a lot of cases. One idea would be to add a Mesos e2e
>>> test
>>> > based on Flink's end-to-end test framework [2]. I think what needs to
>>> be
>>> > done there is to add a Mesos resource and a way to submit jobs to a
>>> Mesos
>>> > cluster to write e2e tests.
>>> >
>>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
>>> > [2]
>>> >
>>> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
>>> >
>>> > Cheers,
>>> > Till
>>> >
>>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
>>> > o.nitavskyi@criteo.com> wrote:
>>> >
>>> >> Hello Xintong,
>>> >>
>>> >> Thanks for the insights and support.
>>> >>
>>> >> Browsing the Mesos backlog and didn't identify anything critical,
>>> which
>>> >> is left there.
>>> >>
>>> >> I see that there are were quite a lot of contributions to the Flink
>>> Mesos
>>> >> in the recent version:
>>> >> https://github.com/apache/flink/commits/master/flink-mesos.
>>> >> We plan to validate the current Flink master (or release 1.12 branch)
>>> our
>>> >> Mesos setup. In case of any issues, we will try to propose changes.
>>> >> My feeling is that our test results shouldn't affect the Flink 1.12
>>> >> release cycle. And if any potential commits will land into the 1.12.1
>>> it
>>> >> should be totally fine.
>>> >>
>>> >> In the future, we would be glad to help you guys with any
>>> >> maintenance-related questions. One of the highest priorities around
>>> this
>>> >> component seems to be the development of the full e2e test.
>>> >>
>>> >> Kind Regards
>>> >> Oleksandr Nitavskyi
>>> >> ________________________________
>>> >> From: Xintong Song <to...@gmail.com>
>>> >> Sent: Tuesday, October 27, 2020 7:14 AM
>>> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
>>> >> Cc: Piyush Narang <p....@criteo.com>
>>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
>>> >>
>>> >> Hi Piyush,
>>> >>
>>> >> Thanks a lot for sharing the information. It would be a great relief
>>> that
>>> >> you are good with Flink on Mesos as is.
>>> >>
>>> >> As for the jira issues, I believe the most essential ones should have
>>> >> already been resolved. You may find some remaining open issues here
>>> [1],
>>> >> but not all of them are necessary if we decide to keep Flink on Mesos
>>> as is.
>>> >>
>>> >> At the moment and in the short future, I think helps are mostly
>>> needed on
>>> >> testing the upcoming release 1.12 with Mesos use cases. The community
>>> is
>>> >> currently actively preparing the new release, and hopefully we could
>>> come
>>> >> up with a release candidate early next month. It would be greatly
>>> >> appreciated if you fork as experienced Flink on Mesos users can help
>>> with
>>> >> verifying the release candidates.
>>> >>
>>> >>
>>> >> Thank you~
>>> >>
>>> >> Xintong Song
>>> >>
>>> >> [1]
>>> >>
>>> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
>>> >> <
>>> >>
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
>>> >> >
>>> >>
>>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
>>> >> <ma...@criteo.com>> wrote:
>>> >>
>>> >> Hi Xintong,
>>> >>
>>> >>
>>> >>
>>> >> Do you have any jiras that cover any of the items on 1 or 2? I can
>>> reach
>>> >> out to folks internally and see if I can get some folks to commit to
>>> >> helping out.
>>> >>
>>> >>
>>> >>
>>> >> To cover the other qs:
>>> >>
>>> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We use
>>> >> Yarn for some our Flink workloads when we can. Mesos is only used
>>> when we
>>> >> need streaming capabilities in our WW dcs (as our Yarn is centralized
>>> in
>>> >> one DC)
>>> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan to
>>> bump
>>> >> to 1.11 / 1.12 this quarter.
>>> >>   *   We typically upgrade once every 6 months to a year (not every
>>> >> release). We’d like to speed up the cadence but we’re not there yet.
>>> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
>>> >> functional while missing out on some of the newer features. We
>>> understand
>>> >> the pain on the communities side and we can take on the work if we
>>> see some
>>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put
>>> in
>>> >> the request to port it over.
>>> >>
>>> >>
>>> >>
>>> >> Thanks,
>>> >>
>>> >>
>>> >>
>>> >> -- Piyush
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> From: Xintong Song <tonysong820@gmail.com<mailto:
>>> tonysong820@gmail.com>>
>>> >> Date: Sunday, October 25, 2020 at 10:57 PM
>>> >> To: dev <de...@flink.apache.org>>, user <
>>> >> user@flink.apache.org<ma...@flink.apache.org>>
>>> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
>>> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
>>> >> p.narang@criteo.com>>
>>> >> Subject: Re: [SURVEY] Remove Mesos support
>>> >>
>>> >>
>>> >>
>>> >> Thanks for sharing the information with us, Piyush an Lasse.
>>> >>
>>> >>
>>> >>
>>> >> @Piyush
>>> >>
>>> >>
>>> >>
>>> >> Thanks for offering the help. IMO, there are currently several
>>> problems
>>> >> that make supporting Flink on Mesos challenging for us.
>>> >>
>>> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if not
>>> >> none) among the active contributors in this community that are
>>> familiar
>>> >> with Mesos and can help with development on this component.
>>> >>   2.  Absence of tests. Mesos does not provide a testing cluster, like
>>> >> `MiniYARNCluster`, making it hard to test interactions between Flink
>>> and
>>> >> Mesos. We have only a few very simple e2e tests running on Mesos
>>> deployed
>>> >> in a docker, covering the most fundamental workflows. We are not sure
>>> how
>>> >> well those tests work, especially against some potential corner cases.
>>> >>   3.  Divergence from other deployment. Because of 1 and 2, the new
>>> >> efforts (features, maintenance, refactors) tend to exclude Mesos if
>>> >> possible. When the new efforts have to touch the Mesos related
>>> components
>>> >> (e.g., changes to the common resource manager interfaces), we have to
>>> be
>>> >> very careful and make as few changes as possible, to avoid
>>> accidentally
>>> >> breaking anything that we are not familiar with. As a result, the
>>> component
>>> >> diverges a lot from other deployment components (K8s/Yarn), which
>>> makes it
>>> >> harder to maintain.
>>> >>
>>> >> It would be greatly appreciated if you can help with either of the
>>> above
>>> >> issues.
>>> >>
>>> >>
>>> >>
>>> >> Additionally, I have a few questions concerning your use cases at
>>> Criteo.
>>> >> IIUC, you are going to stay on Mesos in the foreseeable future, while
>>> >> keeping the Flink version up-to-date? What Flink version are you
>>> currently
>>> >> using? How often do you upgrade (e.g., every release)? Would you be
>>> good
>>> >> with keeping the Flink on Mesos component as it is (means that
>>> deployment
>>> >> and resource management improvements may not be ported to Mesos),
>>> while
>>> >> keeping other components up-to-date (e.g., improvements from
>>> programming
>>> >> APIs, operators, state backens, etc.)?
>>> >>
>>> >>
>>> >>
>>> >> Thank you~
>>> >>
>>> >> Xintong Song
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
>>> >> lassenedergaardflink@gmail.com<mailto:lassenedergaardflink@gmail.com
>>> >>
>>> >> wrote:
>>> >>
>>> >> Hi
>>> >>
>>> >>
>>> >>
>>> >> At Trackunit We have been using Mesos for long time but have now
>>> moved to
>>> >> k8s.
>>> >>
>>> >> Med venlig hilsen / Best regards
>>> >>
>>> >> Lasse Nedergaard
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetzger@apache.org
>>> >> <ma...@apache.org>>:
>>> >>
>>> >> 
>>> >>
>>> >> Hey Piyush,
>>> >>
>>> >> thanks a lot for raising this concern. I believe we should keep Mesos
>>> in
>>> >> Flink then in the foreseeable future.
>>> >>
>>> >> Your offer to help is much appreciated. We'll let you know once there
>>> is
>>> >> something.
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
>>> >> <ma...@criteo.com>> wrote:
>>> >>
>>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be
>>> able
>>> >> to find folks who would be excited to contribute / help in any way.
>>> >>
>>> >> -- Piyush
>>> >>
>>> >>
>>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com<mailto:
>>> >> kkloudas@gmail.com>> wrote:
>>> >>
>>> >>     Thanks Piyush for the message.
>>> >>     After this, I revoke my +1. I agree with the previous opinions
>>> that we
>>> >>     cannot drop code that is actively used by users, especially if it
>>> >>     something that deep in the stack as support for cluster management
>>> >>     framework.
>>> >>
>>> >>     Cheers,
>>> >>     Kostas
>>> >>
>>> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <
>>> p.narang@criteo.com
>>> >> <ma...@criteo.com>> wrote:
>>> >>     >
>>> >>     > Hi folks,
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > We at Criteo are active users of the Flink on Mesos resource
>>> >> management component. We are pretty heavy users of Mesos for
>>> scheduling
>>> >> workloads on our edge datacenters and we do want to continue to be
>>> able to
>>> >> run some of our Flink topologies (to compute machine learning short
>>> term
>>> >> features) on those DCs. If possible our vote would be not to drop
>>> Mesos
>>> >> support as that will tie us to an old release / have to maintain a
>>> fork as
>>> >> we’re not planning to migrate off Mesos anytime soon. Is the burden
>>> >> something that can be helped with by the community? (Or are you
>>> referring
>>> >> to having to ensure PRs handle the Mesos piece as well when they
>>> touch the
>>> >> resource managers?)
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > Thanks,
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > -- Piyush
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
>>> >> trohrmann@apache.org>>
>>> >>     > Date: Friday, October 23, 2020 at 8:19 AM
>>> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
>>> >> tonysong820@gmail.com>>
>>> >>     > Cc: dev <de...@flink.apache.org>>,
>>> user <
>>> >> user@flink.apache.org<ma...@flink.apache.org>>
>>> >>     > Subject: Re: [SURVEY] Remove Mesos support
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > Thanks for starting this survey Robert! I second Konstantin and
>>> >> Xintong in the sense that our Mesos user's opinions should matter most
>>> >> here. If our community is no longer using the Mesos integration, then
>>> I
>>> >> would be +1 for removing it in order to decrease the maintenance
>>> burden.
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > Cheers,
>>> >>     >
>>> >>     > Till
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <
>>> tonysong820@gmail.com
>>> >> <ma...@gmail.com>> wrote:
>>> >>     >
>>> >>     > +1 for adding a warning in 1.12 about planning to remove Mesos
>>> >> support.
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > With my developer hat on, removing the Mesos support would
>>> >> definitely reduce the maintaining overhead for the deployment and
>>> resource
>>> >> management related components. On the other hand, the Flink on Mesos
>>> users'
>>> >> voices definitely matter a lot for this community. Either way, it
>>> would be
>>> >> good to draw users attention to this discussion early.
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > Thank you~
>>> >>     >
>>> >>     > Xintong Song
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     >
>>> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <
>>> knaufk@apache.org
>>> >> <ma...@apache.org>> wrote:
>>> >>     >
>>> >>     > Hi Robert,
>>> >>     >
>>> >>     > +1 to the plan you outlined. If we were to drop support in Flink
>>> >> 1.13+, we
>>> >>     > would still support it in Flink 1.12- with bug fixes for some
>>> time
>>> >> so that
>>> >>     > users have time to move on.
>>> >>     >
>>> >>     > It would certainly be very interesting to hear from current
>>> Flink
>>> >> on Mesos
>>> >>     > users, on how they see the evolution of this part of the
>>> ecosystem.
>>> >>     >
>>> >>     > Best,
>>> >>     >
>>> >>     > Konstantin
>>> >>
>>> >
>>>
>>
>>
>> --
>>
>> Konstantin Knauf
>>
>> https://twitter.com/snntrable
>>
>> https://github.com/knaufk
>>
>

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Till Rohrmann <tr...@apache.org>.
+1 for officially deprecating this component for the 1.13 release.

Cheers,
Till

On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kn...@apache.org> wrote:

> Hi Matthias,
>
> Thank you for following up on this. +1 to officially deprecate Mesos in
> the code and documentation, too. It will be confusing for users if this
> diverges from the roadmap.
>
> Cheers,
>
> Konstantin
>
> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <ma...@ververica.com>
> wrote:
>
>> Hi everyone,
>> considering the upcoming release of Flink 1.13, I wanted to revive the
>> discussion about the Mesos support ones more. Mesos is also already listed
>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align
>> the
>> documentation accordingly to make it more explicit?
>>
>> What do you think?
>>
>> Best,
>> Matthias
>>
>> [1] https://flink.apache.org/roadmap.html#feature-radar
>>
>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>> > Hi Oleksandr,
>> >
>> > yes you are right. The biggest problem is at the moment the lack of test
>> > coverage and thereby confidence to make changes. We have some e2e tests
>> > which you can find here [1]. These tests are, however, quite coarse
>> grained
>> > and are missing a lot of cases. One idea would be to add a Mesos e2e
>> test
>> > based on Flink's end-to-end test framework [2]. I think what needs to be
>> > done there is to add a Mesos resource and a way to submit jobs to a
>> Mesos
>> > cluster to write e2e tests.
>> >
>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
>> > [2]
>> >
>> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
>> >
>> > Cheers,
>> > Till
>> >
>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
>> > o.nitavskyi@criteo.com> wrote:
>> >
>> >> Hello Xintong,
>> >>
>> >> Thanks for the insights and support.
>> >>
>> >> Browsing the Mesos backlog and didn't identify anything critical, which
>> >> is left there.
>> >>
>> >> I see that there are were quite a lot of contributions to the Flink
>> Mesos
>> >> in the recent version:
>> >> https://github.com/apache/flink/commits/master/flink-mesos.
>> >> We plan to validate the current Flink master (or release 1.12 branch)
>> our
>> >> Mesos setup. In case of any issues, we will try to propose changes.
>> >> My feeling is that our test results shouldn't affect the Flink 1.12
>> >> release cycle. And if any potential commits will land into the 1.12.1
>> it
>> >> should be totally fine.
>> >>
>> >> In the future, we would be glad to help you guys with any
>> >> maintenance-related questions. One of the highest priorities around
>> this
>> >> component seems to be the development of the full e2e test.
>> >>
>> >> Kind Regards
>> >> Oleksandr Nitavskyi
>> >> ________________________________
>> >> From: Xintong Song <to...@gmail.com>
>> >> Sent: Tuesday, October 27, 2020 7:14 AM
>> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
>> >> Cc: Piyush Narang <p....@criteo.com>
>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
>> >>
>> >> Hi Piyush,
>> >>
>> >> Thanks a lot for sharing the information. It would be a great relief
>> that
>> >> you are good with Flink on Mesos as is.
>> >>
>> >> As for the jira issues, I believe the most essential ones should have
>> >> already been resolved. You may find some remaining open issues here
>> [1],
>> >> but not all of them are necessary if we decide to keep Flink on Mesos
>> as is.
>> >>
>> >> At the moment and in the short future, I think helps are mostly needed
>> on
>> >> testing the upcoming release 1.12 with Mesos use cases. The community
>> is
>> >> currently actively preparing the new release, and hopefully we could
>> come
>> >> up with a release candidate early next month. It would be greatly
>> >> appreciated if you fork as experienced Flink on Mesos users can help
>> with
>> >> verifying the release candidates.
>> >>
>> >>
>> >> Thank you~
>> >>
>> >> Xintong Song
>> >>
>> >> [1]
>> >>
>> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
>> >> <
>> >>
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
>> >> >
>> >>
>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
>> >> <ma...@criteo.com>> wrote:
>> >>
>> >> Hi Xintong,
>> >>
>> >>
>> >>
>> >> Do you have any jiras that cover any of the items on 1 or 2? I can
>> reach
>> >> out to folks internally and see if I can get some folks to commit to
>> >> helping out.
>> >>
>> >>
>> >>
>> >> To cover the other qs:
>> >>
>> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We use
>> >> Yarn for some our Flink workloads when we can. Mesos is only used when
>> we
>> >> need streaming capabilities in our WW dcs (as our Yarn is centralized
>> in
>> >> one DC)
>> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan to
>> bump
>> >> to 1.11 / 1.12 this quarter.
>> >>   *   We typically upgrade once every 6 months to a year (not every
>> >> release). We’d like to speed up the cadence but we’re not there yet.
>> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
>> >> functional while missing out on some of the newer features. We
>> understand
>> >> the pain on the communities side and we can take on the work if we see
>> some
>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put
>> in
>> >> the request to port it over.
>> >>
>> >>
>> >>
>> >> Thanks,
>> >>
>> >>
>> >>
>> >> -- Piyush
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> From: Xintong Song <tonysong820@gmail.com<mailto:tonysong820@gmail.com
>> >>
>> >> Date: Sunday, October 25, 2020 at 10:57 PM
>> >> To: dev <de...@flink.apache.org>>, user <
>> >> user@flink.apache.org<ma...@flink.apache.org>>
>> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
>> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
>> >> p.narang@criteo.com>>
>> >> Subject: Re: [SURVEY] Remove Mesos support
>> >>
>> >>
>> >>
>> >> Thanks for sharing the information with us, Piyush an Lasse.
>> >>
>> >>
>> >>
>> >> @Piyush
>> >>
>> >>
>> >>
>> >> Thanks for offering the help. IMO, there are currently several problems
>> >> that make supporting Flink on Mesos challenging for us.
>> >>
>> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if not
>> >> none) among the active contributors in this community that are familiar
>> >> with Mesos and can help with development on this component.
>> >>   2.  Absence of tests. Mesos does not provide a testing cluster, like
>> >> `MiniYARNCluster`, making it hard to test interactions between Flink
>> and
>> >> Mesos. We have only a few very simple e2e tests running on Mesos
>> deployed
>> >> in a docker, covering the most fundamental workflows. We are not sure
>> how
>> >> well those tests work, especially against some potential corner cases.
>> >>   3.  Divergence from other deployment. Because of 1 and 2, the new
>> >> efforts (features, maintenance, refactors) tend to exclude Mesos if
>> >> possible. When the new efforts have to touch the Mesos related
>> components
>> >> (e.g., changes to the common resource manager interfaces), we have to
>> be
>> >> very careful and make as few changes as possible, to avoid accidentally
>> >> breaking anything that we are not familiar with. As a result, the
>> component
>> >> diverges a lot from other deployment components (K8s/Yarn), which
>> makes it
>> >> harder to maintain.
>> >>
>> >> It would be greatly appreciated if you can help with either of the
>> above
>> >> issues.
>> >>
>> >>
>> >>
>> >> Additionally, I have a few questions concerning your use cases at
>> Criteo.
>> >> IIUC, you are going to stay on Mesos in the foreseeable future, while
>> >> keeping the Flink version up-to-date? What Flink version are you
>> currently
>> >> using? How often do you upgrade (e.g., every release)? Would you be
>> good
>> >> with keeping the Flink on Mesos component as it is (means that
>> deployment
>> >> and resource management improvements may not be ported to Mesos), while
>> >> keeping other components up-to-date (e.g., improvements from
>> programming
>> >> APIs, operators, state backens, etc.)?
>> >>
>> >>
>> >>
>> >> Thank you~
>> >>
>> >> Xintong Song
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
>> >> lassenedergaardflink@gmail.com<ma...@gmail.com>>
>> >> wrote:
>> >>
>> >> Hi
>> >>
>> >>
>> >>
>> >> At Trackunit We have been using Mesos for long time but have now moved
>> to
>> >> k8s.
>> >>
>> >> Med venlig hilsen / Best regards
>> >>
>> >> Lasse Nedergaard
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetzger@apache.org
>> >> <ma...@apache.org>>:
>> >>
>> >> 
>> >>
>> >> Hey Piyush,
>> >>
>> >> thanks a lot for raising this concern. I believe we should keep Mesos
>> in
>> >> Flink then in the foreseeable future.
>> >>
>> >> Your offer to help is much appreciated. We'll let you know once there
>> is
>> >> something.
>> >>
>> >>
>> >>
>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
>> >> <ma...@criteo.com>> wrote:
>> >>
>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be able
>> >> to find folks who would be excited to contribute / help in any way.
>> >>
>> >> -- Piyush
>> >>
>> >>
>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com<mailto:
>> >> kkloudas@gmail.com>> wrote:
>> >>
>> >>     Thanks Piyush for the message.
>> >>     After this, I revoke my +1. I agree with the previous opinions
>> that we
>> >>     cannot drop code that is actively used by users, especially if it
>> >>     something that deep in the stack as support for cluster management
>> >>     framework.
>> >>
>> >>     Cheers,
>> >>     Kostas
>> >>
>> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <p.narang@criteo.com
>> >> <ma...@criteo.com>> wrote:
>> >>     >
>> >>     > Hi folks,
>> >>     >
>> >>     >
>> >>     >
>> >>     > We at Criteo are active users of the Flink on Mesos resource
>> >> management component. We are pretty heavy users of Mesos for scheduling
>> >> workloads on our edge datacenters and we do want to continue to be
>> able to
>> >> run some of our Flink topologies (to compute machine learning short
>> term
>> >> features) on those DCs. If possible our vote would be not to drop Mesos
>> >> support as that will tie us to an old release / have to maintain a
>> fork as
>> >> we’re not planning to migrate off Mesos anytime soon. Is the burden
>> >> something that can be helped with by the community? (Or are you
>> referring
>> >> to having to ensure PRs handle the Mesos piece as well when they touch
>> the
>> >> resource managers?)
>> >>     >
>> >>     >
>> >>     >
>> >>     > Thanks,
>> >>     >
>> >>     >
>> >>     >
>> >>     > -- Piyush
>> >>     >
>> >>     >
>> >>     >
>> >>     >
>> >>     >
>> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
>> >> trohrmann@apache.org>>
>> >>     > Date: Friday, October 23, 2020 at 8:19 AM
>> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
>> >> tonysong820@gmail.com>>
>> >>     > Cc: dev <de...@flink.apache.org>>,
>> user <
>> >> user@flink.apache.org<ma...@flink.apache.org>>
>> >>     > Subject: Re: [SURVEY] Remove Mesos support
>> >>     >
>> >>     >
>> >>     >
>> >>     > Thanks for starting this survey Robert! I second Konstantin and
>> >> Xintong in the sense that our Mesos user's opinions should matter most
>> >> here. If our community is no longer using the Mesos integration, then I
>> >> would be +1 for removing it in order to decrease the maintenance
>> burden.
>> >>     >
>> >>     >
>> >>     >
>> >>     > Cheers,
>> >>     >
>> >>     > Till
>> >>     >
>> >>     >
>> >>     >
>> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <
>> tonysong820@gmail.com
>> >> <ma...@gmail.com>> wrote:
>> >>     >
>> >>     > +1 for adding a warning in 1.12 about planning to remove Mesos
>> >> support.
>> >>     >
>> >>     >
>> >>     >
>> >>     > With my developer hat on, removing the Mesos support would
>> >> definitely reduce the maintaining overhead for the deployment and
>> resource
>> >> management related components. On the other hand, the Flink on Mesos
>> users'
>> >> voices definitely matter a lot for this community. Either way, it
>> would be
>> >> good to draw users attention to this discussion early.
>> >>     >
>> >>     >
>> >>     >
>> >>     > Thank you~
>> >>     >
>> >>     > Xintong Song
>> >>     >
>> >>     >
>> >>     >
>> >>     >
>> >>     >
>> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <
>> knaufk@apache.org
>> >> <ma...@apache.org>> wrote:
>> >>     >
>> >>     > Hi Robert,
>> >>     >
>> >>     > +1 to the plan you outlined. If we were to drop support in Flink
>> >> 1.13+, we
>> >>     > would still support it in Flink 1.12- with bug fixes for some
>> time
>> >> so that
>> >>     > users have time to move on.
>> >>     >
>> >>     > It would certainly be very interesting to hear from current Flink
>> >> on Mesos
>> >>     > users, on how they see the evolution of this part of the
>> ecosystem.
>> >>     >
>> >>     > Best,
>> >>     >
>> >>     > Konstantin
>> >>
>> >
>>
>
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Till Rohrmann <tr...@apache.org>.
+1 for officially deprecating this component for the 1.13 release.

Cheers,
Till

On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kn...@apache.org> wrote:

> Hi Matthias,
>
> Thank you for following up on this. +1 to officially deprecate Mesos in
> the code and documentation, too. It will be confusing for users if this
> diverges from the roadmap.
>
> Cheers,
>
> Konstantin
>
> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <ma...@ververica.com>
> wrote:
>
>> Hi everyone,
>> considering the upcoming release of Flink 1.13, I wanted to revive the
>> discussion about the Mesos support ones more. Mesos is also already listed
>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align
>> the
>> documentation accordingly to make it more explicit?
>>
>> What do you think?
>>
>> Best,
>> Matthias
>>
>> [1] https://flink.apache.org/roadmap.html#feature-radar
>>
>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>> > Hi Oleksandr,
>> >
>> > yes you are right. The biggest problem is at the moment the lack of test
>> > coverage and thereby confidence to make changes. We have some e2e tests
>> > which you can find here [1]. These tests are, however, quite coarse
>> grained
>> > and are missing a lot of cases. One idea would be to add a Mesos e2e
>> test
>> > based on Flink's end-to-end test framework [2]. I think what needs to be
>> > done there is to add a Mesos resource and a way to submit jobs to a
>> Mesos
>> > cluster to write e2e tests.
>> >
>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
>> > [2]
>> >
>> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
>> >
>> > Cheers,
>> > Till
>> >
>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
>> > o.nitavskyi@criteo.com> wrote:
>> >
>> >> Hello Xintong,
>> >>
>> >> Thanks for the insights and support.
>> >>
>> >> Browsing the Mesos backlog and didn't identify anything critical, which
>> >> is left there.
>> >>
>> >> I see that there are were quite a lot of contributions to the Flink
>> Mesos
>> >> in the recent version:
>> >> https://github.com/apache/flink/commits/master/flink-mesos.
>> >> We plan to validate the current Flink master (or release 1.12 branch)
>> our
>> >> Mesos setup. In case of any issues, we will try to propose changes.
>> >> My feeling is that our test results shouldn't affect the Flink 1.12
>> >> release cycle. And if any potential commits will land into the 1.12.1
>> it
>> >> should be totally fine.
>> >>
>> >> In the future, we would be glad to help you guys with any
>> >> maintenance-related questions. One of the highest priorities around
>> this
>> >> component seems to be the development of the full e2e test.
>> >>
>> >> Kind Regards
>> >> Oleksandr Nitavskyi
>> >> ________________________________
>> >> From: Xintong Song <to...@gmail.com>
>> >> Sent: Tuesday, October 27, 2020 7:14 AM
>> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
>> >> Cc: Piyush Narang <p....@criteo.com>
>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
>> >>
>> >> Hi Piyush,
>> >>
>> >> Thanks a lot for sharing the information. It would be a great relief
>> that
>> >> you are good with Flink on Mesos as is.
>> >>
>> >> As for the jira issues, I believe the most essential ones should have
>> >> already been resolved. You may find some remaining open issues here
>> [1],
>> >> but not all of them are necessary if we decide to keep Flink on Mesos
>> as is.
>> >>
>> >> At the moment and in the short future, I think helps are mostly needed
>> on
>> >> testing the upcoming release 1.12 with Mesos use cases. The community
>> is
>> >> currently actively preparing the new release, and hopefully we could
>> come
>> >> up with a release candidate early next month. It would be greatly
>> >> appreciated if you fork as experienced Flink on Mesos users can help
>> with
>> >> verifying the release candidates.
>> >>
>> >>
>> >> Thank you~
>> >>
>> >> Xintong Song
>> >>
>> >> [1]
>> >>
>> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
>> >> <
>> >>
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
>> >> >
>> >>
>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
>> >> <ma...@criteo.com>> wrote:
>> >>
>> >> Hi Xintong,
>> >>
>> >>
>> >>
>> >> Do you have any jiras that cover any of the items on 1 or 2? I can
>> reach
>> >> out to folks internally and see if I can get some folks to commit to
>> >> helping out.
>> >>
>> >>
>> >>
>> >> To cover the other qs:
>> >>
>> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We use
>> >> Yarn for some our Flink workloads when we can. Mesos is only used when
>> we
>> >> need streaming capabilities in our WW dcs (as our Yarn is centralized
>> in
>> >> one DC)
>> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan to
>> bump
>> >> to 1.11 / 1.12 this quarter.
>> >>   *   We typically upgrade once every 6 months to a year (not every
>> >> release). We’d like to speed up the cadence but we’re not there yet.
>> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
>> >> functional while missing out on some of the newer features. We
>> understand
>> >> the pain on the communities side and we can take on the work if we see
>> some
>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put
>> in
>> >> the request to port it over.
>> >>
>> >>
>> >>
>> >> Thanks,
>> >>
>> >>
>> >>
>> >> -- Piyush
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> From: Xintong Song <tonysong820@gmail.com<mailto:tonysong820@gmail.com
>> >>
>> >> Date: Sunday, October 25, 2020 at 10:57 PM
>> >> To: dev <de...@flink.apache.org>>, user <
>> >> user@flink.apache.org<ma...@flink.apache.org>>
>> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
>> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
>> >> p.narang@criteo.com>>
>> >> Subject: Re: [SURVEY] Remove Mesos support
>> >>
>> >>
>> >>
>> >> Thanks for sharing the information with us, Piyush an Lasse.
>> >>
>> >>
>> >>
>> >> @Piyush
>> >>
>> >>
>> >>
>> >> Thanks for offering the help. IMO, there are currently several problems
>> >> that make supporting Flink on Mesos challenging for us.
>> >>
>> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if not
>> >> none) among the active contributors in this community that are familiar
>> >> with Mesos and can help with development on this component.
>> >>   2.  Absence of tests. Mesos does not provide a testing cluster, like
>> >> `MiniYARNCluster`, making it hard to test interactions between Flink
>> and
>> >> Mesos. We have only a few very simple e2e tests running on Mesos
>> deployed
>> >> in a docker, covering the most fundamental workflows. We are not sure
>> how
>> >> well those tests work, especially against some potential corner cases.
>> >>   3.  Divergence from other deployment. Because of 1 and 2, the new
>> >> efforts (features, maintenance, refactors) tend to exclude Mesos if
>> >> possible. When the new efforts have to touch the Mesos related
>> components
>> >> (e.g., changes to the common resource manager interfaces), we have to
>> be
>> >> very careful and make as few changes as possible, to avoid accidentally
>> >> breaking anything that we are not familiar with. As a result, the
>> component
>> >> diverges a lot from other deployment components (K8s/Yarn), which
>> makes it
>> >> harder to maintain.
>> >>
>> >> It would be greatly appreciated if you can help with either of the
>> above
>> >> issues.
>> >>
>> >>
>> >>
>> >> Additionally, I have a few questions concerning your use cases at
>> Criteo.
>> >> IIUC, you are going to stay on Mesos in the foreseeable future, while
>> >> keeping the Flink version up-to-date? What Flink version are you
>> currently
>> >> using? How often do you upgrade (e.g., every release)? Would you be
>> good
>> >> with keeping the Flink on Mesos component as it is (means that
>> deployment
>> >> and resource management improvements may not be ported to Mesos), while
>> >> keeping other components up-to-date (e.g., improvements from
>> programming
>> >> APIs, operators, state backens, etc.)?
>> >>
>> >>
>> >>
>> >> Thank you~
>> >>
>> >> Xintong Song
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
>> >> lassenedergaardflink@gmail.com<ma...@gmail.com>>
>> >> wrote:
>> >>
>> >> Hi
>> >>
>> >>
>> >>
>> >> At Trackunit We have been using Mesos for long time but have now moved
>> to
>> >> k8s.
>> >>
>> >> Med venlig hilsen / Best regards
>> >>
>> >> Lasse Nedergaard
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetzger@apache.org
>> >> <ma...@apache.org>>:
>> >>
>> >> 
>> >>
>> >> Hey Piyush,
>> >>
>> >> thanks a lot for raising this concern. I believe we should keep Mesos
>> in
>> >> Flink then in the foreseeable future.
>> >>
>> >> Your offer to help is much appreciated. We'll let you know once there
>> is
>> >> something.
>> >>
>> >>
>> >>
>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
>> >> <ma...@criteo.com>> wrote:
>> >>
>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be able
>> >> to find folks who would be excited to contribute / help in any way.
>> >>
>> >> -- Piyush
>> >>
>> >>
>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com<mailto:
>> >> kkloudas@gmail.com>> wrote:
>> >>
>> >>     Thanks Piyush for the message.
>> >>     After this, I revoke my +1. I agree with the previous opinions
>> that we
>> >>     cannot drop code that is actively used by users, especially if it
>> >>     something that deep in the stack as support for cluster management
>> >>     framework.
>> >>
>> >>     Cheers,
>> >>     Kostas
>> >>
>> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <p.narang@criteo.com
>> >> <ma...@criteo.com>> wrote:
>> >>     >
>> >>     > Hi folks,
>> >>     >
>> >>     >
>> >>     >
>> >>     > We at Criteo are active users of the Flink on Mesos resource
>> >> management component. We are pretty heavy users of Mesos for scheduling
>> >> workloads on our edge datacenters and we do want to continue to be
>> able to
>> >> run some of our Flink topologies (to compute machine learning short
>> term
>> >> features) on those DCs. If possible our vote would be not to drop Mesos
>> >> support as that will tie us to an old release / have to maintain a
>> fork as
>> >> we’re not planning to migrate off Mesos anytime soon. Is the burden
>> >> something that can be helped with by the community? (Or are you
>> referring
>> >> to having to ensure PRs handle the Mesos piece as well when they touch
>> the
>> >> resource managers?)
>> >>     >
>> >>     >
>> >>     >
>> >>     > Thanks,
>> >>     >
>> >>     >
>> >>     >
>> >>     > -- Piyush
>> >>     >
>> >>     >
>> >>     >
>> >>     >
>> >>     >
>> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
>> >> trohrmann@apache.org>>
>> >>     > Date: Friday, October 23, 2020 at 8:19 AM
>> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
>> >> tonysong820@gmail.com>>
>> >>     > Cc: dev <de...@flink.apache.org>>,
>> user <
>> >> user@flink.apache.org<ma...@flink.apache.org>>
>> >>     > Subject: Re: [SURVEY] Remove Mesos support
>> >>     >
>> >>     >
>> >>     >
>> >>     > Thanks for starting this survey Robert! I second Konstantin and
>> >> Xintong in the sense that our Mesos user's opinions should matter most
>> >> here. If our community is no longer using the Mesos integration, then I
>> >> would be +1 for removing it in order to decrease the maintenance
>> burden.
>> >>     >
>> >>     >
>> >>     >
>> >>     > Cheers,
>> >>     >
>> >>     > Till
>> >>     >
>> >>     >
>> >>     >
>> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <
>> tonysong820@gmail.com
>> >> <ma...@gmail.com>> wrote:
>> >>     >
>> >>     > +1 for adding a warning in 1.12 about planning to remove Mesos
>> >> support.
>> >>     >
>> >>     >
>> >>     >
>> >>     > With my developer hat on, removing the Mesos support would
>> >> definitely reduce the maintaining overhead for the deployment and
>> resource
>> >> management related components. On the other hand, the Flink on Mesos
>> users'
>> >> voices definitely matter a lot for this community. Either way, it
>> would be
>> >> good to draw users attention to this discussion early.
>> >>     >
>> >>     >
>> >>     >
>> >>     > Thank you~
>> >>     >
>> >>     > Xintong Song
>> >>     >
>> >>     >
>> >>     >
>> >>     >
>> >>     >
>> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <
>> knaufk@apache.org
>> >> <ma...@apache.org>> wrote:
>> >>     >
>> >>     > Hi Robert,
>> >>     >
>> >>     > +1 to the plan you outlined. If we were to drop support in Flink
>> >> 1.13+, we
>> >>     > would still support it in Flink 1.12- with bug fixes for some
>> time
>> >> so that
>> >>     > users have time to move on.
>> >>     >
>> >>     > It would certainly be very interesting to hear from current Flink
>> >> on Mesos
>> >>     > users, on how they see the evolution of this part of the
>> ecosystem.
>> >>     >
>> >>     > Best,
>> >>     >
>> >>     > Konstantin
>> >>
>> >
>>
>
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Konstantin Knauf <kn...@apache.org>.
Hi Matthias,

Thank you for following up on this. +1 to officially deprecate Mesos in the
code and documentation, too. It will be confusing for users if this
diverges from the roadmap.

Cheers,

Konstantin

On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <ma...@ververica.com>
wrote:

> Hi everyone,
> considering the upcoming release of Flink 1.13, I wanted to revive the
> discussion about the Mesos support ones more. Mesos is also already listed
> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align the
> documentation accordingly to make it more explicit?
>
> What do you think?
>
> Best,
> Matthias
>
> [1] https://flink.apache.org/roadmap.html#feature-radar
>
> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Hi Oleksandr,
> >
> > yes you are right. The biggest problem is at the moment the lack of test
> > coverage and thereby confidence to make changes. We have some e2e tests
> > which you can find here [1]. These tests are, however, quite coarse
> grained
> > and are missing a lot of cases. One idea would be to add a Mesos e2e test
> > based on Flink's end-to-end test framework [2]. I think what needs to be
> > done there is to add a Mesos resource and a way to submit jobs to a Mesos
> > cluster to write e2e tests.
> >
> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
> > [2]
> >
> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
> >
> > Cheers,
> > Till
> >
> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
> > o.nitavskyi@criteo.com> wrote:
> >
> >> Hello Xintong,
> >>
> >> Thanks for the insights and support.
> >>
> >> Browsing the Mesos backlog and didn't identify anything critical, which
> >> is left there.
> >>
> >> I see that there are were quite a lot of contributions to the Flink
> Mesos
> >> in the recent version:
> >> https://github.com/apache/flink/commits/master/flink-mesos.
> >> We plan to validate the current Flink master (or release 1.12 branch)
> our
> >> Mesos setup. In case of any issues, we will try to propose changes.
> >> My feeling is that our test results shouldn't affect the Flink 1.12
> >> release cycle. And if any potential commits will land into the 1.12.1 it
> >> should be totally fine.
> >>
> >> In the future, we would be glad to help you guys with any
> >> maintenance-related questions. One of the highest priorities around this
> >> component seems to be the development of the full e2e test.
> >>
> >> Kind Regards
> >> Oleksandr Nitavskyi
> >> ________________________________
> >> From: Xintong Song <to...@gmail.com>
> >> Sent: Tuesday, October 27, 2020 7:14 AM
> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
> >> Cc: Piyush Narang <p....@criteo.com>
> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
> >>
> >> Hi Piyush,
> >>
> >> Thanks a lot for sharing the information. It would be a great relief
> that
> >> you are good with Flink on Mesos as is.
> >>
> >> As for the jira issues, I believe the most essential ones should have
> >> already been resolved. You may find some remaining open issues here [1],
> >> but not all of them are necessary if we decide to keep Flink on Mesos
> as is.
> >>
> >> At the moment and in the short future, I think helps are mostly needed
> on
> >> testing the upcoming release 1.12 with Mesos use cases. The community is
> >> currently actively preparing the new release, and hopefully we could
> come
> >> up with a release candidate early next month. It would be greatly
> >> appreciated if you fork as experienced Flink on Mesos users can help
> with
> >> verifying the release candidates.
> >>
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >> [1]
> >>
> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
> >> <
> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
> >> >
> >>
> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
> >> <ma...@criteo.com>> wrote:
> >>
> >> Hi Xintong,
> >>
> >>
> >>
> >> Do you have any jiras that cover any of the items on 1 or 2? I can reach
> >> out to folks internally and see if I can get some folks to commit to
> >> helping out.
> >>
> >>
> >>
> >> To cover the other qs:
> >>
> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We use
> >> Yarn for some our Flink workloads when we can. Mesos is only used when
> we
> >> need streaming capabilities in our WW dcs (as our Yarn is centralized in
> >> one DC)
> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan to bump
> >> to 1.11 / 1.12 this quarter.
> >>   *   We typically upgrade once every 6 months to a year (not every
> >> release). We’d like to speed up the cadence but we’re not there yet.
> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
> >> functional while missing out on some of the newer features. We
> understand
> >> the pain on the communities side and we can take on the work if we see
> some
> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put in
> >> the request to port it over.
> >>
> >>
> >>
> >> Thanks,
> >>
> >>
> >>
> >> -- Piyush
> >>
> >>
> >>
> >>
> >>
> >> From: Xintong Song <tonysong820@gmail.com<mailto:tonysong820@gmail.com
> >>
> >> Date: Sunday, October 25, 2020 at 10:57 PM
> >> To: dev <de...@flink.apache.org>>, user <
> >> user@flink.apache.org<ma...@flink.apache.org>>
> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
> >> p.narang@criteo.com>>
> >> Subject: Re: [SURVEY] Remove Mesos support
> >>
> >>
> >>
> >> Thanks for sharing the information with us, Piyush an Lasse.
> >>
> >>
> >>
> >> @Piyush
> >>
> >>
> >>
> >> Thanks for offering the help. IMO, there are currently several problems
> >> that make supporting Flink on Mesos challenging for us.
> >>
> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if not
> >> none) among the active contributors in this community that are familiar
> >> with Mesos and can help with development on this component.
> >>   2.  Absence of tests. Mesos does not provide a testing cluster, like
> >> `MiniYARNCluster`, making it hard to test interactions between Flink and
> >> Mesos. We have only a few very simple e2e tests running on Mesos
> deployed
> >> in a docker, covering the most fundamental workflows. We are not sure
> how
> >> well those tests work, especially against some potential corner cases.
> >>   3.  Divergence from other deployment. Because of 1 and 2, the new
> >> efforts (features, maintenance, refactors) tend to exclude Mesos if
> >> possible. When the new efforts have to touch the Mesos related
> components
> >> (e.g., changes to the common resource manager interfaces), we have to be
> >> very careful and make as few changes as possible, to avoid accidentally
> >> breaking anything that we are not familiar with. As a result, the
> component
> >> diverges a lot from other deployment components (K8s/Yarn), which makes
> it
> >> harder to maintain.
> >>
> >> It would be greatly appreciated if you can help with either of the above
> >> issues.
> >>
> >>
> >>
> >> Additionally, I have a few questions concerning your use cases at
> Criteo.
> >> IIUC, you are going to stay on Mesos in the foreseeable future, while
> >> keeping the Flink version up-to-date? What Flink version are you
> currently
> >> using? How often do you upgrade (e.g., every release)? Would you be good
> >> with keeping the Flink on Mesos component as it is (means that
> deployment
> >> and resource management improvements may not be ported to Mesos), while
> >> keeping other components up-to-date (e.g., improvements from programming
> >> APIs, operators, state backens, etc.)?
> >>
> >>
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
> >> lassenedergaardflink@gmail.com<ma...@gmail.com>>
> >> wrote:
> >>
> >> Hi
> >>
> >>
> >>
> >> At Trackunit We have been using Mesos for long time but have now moved
> to
> >> k8s.
> >>
> >> Med venlig hilsen / Best regards
> >>
> >> Lasse Nedergaard
> >>
> >>
> >>
> >>
> >>
> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetzger@apache.org
> >> <ma...@apache.org>>:
> >>
> >> 
> >>
> >> Hey Piyush,
> >>
> >> thanks a lot for raising this concern. I believe we should keep Mesos in
> >> Flink then in the foreseeable future.
> >>
> >> Your offer to help is much appreciated. We'll let you know once there is
> >> something.
> >>
> >>
> >>
> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
> >> <ma...@criteo.com>> wrote:
> >>
> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be able
> >> to find folks who would be excited to contribute / help in any way.
> >>
> >> -- Piyush
> >>
> >>
> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com<mailto:
> >> kkloudas@gmail.com>> wrote:
> >>
> >>     Thanks Piyush for the message.
> >>     After this, I revoke my +1. I agree with the previous opinions that
> we
> >>     cannot drop code that is actively used by users, especially if it
> >>     something that deep in the stack as support for cluster management
> >>     framework.
> >>
> >>     Cheers,
> >>     Kostas
> >>
> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <p.narang@criteo.com
> >> <ma...@criteo.com>> wrote:
> >>     >
> >>     > Hi folks,
> >>     >
> >>     >
> >>     >
> >>     > We at Criteo are active users of the Flink on Mesos resource
> >> management component. We are pretty heavy users of Mesos for scheduling
> >> workloads on our edge datacenters and we do want to continue to be able
> to
> >> run some of our Flink topologies (to compute machine learning short term
> >> features) on those DCs. If possible our vote would be not to drop Mesos
> >> support as that will tie us to an old release / have to maintain a fork
> as
> >> we’re not planning to migrate off Mesos anytime soon. Is the burden
> >> something that can be helped with by the community? (Or are you
> referring
> >> to having to ensure PRs handle the Mesos piece as well when they touch
> the
> >> resource managers?)
> >>     >
> >>     >
> >>     >
> >>     > Thanks,
> >>     >
> >>     >
> >>     >
> >>     > -- Piyush
> >>     >
> >>     >
> >>     >
> >>     >
> >>     >
> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
> >> trohrmann@apache.org>>
> >>     > Date: Friday, October 23, 2020 at 8:19 AM
> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
> >> tonysong820@gmail.com>>
> >>     > Cc: dev <de...@flink.apache.org>>,
> user <
> >> user@flink.apache.org<ma...@flink.apache.org>>
> >>     > Subject: Re: [SURVEY] Remove Mesos support
> >>     >
> >>     >
> >>     >
> >>     > Thanks for starting this survey Robert! I second Konstantin and
> >> Xintong in the sense that our Mesos user's opinions should matter most
> >> here. If our community is no longer using the Mesos integration, then I
> >> would be +1 for removing it in order to decrease the maintenance burden.
> >>     >
> >>     >
> >>     >
> >>     > Cheers,
> >>     >
> >>     > Till
> >>     >
> >>     >
> >>     >
> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <
> tonysong820@gmail.com
> >> <ma...@gmail.com>> wrote:
> >>     >
> >>     > +1 for adding a warning in 1.12 about planning to remove Mesos
> >> support.
> >>     >
> >>     >
> >>     >
> >>     > With my developer hat on, removing the Mesos support would
> >> definitely reduce the maintaining overhead for the deployment and
> resource
> >> management related components. On the other hand, the Flink on Mesos
> users'
> >> voices definitely matter a lot for this community. Either way, it would
> be
> >> good to draw users attention to this discussion early.
> >>     >
> >>     >
> >>     >
> >>     > Thank you~
> >>     >
> >>     > Xintong Song
> >>     >
> >>     >
> >>     >
> >>     >
> >>     >
> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <
> knaufk@apache.org
> >> <ma...@apache.org>> wrote:
> >>     >
> >>     > Hi Robert,
> >>     >
> >>     > +1 to the plan you outlined. If we were to drop support in Flink
> >> 1.13+, we
> >>     > would still support it in Flink 1.12- with bug fixes for some time
> >> so that
> >>     > users have time to move on.
> >>     >
> >>     > It would certainly be very interesting to hear from current Flink
> >> on Mesos
> >>     > users, on how they see the evolution of this part of the
> ecosystem.
> >>     >
> >>     > Best,
> >>     >
> >>     > Konstantin
> >>
> >
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [BULK]Re: [SURVEY] Remove Mesos support

Posted by Konstantin Knauf <kn...@apache.org>.
Hi Matthias,

Thank you for following up on this. +1 to officially deprecate Mesos in the
code and documentation, too. It will be confusing for users if this
diverges from the roadmap.

Cheers,

Konstantin

On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <ma...@ververica.com>
wrote:

> Hi everyone,
> considering the upcoming release of Flink 1.13, I wanted to revive the
> discussion about the Mesos support ones more. Mesos is also already listed
> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align the
> documentation accordingly to make it more explicit?
>
> What do you think?
>
> Best,
> Matthias
>
> [1] https://flink.apache.org/roadmap.html#feature-radar
>
> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Hi Oleksandr,
> >
> > yes you are right. The biggest problem is at the moment the lack of test
> > coverage and thereby confidence to make changes. We have some e2e tests
> > which you can find here [1]. These tests are, however, quite coarse
> grained
> > and are missing a lot of cases. One idea would be to add a Mesos e2e test
> > based on Flink's end-to-end test framework [2]. I think what needs to be
> > done there is to add a Mesos resource and a way to submit jobs to a Mesos
> > cluster to write e2e tests.
> >
> > [1] https://github.com/apache/flink/tree/master/flink-jepsen
> > [2]
> >
> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
> >
> > Cheers,
> > Till
> >
> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi <
> > o.nitavskyi@criteo.com> wrote:
> >
> >> Hello Xintong,
> >>
> >> Thanks for the insights and support.
> >>
> >> Browsing the Mesos backlog and didn't identify anything critical, which
> >> is left there.
> >>
> >> I see that there are were quite a lot of contributions to the Flink
> Mesos
> >> in the recent version:
> >> https://github.com/apache/flink/commits/master/flink-mesos.
> >> We plan to validate the current Flink master (or release 1.12 branch)
> our
> >> Mesos setup. In case of any issues, we will try to propose changes.
> >> My feeling is that our test results shouldn't affect the Flink 1.12
> >> release cycle. And if any potential commits will land into the 1.12.1 it
> >> should be totally fine.
> >>
> >> In the future, we would be glad to help you guys with any
> >> maintenance-related questions. One of the highest priorities around this
> >> component seems to be the development of the full e2e test.
> >>
> >> Kind Regards
> >> Oleksandr Nitavskyi
> >> ________________________________
> >> From: Xintong Song <to...@gmail.com>
> >> Sent: Tuesday, October 27, 2020 7:14 AM
> >> To: dev <de...@flink.apache.org>; user <us...@flink.apache.org>
> >> Cc: Piyush Narang <p....@criteo.com>
> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support
> >>
> >> Hi Piyush,
> >>
> >> Thanks a lot for sharing the information. It would be a great relief
> that
> >> you are good with Flink on Mesos as is.
> >>
> >> As for the jira issues, I believe the most essential ones should have
> >> already been resolved. You may find some remaining open issues here [1],
> >> but not all of them are necessary if we decide to keep Flink on Mesos
> as is.
> >>
> >> At the moment and in the short future, I think helps are mostly needed
> on
> >> testing the upcoming release 1.12 with Mesos use cases. The community is
> >> currently actively preparing the new release, and hopefully we could
> come
> >> up with a release candidate early next month. It would be greatly
> >> appreciated if you fork as experienced Flink on Mesos users can help
> with
> >> verifying the release candidates.
> >>
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >> [1]
> >>
> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
> >> <
> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0
> >> >
> >>
> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.narang@criteo.com
> >> <ma...@criteo.com>> wrote:
> >>
> >> Hi Xintong,
> >>
> >>
> >>
> >> Do you have any jiras that cover any of the items on 1 or 2? I can reach
> >> out to folks internally and see if I can get some folks to commit to
> >> helping out.
> >>
> >>
> >>
> >> To cover the other qs:
> >>
> >>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We use
> >> Yarn for some our Flink workloads when we can. Mesos is only used when
> we
> >> need streaming capabilities in our WW dcs (as our Yarn is centralized in
> >> one DC)
> >>   *   We’re currently on Flink 1.9 (old planner). We have a plan to bump
> >> to 1.11 / 1.12 this quarter.
> >>   *   We typically upgrade once every 6 months to a year (not every
> >> release). We’d like to speed up the cadence but we’re not there yet.
> >>   *   We’d largely be good with keeping Flink on Mesos as-is and
> >> functional while missing out on some of the newer features. We
> understand
> >> the pain on the communities side and we can take on the work if we see
> some
> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put in
> >> the request to port it over.
> >>
> >>
> >>
> >> Thanks,
> >>
> >>
> >>
> >> -- Piyush
> >>
> >>
> >>
> >>
> >>
> >> From: Xintong Song <tonysong820@gmail.com<mailto:tonysong820@gmail.com
> >>
> >> Date: Sunday, October 25, 2020 at 10:57 PM
> >> To: dev <de...@flink.apache.org>>, user <
> >> user@flink.apache.org<ma...@flink.apache.org>>
> >> Cc: Lasse Nedergaard <lassenedergaardflink@gmail.com<mailto:
> >> lassenedergaardflink@gmail.com>>, <p.narang@criteo.com<mailto:
> >> p.narang@criteo.com>>
> >> Subject: Re: [SURVEY] Remove Mesos support
> >>
> >>
> >>
> >> Thanks for sharing the information with us, Piyush an Lasse.
> >>
> >>
> >>
> >> @Piyush
> >>
> >>
> >>
> >> Thanks for offering the help. IMO, there are currently several problems
> >> that make supporting Flink on Mesos challenging for us.
> >>
> >>   1.  Lack of Mesos experts. AFAIK, there are very few people (if not
> >> none) among the active contributors in this community that are familiar
> >> with Mesos and can help with development on this component.
> >>   2.  Absence of tests. Mesos does not provide a testing cluster, like
> >> `MiniYARNCluster`, making it hard to test interactions between Flink and
> >> Mesos. We have only a few very simple e2e tests running on Mesos
> deployed
> >> in a docker, covering the most fundamental workflows. We are not sure
> how
> >> well those tests work, especially against some potential corner cases.
> >>   3.  Divergence from other deployment. Because of 1 and 2, the new
> >> efforts (features, maintenance, refactors) tend to exclude Mesos if
> >> possible. When the new efforts have to touch the Mesos related
> components
> >> (e.g., changes to the common resource manager interfaces), we have to be
> >> very careful and make as few changes as possible, to avoid accidentally
> >> breaking anything that we are not familiar with. As a result, the
> component
> >> diverges a lot from other deployment components (K8s/Yarn), which makes
> it
> >> harder to maintain.
> >>
> >> It would be greatly appreciated if you can help with either of the above
> >> issues.
> >>
> >>
> >>
> >> Additionally, I have a few questions concerning your use cases at
> Criteo.
> >> IIUC, you are going to stay on Mesos in the foreseeable future, while
> >> keeping the Flink version up-to-date? What Flink version are you
> currently
> >> using? How often do you upgrade (e.g., every release)? Would you be good
> >> with keeping the Flink on Mesos component as it is (means that
> deployment
> >> and resource management improvements may not be ported to Mesos), while
> >> keeping other components up-to-date (e.g., improvements from programming
> >> APIs, operators, state backens, etc.)?
> >>
> >>
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard <
> >> lassenedergaardflink@gmail.com<ma...@gmail.com>>
> >> wrote:
> >>
> >> Hi
> >>
> >>
> >>
> >> At Trackunit We have been using Mesos for long time but have now moved
> to
> >> k8s.
> >>
> >> Med venlig hilsen / Best regards
> >>
> >> Lasse Nedergaard
> >>
> >>
> >>
> >>
> >>
> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetzger@apache.org
> >> <ma...@apache.org>>:
> >>
> >> 
> >>
> >> Hey Piyush,
> >>
> >> thanks a lot for raising this concern. I believe we should keep Mesos in
> >> Flink then in the foreseeable future.
> >>
> >> Your offer to help is much appreciated. We'll let you know once there is
> >> something.
> >>
> >>
> >>
> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.narang@criteo.com
> >> <ma...@criteo.com>> wrote:
> >>
> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be able
> >> to find folks who would be excited to contribute / help in any way.
> >>
> >> -- Piyush
> >>
> >>
> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kkloudas@gmail.com<mailto:
> >> kkloudas@gmail.com>> wrote:
> >>
> >>     Thanks Piyush for the message.
> >>     After this, I revoke my +1. I agree with the previous opinions that
> we
> >>     cannot drop code that is actively used by users, especially if it
> >>     something that deep in the stack as support for cluster management
> >>     framework.
> >>
> >>     Cheers,
> >>     Kostas
> >>
> >>     On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang <p.narang@criteo.com
> >> <ma...@criteo.com>> wrote:
> >>     >
> >>     > Hi folks,
> >>     >
> >>     >
> >>     >
> >>     > We at Criteo are active users of the Flink on Mesos resource
> >> management component. We are pretty heavy users of Mesos for scheduling
> >> workloads on our edge datacenters and we do want to continue to be able
> to
> >> run some of our Flink topologies (to compute machine learning short term
> >> features) on those DCs. If possible our vote would be not to drop Mesos
> >> support as that will tie us to an old release / have to maintain a fork
> as
> >> we’re not planning to migrate off Mesos anytime soon. Is the burden
> >> something that can be helped with by the community? (Or are you
> referring
> >> to having to ensure PRs handle the Mesos piece as well when they touch
> the
> >> resource managers?)
> >>     >
> >>     >
> >>     >
> >>     > Thanks,
> >>     >
> >>     >
> >>     >
> >>     > -- Piyush
> >>     >
> >>     >
> >>     >
> >>     >
> >>     >
> >>     > From: Till Rohrmann <trohrmann@apache.org<mailto:
> >> trohrmann@apache.org>>
> >>     > Date: Friday, October 23, 2020 at 8:19 AM
> >>     > To: Xintong Song <tonysong820@gmail.com<mailto:
> >> tonysong820@gmail.com>>
> >>     > Cc: dev <de...@flink.apache.org>>,
> user <
> >> user@flink.apache.org<ma...@flink.apache.org>>
> >>     > Subject: Re: [SURVEY] Remove Mesos support
> >>     >
> >>     >
> >>     >
> >>     > Thanks for starting this survey Robert! I second Konstantin and
> >> Xintong in the sense that our Mesos user's opinions should matter most
> >> here. If our community is no longer using the Mesos integration, then I
> >> would be +1 for removing it in order to decrease the maintenance burden.
> >>     >
> >>     >
> >>     >
> >>     > Cheers,
> >>     >
> >>     > Till
> >>     >
> >>     >
> >>     >
> >>     > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song <
> tonysong820@gmail.com
> >> <ma...@gmail.com>> wrote:
> >>     >
> >>     > +1 for adding a warning in 1.12 about planning to remove Mesos
> >> support.
> >>     >
> >>     >
> >>     >
> >>     > With my developer hat on, removing the Mesos support would
> >> definitely reduce the maintaining overhead for the deployment and
> resource
> >> management related components. On the other hand, the Flink on Mesos
> users'
> >> voices definitely matter a lot for this community. Either way, it would
> be
> >> good to draw users attention to this discussion early.
> >>     >
> >>     >
> >>     >
> >>     > Thank you~
> >>     >
> >>     > Xintong Song
> >>     >
> >>     >
> >>     >
> >>     >
> >>     >
> >>     > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf <
> knaufk@apache.org
> >> <ma...@apache.org>> wrote:
> >>     >
> >>     > Hi Robert,
> >>     >
> >>     > +1 to the plan you outlined. If we were to drop support in Flink
> >> 1.13+, we
> >>     > would still support it in Flink 1.12- with bug fixes for some time
> >> so that
> >>     > users have time to move on.
> >>     >
> >>     > It would certainly be very interesting to hear from current Flink
> >> on Mesos
> >>     > users, on how they see the evolution of this part of the
> ecosystem.
> >>     >
> >>     > Best,
> >>     >
> >>     > Konstantin
> >>
> >
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk