You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Chesnay Schepler <ch...@apache.org> on 2021/06/22 08:29:36 UTC
[DISCUSS] Drop Mesos in 1.14
Last week I spent some time looking into making flink-runtime scala
free, which effectively means to move the Akka-reliant classes to
another module, and load that module along with Akka and all of it's
dependencies (including Scala) through a separate classloader.
This would finally decouple the Scala versions required by the runtime
and API, and would allow us to upgrade Akka as we'd no longer be limited
to Scala 2.11. It would rid the classpath of a few dependencies, and
remove the need for scala suffixes on quite a few modules.
However, our Mesos support has unfortunately a hard dependency on Akka,
which naturally does not play well with the goal of isolating Akka in
it's own ClassLoader.
To solve this issue I was thinking of simple dropping flink-mesos in
1.14 (it was deprecated in 1.13).
Truth be told, I picked this option because it is the easiest to do. We
_could_ probably make things work somehow (likely by shipping a second
Akka version just for flink-mesos), but it doesn't seem worth the hassle
and would void some of the benefits. So far we kept flink-mesos around,
despite not really developing it further, because it didn't hurt to have
it in still in Flink, but this has now changed.
Please tell me what you think.
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Etienne Chauchot <ec...@apache.org>.
Hi,
+1 on dropping Mesos support to clear dependencies in the flink-runtime
module:
- I tend to think as Seth and Israel and see a bit of a decline of the
use of Mesos: as an example of that, AFAICT, Amazon EMR and Amazon EKS
provide Yarn and Kubernates support but no Mesos. I did not know about
most of the Mesos PMC leaving that Fabian mentioned but it is once again
a sign IMHO.
- the survey among users seems to have concluded that the removal is
possible.
Best,
Etienne
On 23/06/2021 14:29, Stephan Ewen wrote:
> I would prefer to remove Mesos from the Flink core as well.
>
> I also had a similar thought as Seth: As far as I know, you can package
> applications to run on Mesos with "Marathon". That would be like deploying
> an opaque Flink standalone cluster on Mesos
> The implication is similar to going from an active integration to a
> standalone cluster (like from native Flink Kubernetes Application
> Deployment to a Standalone Application Deployment on Kubernetes): You need
> to make sure the number of TMs / slots and the parallelism fit together (or
> use the new reactive mode). Other than that, I think it should work well
> for streaming jobs.
>
> Having a Flink-Marathon template in https://flink-packages.org/ would be a
> nice thing for Mesos users.
>
> @Oleksandr What do you think about that?
>
> On Wed, Jun 23, 2021 at 11:31 AM Leonard Xu <xb...@gmail.com> wrote:
>
>> + 1 for dropping Mesos. I checked both commit history and mail list, the
>> Mesos related issue/user question has been rarely appeared.
>>
>> Best,
>> Leonard
>>
>>
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Stephan Ewen <se...@apache.org>.
I would prefer to remove Mesos from the Flink core as well.
I also had a similar thought as Seth: As far as I know, you can package
applications to run on Mesos with "Marathon". That would be like deploying
an opaque Flink standalone cluster on Mesos
The implication is similar to going from an active integration to a
standalone cluster (like from native Flink Kubernetes Application
Deployment to a Standalone Application Deployment on Kubernetes): You need
to make sure the number of TMs / slots and the parallelism fit together (or
use the new reactive mode). Other than that, I think it should work well
for streaming jobs.
Having a Flink-Marathon template in https://flink-packages.org/ would be a
nice thing for Mesos users.
@Oleksandr What do you think about that?
On Wed, Jun 23, 2021 at 11:31 AM Leonard Xu <xb...@gmail.com> wrote:
> + 1 for dropping Mesos. I checked both commit history and mail list, the
> Mesos related issue/user question has been rarely appeared.
>
> Best,
> Leonard
>
>
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Leonard Xu <xb...@gmail.com>.
+ 1 for dropping Mesos. I checked both commit history and mail list, the Mesos related issue/user question has been rarely appeared.
Best,
Leonard
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Fabian Paul <fa...@data-artisans.com>.
+ 1 for dropping mesos. Most of the PMCs have already left the project [1] and a move to attic
was barely avoided. Overall kubernetes has taken its place and it is unlikely that we will see a
surge in Mesos very soon.
Best,
Fabian
[1] https://lists.apache.org/thread.html/rab2a820507f7c846e54a847398ab20f47698ec5bce0c8e182bfe51ba%40%3Cdev.mesos.apache.org%3E
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Yang Wang <da...@gmail.com>.
+1 for dropping mesos support.
AFAIK, mesos(including marathon for the container management) is phasing
out gradually and has been replaced with Kubernetes in the containerized
world.
Best,
Yang
Matthias Pohl <ma...@ververica.com> 于2021年6月23日周三 下午2:04写道：
> +1 for dropping Mesos support. There was no feedback opposing the direction
> from the community in the most-recent discussion [1,2] on deprecating it.
>
> Matthias
>
> [1]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-td45974.html
> [2]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Deprecating-Mesos-support-td50142.html
>
> On Wed, Jun 23, 2021 at 4:21 AM Yangze Guo <ka...@gmail.com> wrote:
>
> > +1 for dropping if there is no strong demand from the community.
> >
> > I'm willing to help with the removal of e2e tests part.
> >
> > Best,
> > Yangze Guo
> >
> > On Wed, Jun 23, 2021 at 10:09 AM Xintong Song <to...@gmail.com>
> > wrote:
> > >
> > > +1 for dropping.
> > >
> > > I like Seth's idea. I don't have any real Mesos experience either.
> > > According to this article [1], it looks like we can deploy a standalone
> > > cluster on Mesos similar to Kubernetes. However, we should only do it
> if
> > > there's indeed a strong demand from the community for deploying a
> > > latest version of Flink on Mesos.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > > [1] https://www.baeldung.com/ops/mesos-kubernetes-comparison
> > >
> > > On Tue, Jun 22, 2021 at 11:59 PM Israel Ekpo <is...@gmail.com>
> > wrote:
> > >
> > > > I am in favor of dropping the support for Mesos.
> > > >
> > > > In terms of the landscape for users leveraging Mesos for the kind of
> > > > workloads Flink is used, I think it is on the decline.
> > > >
> > > > +1 from me
> > > >
> > > > On Tue, Jun 22, 2021 at 11:32 AM Seth Wiesman <sj...@gmail.com>
> > wrote:
> > > >
> > > > > Sorry if this is a naive question, I don't have any real Mesos
> > > > experience.
> > > > > Is it possible to deploy a standalone cluster on top of Mesos in
> the
> > same
> > > > > way you can with Kubernetes? If so, and there is still Mesos demand
> > from
> > > > > the community, we could document that process as the recommended
> > > > deployment
> > > > > mode going forward.
> > > > >
> > > > > Seth
> > > > >
> > > > > On Tue, Jun 22, 2021 at 5:02 AM Arvid Heise <ar...@apache.org>
> > wrote:
> > > > >
> > > > > > +1 for dropping. Frankly speaking, I don't see it having any
> future
> > > > (and
> > > > > > D2iQ
> > > > > > agrees).
> > > > > >
> > > > > > If there is a surprisingly huge demand, I'd try to evaluate
> > plugins for
> > > > > it.
> > > > > >
> > > > > > On Tue, Jun 22, 2021 at 11:46 AM Till Rohrmann <
> > trohrmann@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > I'd be ok with dropping support for Mesos if it helps us to
> > clear our
> > > > > > > dependencies in the flink-runtime module. If we do it, then we
> > should
> > > > > > > probably update our documentation with a pointer to the latest
> > Flink
> > > > > > > version that supports Mesos in case of users strictly need
> Mesos.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Till
> > > > > > >
> > > > > > > On Tue, Jun 22, 2021 at 10:29 AM Chesnay Schepler <
> > > > chesnay@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Last week I spent some time looking into making flink-runtime
> > scala
> > > > > > > > free, which effectively means to move the Akka-reliant
> classes
> > to
> > > > > > > > another module, and load that module along with Akka and all
> of
> > > > it's
> > > > > > > > dependencies (including Scala) through a separate
> classloader.
> > > > > > > >
> > > > > > > > This would finally decouple the Scala versions required by
> the
> > > > > runtime
> > > > > > > > and API, and would allow us to upgrade Akka as we'd no longer
> > be
> > > > > > limited
> > > > > > > > to Scala 2.11. It would rid the classpath of a few
> > dependencies,
> > > > and
> > > > > > > > remove the need for scala suffixes on quite a few modules.
> > > > > > > >
> > > > > > > > However, our Mesos support has unfortunately a hard
> dependency
> > on
> > > > > Akka,
> > > > > > > > which naturally does not play well with the goal of isolating
> > Akka
> > > > in
> > > > > > > > it's own ClassLoader.
> > > > > > > >
> > > > > > > > To solve this issue I was thinking of simple dropping
> > flink-mesos
> > > > in
> > > > > > > > 1.14 (it was deprecated in 1.13).
> > > > > > > >
> > > > > > > > Truth be told, I picked this option because it is the easiest
> > to
> > > > do.
> > > > > We
> > > > > > > > _could_ probably make things work somehow (likely by
> shipping a
> > > > > second
> > > > > > > > Akka version just for flink-mesos), but it doesn't seem worth
> > the
> > > > > > hassle
> > > > > > > > and would void some of the benefits. So far we kept
> flink-mesos
> > > > > around,
> > > > > > > > despite not really developing it further, because it didn't
> > hurt to
> > > > > > have
> > > > > > > > it in still in Flink, but this has now changed.
> > > > > > > >
> > > > > > > > Please tell me what you think.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
>
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Matthias Pohl <ma...@ververica.com>.
+1 for dropping Mesos support. There was no feedback opposing the direction
from the community in the most-recent discussion [1,2] on deprecating it.
Matthias
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-td45974.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Deprecating-Mesos-support-td50142.html
On Wed, Jun 23, 2021 at 4:21 AM Yangze Guo <ka...@gmail.com> wrote:
> +1 for dropping if there is no strong demand from the community.
>
> I'm willing to help with the removal of e2e tests part.
>
> Best,
> Yangze Guo
>
> On Wed, Jun 23, 2021 at 10:09 AM Xintong Song <to...@gmail.com>
> wrote:
> >
> > +1 for dropping.
> >
> > I like Seth's idea. I don't have any real Mesos experience either.
> > According to this article [1], it looks like we can deploy a standalone
> > cluster on Mesos similar to Kubernetes. However, we should only do it if
> > there's indeed a strong demand from the community for deploying a
> > latest version of Flink on Mesos.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1] https://www.baeldung.com/ops/mesos-kubernetes-comparison
> >
> > On Tue, Jun 22, 2021 at 11:59 PM Israel Ekpo <is...@gmail.com>
> wrote:
> >
> > > I am in favor of dropping the support for Mesos.
> > >
> > > In terms of the landscape for users leveraging Mesos for the kind of
> > > workloads Flink is used, I think it is on the decline.
> > >
> > > +1 from me
> > >
> > > On Tue, Jun 22, 2021 at 11:32 AM Seth Wiesman <sj...@gmail.com>
> wrote:
> > >
> > > > Sorry if this is a naive question, I don't have any real Mesos
> > > experience.
> > > > Is it possible to deploy a standalone cluster on top of Mesos in the
> same
> > > > way you can with Kubernetes? If so, and there is still Mesos demand
> from
> > > > the community, we could document that process as the recommended
> > > deployment
> > > > mode going forward.
> > > >
> > > > Seth
> > > >
> > > > On Tue, Jun 22, 2021 at 5:02 AM Arvid Heise <ar...@apache.org>
> wrote:
> > > >
> > > > > +1 for dropping. Frankly speaking, I don't see it having any future
> > > (and
> > > > > D2iQ
> > > > > agrees).
> > > > >
> > > > > If there is a surprisingly huge demand, I'd try to evaluate
> plugins for
> > > > it.
> > > > >
> > > > > On Tue, Jun 22, 2021 at 11:46 AM Till Rohrmann <
> trohrmann@apache.org>
> > > > > wrote:
> > > > >
> > > > > > I'd be ok with dropping support for Mesos if it helps us to
> clear our
> > > > > > dependencies in the flink-runtime module. If we do it, then we
> should
> > > > > > probably update our documentation with a pointer to the latest
> Flink
> > > > > > version that supports Mesos in case of users strictly need Mesos.
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Tue, Jun 22, 2021 at 10:29 AM Chesnay Schepler <
> > > chesnay@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Last week I spent some time looking into making flink-runtime
> scala
> > > > > > > free, which effectively means to move the Akka-reliant classes
> to
> > > > > > > another module, and load that module along with Akka and all of
> > > it's
> > > > > > > dependencies (including Scala) through a separate classloader.
> > > > > > >
> > > > > > > This would finally decouple the Scala versions required by the
> > > > runtime
> > > > > > > and API, and would allow us to upgrade Akka as we'd no longer
> be
> > > > > limited
> > > > > > > to Scala 2.11. It would rid the classpath of a few
> dependencies,
> > > and
> > > > > > > remove the need for scala suffixes on quite a few modules.
> > > > > > >
> > > > > > > However, our Mesos support has unfortunately a hard dependency
> on
> > > > Akka,
> > > > > > > which naturally does not play well with the goal of isolating
> Akka
> > > in
> > > > > > > it's own ClassLoader.
> > > > > > >
> > > > > > > To solve this issue I was thinking of simple dropping
> flink-mesos
> > > in
> > > > > > > 1.14 (it was deprecated in 1.13).
> > > > > > >
> > > > > > > Truth be told, I picked this option because it is the easiest
> to
> > > do.
> > > > We
> > > > > > > _could_ probably make things work somehow (likely by shipping a
> > > > second
> > > > > > > Akka version just for flink-mesos), but it doesn't seem worth
> the
> > > > > hassle
> > > > > > > and would void some of the benefits. So far we kept flink-mesos
> > > > around,
> > > > > > > despite not really developing it further, because it didn't
> hurt to
> > > > > have
> > > > > > > it in still in Flink, but this has now changed.
> > > > > > >
> > > > > > > Please tell me what you think.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Yangze Guo <ka...@gmail.com>.
+1 for dropping if there is no strong demand from the community.
I'm willing to help with the removal of e2e tests part.
Best,
Yangze Guo
On Wed, Jun 23, 2021 at 10:09 AM Xintong Song <to...@gmail.com> wrote:
>
> +1 for dropping.
>
> I like Seth's idea. I don't have any real Mesos experience either.
> According to this article [1], it looks like we can deploy a standalone
> cluster on Mesos similar to Kubernetes. However, we should only do it if
> there's indeed a strong demand from the community for deploying a
> latest version of Flink on Mesos.
>
> Thank you~
>
> Xintong Song
>
>
> [1] https://www.baeldung.com/ops/mesos-kubernetes-comparison
>
> On Tue, Jun 22, 2021 at 11:59 PM Israel Ekpo <is...@gmail.com> wrote:
>
> > I am in favor of dropping the support for Mesos.
> >
> > In terms of the landscape for users leveraging Mesos for the kind of
> > workloads Flink is used, I think it is on the decline.
> >
> > +1 from me
> >
> > On Tue, Jun 22, 2021 at 11:32 AM Seth Wiesman <sj...@gmail.com> wrote:
> >
> > > Sorry if this is a naive question, I don't have any real Mesos
> > experience.
> > > Is it possible to deploy a standalone cluster on top of Mesos in the same
> > > way you can with Kubernetes? If so, and there is still Mesos demand from
> > > the community, we could document that process as the recommended
> > deployment
> > > mode going forward.
> > >
> > > Seth
> > >
> > > On Tue, Jun 22, 2021 at 5:02 AM Arvid Heise <ar...@apache.org> wrote:
> > >
> > > > +1 for dropping. Frankly speaking, I don't see it having any future
> > (and
> > > > D2iQ
> > > > agrees).
> > > >
> > > > If there is a surprisingly huge demand, I'd try to evaluate plugins for
> > > it.
> > > >
> > > > On Tue, Jun 22, 2021 at 11:46 AM Till Rohrmann <tr...@apache.org>
> > > > wrote:
> > > >
> > > > > I'd be ok with dropping support for Mesos if it helps us to clear our
> > > > > dependencies in the flink-runtime module. If we do it, then we should
> > > > > probably update our documentation with a pointer to the latest Flink
> > > > > version that supports Mesos in case of users strictly need Mesos.
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Tue, Jun 22, 2021 at 10:29 AM Chesnay Schepler <
> > chesnay@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Last week I spent some time looking into making flink-runtime scala
> > > > > > free, which effectively means to move the Akka-reliant classes to
> > > > > > another module, and load that module along with Akka and all of
> > it's
> > > > > > dependencies (including Scala) through a separate classloader.
> > > > > >
> > > > > > This would finally decouple the Scala versions required by the
> > > runtime
> > > > > > and API, and would allow us to upgrade Akka as we'd no longer be
> > > > limited
> > > > > > to Scala 2.11. It would rid the classpath of a few dependencies,
> > and
> > > > > > remove the need for scala suffixes on quite a few modules.
> > > > > >
> > > > > > However, our Mesos support has unfortunately a hard dependency on
> > > Akka,
> > > > > > which naturally does not play well with the goal of isolating Akka
> > in
> > > > > > it's own ClassLoader.
> > > > > >
> > > > > > To solve this issue I was thinking of simple dropping flink-mesos
> > in
> > > > > > 1.14 (it was deprecated in 1.13).
> > > > > >
> > > > > > Truth be told, I picked this option because it is the easiest to
> > do.
> > > We
> > > > > > _could_ probably make things work somehow (likely by shipping a
> > > second
> > > > > > Akka version just for flink-mesos), but it doesn't seem worth the
> > > > hassle
> > > > > > and would void some of the benefits. So far we kept flink-mesos
> > > around,
> > > > > > despite not really developing it further, because it didn't hurt to
> > > > have
> > > > > > it in still in Flink, but this has now changed.
> > > > > >
> > > > > > Please tell me what you think.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Xintong Song <to...@gmail.com>.
+1 for dropping.
I like Seth's idea. I don't have any real Mesos experience either.
According to this article [1], it looks like we can deploy a standalone
cluster on Mesos similar to Kubernetes. However, we should only do it if
there's indeed a strong demand from the community for deploying a
latest version of Flink on Mesos.
Thank you~
Xintong Song
[1] https://www.baeldung.com/ops/mesos-kubernetes-comparison
On Tue, Jun 22, 2021 at 11:59 PM Israel Ekpo <is...@gmail.com> wrote:
> I am in favor of dropping the support for Mesos.
>
> In terms of the landscape for users leveraging Mesos for the kind of
> workloads Flink is used, I think it is on the decline.
>
> +1 from me
>
> On Tue, Jun 22, 2021 at 11:32 AM Seth Wiesman <sj...@gmail.com> wrote:
>
> > Sorry if this is a naive question, I don't have any real Mesos
> experience.
> > Is it possible to deploy a standalone cluster on top of Mesos in the same
> > way you can with Kubernetes? If so, and there is still Mesos demand from
> > the community, we could document that process as the recommended
> deployment
> > mode going forward.
> >
> > Seth
> >
> > On Tue, Jun 22, 2021 at 5:02 AM Arvid Heise <ar...@apache.org> wrote:
> >
> > > +1 for dropping. Frankly speaking, I don't see it having any future
> (and
> > > D2iQ
> > > agrees).
> > >
> > > If there is a surprisingly huge demand, I'd try to evaluate plugins for
> > it.
> > >
> > > On Tue, Jun 22, 2021 at 11:46 AM Till Rohrmann <tr...@apache.org>
> > > wrote:
> > >
> > > > I'd be ok with dropping support for Mesos if it helps us to clear our
> > > > dependencies in the flink-runtime module. If we do it, then we should
> > > > probably update our documentation with a pointer to the latest Flink
> > > > version that supports Mesos in case of users strictly need Mesos.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Tue, Jun 22, 2021 at 10:29 AM Chesnay Schepler <
> chesnay@apache.org>
> > > > wrote:
> > > >
> > > > > Last week I spent some time looking into making flink-runtime scala
> > > > > free, which effectively means to move the Akka-reliant classes to
> > > > > another module, and load that module along with Akka and all of
> it's
> > > > > dependencies (including Scala) through a separate classloader.
> > > > >
> > > > > This would finally decouple the Scala versions required by the
> > runtime
> > > > > and API, and would allow us to upgrade Akka as we'd no longer be
> > > limited
> > > > > to Scala 2.11. It would rid the classpath of a few dependencies,
> and
> > > > > remove the need for scala suffixes on quite a few modules.
> > > > >
> > > > > However, our Mesos support has unfortunately a hard dependency on
> > Akka,
> > > > > which naturally does not play well with the goal of isolating Akka
> in
> > > > > it's own ClassLoader.
> > > > >
> > > > > To solve this issue I was thinking of simple dropping flink-mesos
> in
> > > > > 1.14 (it was deprecated in 1.13).
> > > > >
> > > > > Truth be told, I picked this option because it is the easiest to
> do.
> > We
> > > > > _could_ probably make things work somehow (likely by shipping a
> > second
> > > > > Akka version just for flink-mesos), but it doesn't seem worth the
> > > hassle
> > > > > and would void some of the benefits. So far we kept flink-mesos
> > around,
> > > > > despite not really developing it further, because it didn't hurt to
> > > have
> > > > > it in still in Flink, but this has now changed.
> > > > >
> > > > > Please tell me what you think.
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Israel Ekpo <is...@gmail.com>.
I am in favor of dropping the support for Mesos.
In terms of the landscape for users leveraging Mesos for the kind of
workloads Flink is used, I think it is on the decline.
+1 from me
On Tue, Jun 22, 2021 at 11:32 AM Seth Wiesman <sj...@gmail.com> wrote:
> Sorry if this is a naive question, I don't have any real Mesos experience.
> Is it possible to deploy a standalone cluster on top of Mesos in the same
> way you can with Kubernetes? If so, and there is still Mesos demand from
> the community, we could document that process as the recommended deployment
> mode going forward.
>
> Seth
>
> On Tue, Jun 22, 2021 at 5:02 AM Arvid Heise <ar...@apache.org> wrote:
>
> > +1 for dropping. Frankly speaking, I don't see it having any future (and
> > D2iQ
> > agrees).
> >
> > If there is a surprisingly huge demand, I'd try to evaluate plugins for
> it.
> >
> > On Tue, Jun 22, 2021 at 11:46 AM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > I'd be ok with dropping support for Mesos if it helps us to clear our
> > > dependencies in the flink-runtime module. If we do it, then we should
> > > probably update our documentation with a pointer to the latest Flink
> > > version that supports Mesos in case of users strictly need Mesos.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Jun 22, 2021 at 10:29 AM Chesnay Schepler <ch...@apache.org>
> > > wrote:
> > >
> > > > Last week I spent some time looking into making flink-runtime scala
> > > > free, which effectively means to move the Akka-reliant classes to
> > > > another module, and load that module along with Akka and all of it's
> > > > dependencies (including Scala) through a separate classloader.
> > > >
> > > > This would finally decouple the Scala versions required by the
> runtime
> > > > and API, and would allow us to upgrade Akka as we'd no longer be
> > limited
> > > > to Scala 2.11. It would rid the classpath of a few dependencies, and
> > > > remove the need for scala suffixes on quite a few modules.
> > > >
> > > > However, our Mesos support has unfortunately a hard dependency on
> Akka,
> > > > which naturally does not play well with the goal of isolating Akka in
> > > > it's own ClassLoader.
> > > >
> > > > To solve this issue I was thinking of simple dropping flink-mesos in
> > > > 1.14 (it was deprecated in 1.13).
> > > >
> > > > Truth be told, I picked this option because it is the easiest to do.
> We
> > > > _could_ probably make things work somehow (likely by shipping a
> second
> > > > Akka version just for flink-mesos), but it doesn't seem worth the
> > hassle
> > > > and would void some of the benefits. So far we kept flink-mesos
> around,
> > > > despite not really developing it further, because it didn't hurt to
> > have
> > > > it in still in Flink, but this has now changed.
> > > >
> > > > Please tell me what you think.
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Seth Wiesman <sj...@gmail.com>.
Sorry if this is a naive question, I don't have any real Mesos experience.
Is it possible to deploy a standalone cluster on top of Mesos in the same
way you can with Kubernetes? If so, and there is still Mesos demand from
the community, we could document that process as the recommended deployment
mode going forward.
Seth
On Tue, Jun 22, 2021 at 5:02 AM Arvid Heise <ar...@apache.org> wrote:
> +1 for dropping. Frankly speaking, I don't see it having any future (and
> D2iQ
> agrees).
>
> If there is a surprisingly huge demand, I'd try to evaluate plugins for it.
>
> On Tue, Jun 22, 2021 at 11:46 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > I'd be ok with dropping support for Mesos if it helps us to clear our
> > dependencies in the flink-runtime module. If we do it, then we should
> > probably update our documentation with a pointer to the latest Flink
> > version that supports Mesos in case of users strictly need Mesos.
> >
> > Cheers,
> > Till
> >
> > On Tue, Jun 22, 2021 at 10:29 AM Chesnay Schepler <ch...@apache.org>
> > wrote:
> >
> > > Last week I spent some time looking into making flink-runtime scala
> > > free, which effectively means to move the Akka-reliant classes to
> > > another module, and load that module along with Akka and all of it's
> > > dependencies (including Scala) through a separate classloader.
> > >
> > > This would finally decouple the Scala versions required by the runtime
> > > and API, and would allow us to upgrade Akka as we'd no longer be
> limited
> > > to Scala 2.11. It would rid the classpath of a few dependencies, and
> > > remove the need for scala suffixes on quite a few modules.
> > >
> > > However, our Mesos support has unfortunately a hard dependency on Akka,
> > > which naturally does not play well with the goal of isolating Akka in
> > > it's own ClassLoader.
> > >
> > > To solve this issue I was thinking of simple dropping flink-mesos in
> > > 1.14 (it was deprecated in 1.13).
> > >
> > > Truth be told, I picked this option because it is the easiest to do. We
> > > _could_ probably make things work somehow (likely by shipping a second
> > > Akka version just for flink-mesos), but it doesn't seem worth the
> hassle
> > > and would void some of the benefits. So far we kept flink-mesos around,
> > > despite not really developing it further, because it didn't hurt to
> have
> > > it in still in Flink, but this has now changed.
> > >
> > > Please tell me what you think.
> > >
> > >
> >
>
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Arvid Heise <ar...@apache.org>.
+1 for dropping. Frankly speaking, I don't see it having any future (and D2iQ
agrees).
If there is a surprisingly huge demand, I'd try to evaluate plugins for it.
On Tue, Jun 22, 2021 at 11:46 AM Till Rohrmann <tr...@apache.org> wrote:
> I'd be ok with dropping support for Mesos if it helps us to clear our
> dependencies in the flink-runtime module. If we do it, then we should
> probably update our documentation with a pointer to the latest Flink
> version that supports Mesos in case of users strictly need Mesos.
>
> Cheers,
> Till
>
> On Tue, Jun 22, 2021 at 10:29 AM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > Last week I spent some time looking into making flink-runtime scala
> > free, which effectively means to move the Akka-reliant classes to
> > another module, and load that module along with Akka and all of it's
> > dependencies (including Scala) through a separate classloader.
> >
> > This would finally decouple the Scala versions required by the runtime
> > and API, and would allow us to upgrade Akka as we'd no longer be limited
> > to Scala 2.11. It would rid the classpath of a few dependencies, and
> > remove the need for scala suffixes on quite a few modules.
> >
> > However, our Mesos support has unfortunately a hard dependency on Akka,
> > which naturally does not play well with the goal of isolating Akka in
> > it's own ClassLoader.
> >
> > To solve this issue I was thinking of simple dropping flink-mesos in
> > 1.14 (it was deprecated in 1.13).
> >
> > Truth be told, I picked this option because it is the easiest to do. We
> > _could_ probably make things work somehow (likely by shipping a second
> > Akka version just for flink-mesos), but it doesn't seem worth the hassle
> > and would void some of the benefits. So far we kept flink-mesos around,
> > despite not really developing it further, because it didn't hurt to have
> > it in still in Flink, but this has now changed.
> >
> > Please tell me what you think.
> >
> >
>
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Till Rohrmann <tr...@apache.org>.
I'd be ok with dropping support for Mesos if it helps us to clear our
dependencies in the flink-runtime module. If we do it, then we should
probably update our documentation with a pointer to the latest Flink
version that supports Mesos in case of users strictly need Mesos.
Cheers,
Till
On Tue, Jun 22, 2021 at 10:29 AM Chesnay Schepler <ch...@apache.org>
wrote:
> Last week I spent some time looking into making flink-runtime scala
> free, which effectively means to move the Akka-reliant classes to
> another module, and load that module along with Akka and all of it's
> dependencies (including Scala) through a separate classloader.
>
> This would finally decouple the Scala versions required by the runtime
> and API, and would allow us to upgrade Akka as we'd no longer be limited
> to Scala 2.11. It would rid the classpath of a few dependencies, and
> remove the need for scala suffixes on quite a few modules.
>
> However, our Mesos support has unfortunately a hard dependency on Akka,
> which naturally does not play well with the goal of isolating Akka in
> it's own ClassLoader.
>
> To solve this issue I was thinking of simple dropping flink-mesos in
> 1.14 (it was deprecated in 1.13).
>
> Truth be told, I picked this option because it is the easiest to do. We
> _could_ probably make things work somehow (likely by shipping a second
> Akka version just for flink-mesos), but it doesn't seem worth the hassle
> and would void some of the benefits. So far we kept flink-mesos around,
> despite not really developing it further, because it didn't hurt to have
> it in still in Flink, but this has now changed.
>
> Please tell me what you think.
>
>
Re: [DISCUSS] Drop Mesos in 1.14
Posted by Robert Metzger <rm...@apache.org>.
I'm adding Oleksandr from Criteo to this thread, they've mentioned in the
past that they are using Flink on Mesos (IIRC the only company that seemed
to be doing so)
I'm personally fine with dropping Mesos support.
On Tue, Jun 22, 2021 at 10:29 AM Chesnay Schepler <ch...@apache.org>
wrote:
> Last week I spent some time looking into making flink-runtime scala
> free, which effectively means to move the Akka-reliant classes to
> another module, and load that module along with Akka and all of it's
> dependencies (including Scala) through a separate classloader.
>
> This would finally decouple the Scala versions required by the runtime
> and API, and would allow us to upgrade Akka as we'd no longer be limited
> to Scala 2.11. It would rid the classpath of a few dependencies, and
> remove the need for scala suffixes on quite a few modules.
>
> However, our Mesos support has unfortunately a hard dependency on Akka,
> which naturally does not play well with the goal of isolating Akka in
> it's own ClassLoader.
>
> To solve this issue I was thinking of simple dropping flink-mesos in
> 1.14 (it was deprecated in 1.13).
>
> Truth be told, I picked this option because it is the easiest to do. We
> _could_ probably make things work somehow (likely by shipping a second
> Akka version just for flink-mesos), but it doesn't seem worth the hassle
> and would void some of the benefits. So far we kept flink-mesos around,
> despite not really developing it further, because it didn't hurt to have
> it in still in Flink, but this has now changed.
>
> Please tell me what you think.
>
>