You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Klaus Ma <kl...@gmail.com> on 2019/11/04 09:04:10 UTC
Build customized resource manager
Hi team,
AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did
some enhancement to them, e.g. integrate with Volcano
<http://github.com/volcano-sh/volcano> in k8s. Is that possible to do that
without fork the whole spark project? For example, enable customized
resource manager with configuration, e.g. replace
`org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with
`MyK8SClient`, so I can only maintain the resource manager instead of the
whole project.
-- Klaus
Re: Build customized resource manager
Posted by Klaus Ma <kl...@gmail.com>.
hm.... that'll be better to me if we can build customized resource manager
out of core; otherwise, we have to go through the long discussion in the
community :)
But if we support that, why still mesos/yarn/k8s resource manager there in
the tree?
On Fri, Nov 8, 2019 at 10:18 PM Tom Graves <tg...@yahoo.com> wrote:
> I don't know if it all works but some work was done to make cluster
> manager pluggable, see SPARK-13904.
>
> Tom
>
> On Wednesday, November 6, 2019, 07:22:59 PM CST, Klaus Ma <
> klaus1982.cn@gmail.com> wrote:
>
>
> Any suggestions?
>
> - Klaus
>
> On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma <kl...@gmail.com> wrote:
>
> Hi team,
>
> AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did
> some enhancement to them, e.g. integrate with Volcano
> <http://github.com/volcano-sh/volcano> in k8s. Is that possible to do
> that without fork the whole spark project? For example, enable customized
> resource manager with configuration, e.g. replace
> `org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with
> `MyK8SClient`, so I can only maintain the resource manager instead of the
> whole project.
>
> -- Klaus
>
>
Re: Build customized resource manager
Posted by Klaus Ma <kl...@gmail.com>.
hm.... that'll be better to me if we can build customized resource manager
out of core; otherwise, we have to go through the long discussion in the
community :)
But if we support that, why still mesos/yarn/k8s resource manager there in
the tree?
On Fri, Nov 8, 2019 at 10:18 PM Tom Graves <tg...@yahoo.com> wrote:
> I don't know if it all works but some work was done to make cluster
> manager pluggable, see SPARK-13904.
>
> Tom
>
> On Wednesday, November 6, 2019, 07:22:59 PM CST, Klaus Ma <
> klaus1982.cn@gmail.com> wrote:
>
>
> Any suggestions?
>
> - Klaus
>
> On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma <kl...@gmail.com> wrote:
>
> Hi team,
>
> AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did
> some enhancement to them, e.g. integrate with Volcano
> <http://github.com/volcano-sh/volcano> in k8s. Is that possible to do
> that without fork the whole spark project? For example, enable customized
> resource manager with configuration, e.g. replace
> `org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with
> `MyK8SClient`, so I can only maintain the resource manager instead of the
> whole project.
>
> -- Klaus
>
>
Re: Build customized resource manager
Posted by Tom Graves <tg...@yahoo.com.INVALID>.
I don't know if it all works but some work was done to make cluster manager pluggable, see SPARK-13904.
Tom
On Wednesday, November 6, 2019, 07:22:59 PM CST, Klaus Ma <kl...@gmail.com> wrote:
Any suggestions?
- Klaus
On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma <kl...@gmail.com> wrote:
Hi team,
AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did some enhancement to them, e.g. integrate with Volcano in k8s. Is that possible to do that without fork the whole spark project? For example, enable customized resource manager with configuration, e.g. replace `org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with `MyK8SClient`, so I can only maintain the resource manager instead of the whole project.
-- Klaus
Re: Build customized resource manager
Posted by Tom Graves <tg...@yahoo.com.INVALID>.
I don't know if it all works but some work was done to make cluster manager pluggable, see SPARK-13904.
Tom
On Wednesday, November 6, 2019, 07:22:59 PM CST, Klaus Ma <kl...@gmail.com> wrote:
Any suggestions?
- Klaus
On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma <kl...@gmail.com> wrote:
Hi team,
AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did some enhancement to them, e.g. integrate with Volcano in k8s. Is that possible to do that without fork the whole spark project? For example, enable customized resource manager with configuration, e.g. replace `org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with `MyK8SClient`, so I can only maintain the resource manager instead of the whole project.
-- Klaus
Re: Build customized resource manager
Posted by Klaus Ma <kl...@gmail.com>.
Any suggestions?
- Klaus
On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma <kl...@gmail.com> wrote:
> Hi team,
>
> AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did
> some enhancement to them, e.g. integrate with Volcano
> <http://github.com/volcano-sh/volcano> in k8s. Is that possible to do
> that without fork the whole spark project? For example, enable customized
> resource manager with configuration, e.g. replace
> `org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with
> `MyK8SClient`, so I can only maintain the resource manager instead of the
> whole project.
>
> -- Klaus
>
Re: Build customized resource manager
Posted by Klaus Ma <kl...@gmail.com>.
Any suggestions?
- Klaus
On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma <kl...@gmail.com> wrote:
> Hi team,
>
> AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did
> some enhancement to them, e.g. integrate with Volcano
> <http://github.com/volcano-sh/volcano> in k8s. Is that possible to do
> that without fork the whole spark project? For example, enable customized
> resource manager with configuration, e.g. replace
> `org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with
> `MyK8SClient`, so I can only maintain the resource manager instead of the
> whole project.
>
> -- Klaus
>