You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Marcelo Vanzin <va...@cloudera.com> on 2018/01/10 01:22:09 UTC
Kubernetes: why use init containers?
Hello,
Me again. I was playing some more with the kubernetes backend and the
whole init container thing seemed unnecessary to me.
Currently it's used to download remote jars and files, mount the
volume into the driver / executor, and place those jars in the
classpath / move the files to the working directory. This is all stuff
that spark-submit already does without needing extra help.
So I spent some time hacking stuff and removing the init container
code, and launching the driver inside kubernetes using spark-submit
(similar to how standalone and mesos cluster mode works):
https://github.com/vanzin/spark/commit/k8s-no-init
I'd like to point out the output of "git show --stat" for that diff:
29 files changed, 130 insertions(+), 1560 deletions(-)
You get massive code reuse by simply using spark-submit. The remote
dependencies are downloaded in the driver, and the driver does the job
of service them to executors.
So I guess my question is: is there any advantage in using an init container?
The current init container code can download stuff in parallel, but
that's an easy improvement to make in spark-submit and that would
benefit everybody. You can argue that executors downloading from
external servers would be faster than downloading from the driver, but
I'm not sure I'd agree - it can go both ways.
Also the same idea could probably be applied to starting executors;
Mesos starts executors using "spark-class" already, so doing that
would both improve code sharing and potentially simplify some code in
the k8s backend.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Andrew Ash <an...@andrewash.com>.
It seems we have two standard practices for resource distribution in place
here:
- the Spark way is that the application (Spark) distributes the resources
*during* app execution, and does this by exposing files/jars on an http
server on the driver (or pre-staged elsewhere), and executors downloading
from that location (driver or remote)
- the Kubernetes way is that the cluster manager (Kubernetes) distributes
the resources *before* app execution, and does this primarily via docker
images, and secondarily through init containers for non-image resources.
I'd imagine a motivation for this choice in k8s' part is immutability of
the application at runtime
When the Spark and K8s standard practices are in conflict (as they seem to
be here), which convention should be followed?
Looking at the Spark-on-YARN integration, there's almost a parallel here
with spark.yarn.archive, where that configures the cluster (YARN) to do
distribution pre-runtime instead of the application mid-runtime.
Based purely on the lines-of-code removal, right now I lean towards
eliminating init containers. It doesn't seem like credential segregation
between init container and main pod container is that valuable right now,
and the retryability could/should be in all of Spark's cluster managers,
not just k8s.
So I support Anirudh's suggestion to move towards bringing the change
demonstrated in Marcelo's POC into master.
On Wed, Jan 10, 2018 at 3:00 PM, Anirudh Ramanathan <
ramanathana@google.com.invalid> wrote:
> Thanks for this discussion everyone. It has been very useful in getting an
> overall understanding here.
> I think in general, consensus is that this change doesn't introduce
> behavioral changes, and it's definitely an advantage to reuse the
> constructs that Spark provides to us.
>
> Moving on to a different question here - of pushing this through to Spark.
> The init containers have been tested over the past two Spark releases by
> external users and integration testing - and this would be a fundamental
> change to that behavior.
> We should work on getting enough test coverage and confidence here.
>
> We can start by getting a PR going perhaps, and start augmenting the
> integration testing to ensure that there are no surprises - with/without
> credentials, accessing GCS, S3 etc as well.
> When we get enough confidence and test coverage, let's merge this in.
> Does that sound like a reasonable path forward?
>
>
>
> On Wed, Jan 10, 2018 at 2:53 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
>> On Wed, Jan 10, 2018 at 2:51 PM, Matt Cheah <mc...@palantir.com> wrote:
>> > those sidecars may perform side effects that are undesirable if the
>> main Spark application failed because dependencies weren’t available
>>
>> If the contract is that the Spark driver pod does not have an init
>> container, and the driver handles its own dependencies, then by
>> definition that situation cannot exist.
>>
>> --
>> Marcelo
>>
>
>
>
> --
> Anirudh Ramanathan
>
Re: Kubernetes: why use init containers?
Posted by Andrew Ash <an...@andrewash.com>.
+1 on the first release being marked experimental. Many major features
coming into Spark in the past have gone through a stabilization process
On Fri, Jan 12, 2018 at 1:18 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> BTW I most probably will not have time to get back to this at any time
> soon, so if anyone is interested in doing some clean up, I'll leave my
> branch up.
>
> I'm seriously thinking about proposing that we document the k8s
> backend as experimental in 2.3; it seems there still a lot to be
> cleaned up in terms of user interface (as in extensibility and
> customizability), documentation, and mainly testing, and we're pretty
> far into the 2.3 cycle for all of those to be sorted out.
>
> On Thu, Jan 11, 2018 at 8:19 AM, Anirudh Ramanathan
> <ra...@google.com> wrote:
> > If we can separate concerns those out, that might make sense in the short
> > term IMO.
> > There are several benefits to reusing spark-submit and spark-class as you
> > pointed out previously,
> > so, we should be looking to leverage those irrespective of how we do
> > dependency management -
> > in the interest of conformance with the other cluster managers.
> >
> > I like the idea of passing arguments through in a way that it doesn't
> > trigger the dependency management code for now.
> > In the interest of time for 2.3, if we could target the just that (and
> > revisit the init containers afterwards),
> > there should be enough time to make the change, test and release with
> > confidence.
> >
> > On Wed, Jan 10, 2018 at 3:45 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
> >>
> >> On Wed, Jan 10, 2018 at 3:00 PM, Anirudh Ramanathan
> >> <ra...@google.com> wrote:
> >> > We can start by getting a PR going perhaps, and start augmenting the
> >> > integration testing to ensure that there are no surprises -
> with/without
> >> > credentials, accessing GCS, S3 etc as well.
> >> > When we get enough confidence and test coverage, let's merge this in.
> >> > Does that sound like a reasonable path forward?
> >>
> >> I think it's beneficial to separate this into two separate things as
> >> far as discussion goes:
> >>
> >> - using spark-submit: the code should definitely be starting the
> >> driver using spark-submit, and potentially the executor using
> >> spark-class.
> >>
> >> - separately, we can decide on whether to keep or remove init
> containers.
> >>
> >> Unfortunately, code-wise, those are not separate. If you get rid of
> >> init containers, my current p.o.c. has most of the needed changes
> >> (only lightly tested).
> >>
> >> But if you keep init containers, you'll need to mess with the
> >> configuration so that spark-submit never sees spark.jars /
> >> spark.files, so it doesn't trigger its dependency download code. (YARN
> >> does something similar, btw.) That will surely mean different changes
> >> in the current k8s code (which I wanted to double check anyway because
> >> I remember seeing some oddities related to those configs in the logs).
> >>
> >> To comment on one point made by Andrew:
> >> > there's almost a parallel here with spark.yarn.archive, where that
> >> > configures the cluster (YARN) to do distribution pre-runtime
> >>
> >> That's more of a parallel to the docker image; spark.yarn.archive
> >> points to a jar file with Spark jars in it so that YARN can make Spark
> >> available to the driver / executors running in the cluster.
> >>
> >> Like the docker image, you could include other stuff that is not
> >> really part of standard Spark in that archive too, or even not have
> >> Spark at all there, if you want things to just fail. :-)
> >>
> >> --
> >> Marcelo
> >
> >
> >
> >
> > --
> > Anirudh Ramanathan
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: Kubernetes: why use init containers?
Posted by Anirudh Ramanathan <ra...@google.com.INVALID>.
That's fair - I guess it would be a stretch to assume users wouldn't put
custom logic in their init containers if that hook is provided to them. :)
Experimental sounds like a good idea for 2.3. Gives us enough wriggle room
for the next one, and hopefully user feedback in the meantime.
Thanks,
Anirudh
On Jan 12, 2018 2:00 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
> On Fri, Jan 12, 2018 at 1:53 PM, Anirudh Ramanathan
> <ra...@google.com> wrote:
> > As I understand, the bigger change discussed here are like the init
> > containers, which will be more on the implementation side than a user
> facing
> > change/behavioral change - which is why it seemed okay to pursue it post
> 2.3
> > as well.
>
> It's not just a code change.
>
> There are multiple configurations exposed to control the init
> container. There's a whole step - the running of the init container -
> that currently can be customized (even though there is no
> documentation on how to safely do that). If you ship that as "stable",
> you cannot later change it in way that will break applications. So
> you'd not only be stuck with the existence of the init container, but
> with all its current behavior.
>
> Marking as experimental gives us time to stabilize these details. Not
> just whether the init container exists, but what is its actual
> behavior and how the use can affect it. A lot of the replies here
> always mention that init containers can be customized, but I just want
> to point out again that there is currently zero documentation about
> how to do that and not break the assumptions the spark-on-k8s
> submission code makes.
>
> The same applies to the other images, by the way.
>
> --
> Marcelo
>
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Fri, Jan 12, 2018 at 1:53 PM, Anirudh Ramanathan
<ra...@google.com> wrote:
> As I understand, the bigger change discussed here are like the init
> containers, which will be more on the implementation side than a user facing
> change/behavioral change - which is why it seemed okay to pursue it post 2.3
> as well.
It's not just a code change.
There are multiple configurations exposed to control the init
container. There's a whole step - the running of the init container -
that currently can be customized (even though there is no
documentation on how to safely do that). If you ship that as "stable",
you cannot later change it in way that will break applications. So
you'd not only be stuck with the existence of the init container, but
with all its current behavior.
Marking as experimental gives us time to stabilize these details. Not
just whether the init container exists, but what is its actual
behavior and how the use can affect it. A lot of the replies here
always mention that init containers can be customized, but I just want
to point out again that there is currently zero documentation about
how to do that and not break the assumptions the spark-on-k8s
submission code makes.
The same applies to the other images, by the way.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Anirudh Ramanathan <ra...@google.com.INVALID>.
I'd like to discuss the criteria here for graduating from experimental
status. (as a fork, we were mentioned in the documentation as
experimental).
As I understand, the bigger change discussed here are like the init
containers, which will be more on the implementation side than a user
facing change/behavioral change - which is why it seemed okay to pursue it
post 2.3 as well.
If the reasoning is mostly lack of confidence in testing or documentation,
we are working on that, but we would love to have more visibility into what
we're missing, so, we can prioritize and augment that, and maybe even get
there by this release - or at least have a clear path ahead to graduating
in the next one.
On Jan 12, 2018 1:18 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
> BTW I most probably will not have time to get back to this at any time
> soon, so if anyone is interested in doing some clean up, I'll leave my
> branch up.
>
> I'm seriously thinking about proposing that we document the k8s
> backend as experimental in 2.3; it seems there still a lot to be
> cleaned up in terms of user interface (as in extensibility and
> customizability), documentation, and mainly testing, and we're pretty
> far into the 2.3 cycle for all of those to be sorted out.
>
> On Thu, Jan 11, 2018 at 8:19 AM, Anirudh Ramanathan
> <ra...@google.com> wrote:
> > If we can separate concerns those out, that might make sense in the short
> > term IMO.
> > There are several benefits to reusing spark-submit and spark-class as you
> > pointed out previously,
> > so, we should be looking to leverage those irrespective of how we do
> > dependency management -
> > in the interest of conformance with the other cluster managers.
> >
> > I like the idea of passing arguments through in a way that it doesn't
> > trigger the dependency management code for now.
> > In the interest of time for 2.3, if we could target the just that (and
> > revisit the init containers afterwards),
> > there should be enough time to make the change, test and release with
> > confidence.
> >
> > On Wed, Jan 10, 2018 at 3:45 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
> >>
> >> On Wed, Jan 10, 2018 at 3:00 PM, Anirudh Ramanathan
> >> <ra...@google.com> wrote:
> >> > We can start by getting a PR going perhaps, and start augmenting the
> >> > integration testing to ensure that there are no surprises -
> with/without
> >> > credentials, accessing GCS, S3 etc as well.
> >> > When we get enough confidence and test coverage, let's merge this in.
> >> > Does that sound like a reasonable path forward?
> >>
> >> I think it's beneficial to separate this into two separate things as
> >> far as discussion goes:
> >>
> >> - using spark-submit: the code should definitely be starting the
> >> driver using spark-submit, and potentially the executor using
> >> spark-class.
> >>
> >> - separately, we can decide on whether to keep or remove init
> containers.
> >>
> >> Unfortunately, code-wise, those are not separate. If you get rid of
> >> init containers, my current p.o.c. has most of the needed changes
> >> (only lightly tested).
> >>
> >> But if you keep init containers, you'll need to mess with the
> >> configuration so that spark-submit never sees spark.jars /
> >> spark.files, so it doesn't trigger its dependency download code. (YARN
> >> does something similar, btw.) That will surely mean different changes
> >> in the current k8s code (which I wanted to double check anyway because
> >> I remember seeing some oddities related to those configs in the logs).
> >>
> >> To comment on one point made by Andrew:
> >> > there's almost a parallel here with spark.yarn.archive, where that
> >> > configures the cluster (YARN) to do distribution pre-runtime
> >>
> >> That's more of a parallel to the docker image; spark.yarn.archive
> >> points to a jar file with Spark jars in it so that YARN can make Spark
> >> available to the driver / executors running in the cluster.
> >>
> >> Like the docker image, you could include other stuff that is not
> >> really part of standard Spark in that archive too, or even not have
> >> Spark at all there, if you want things to just fail. :-)
> >>
> >> --
> >> Marcelo
> >
> >
> >
> >
> > --
> > Anirudh Ramanathan
>
>
>
> --
> Marcelo
>
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
BTW I most probably will not have time to get back to this at any time
soon, so if anyone is interested in doing some clean up, I'll leave my
branch up.
I'm seriously thinking about proposing that we document the k8s
backend as experimental in 2.3; it seems there still a lot to be
cleaned up in terms of user interface (as in extensibility and
customizability), documentation, and mainly testing, and we're pretty
far into the 2.3 cycle for all of those to be sorted out.
On Thu, Jan 11, 2018 at 8:19 AM, Anirudh Ramanathan
<ra...@google.com> wrote:
> If we can separate concerns those out, that might make sense in the short
> term IMO.
> There are several benefits to reusing spark-submit and spark-class as you
> pointed out previously,
> so, we should be looking to leverage those irrespective of how we do
> dependency management -
> in the interest of conformance with the other cluster managers.
>
> I like the idea of passing arguments through in a way that it doesn't
> trigger the dependency management code for now.
> In the interest of time for 2.3, if we could target the just that (and
> revisit the init containers afterwards),
> there should be enough time to make the change, test and release with
> confidence.
>
> On Wed, Jan 10, 2018 at 3:45 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
>>
>> On Wed, Jan 10, 2018 at 3:00 PM, Anirudh Ramanathan
>> <ra...@google.com> wrote:
>> > We can start by getting a PR going perhaps, and start augmenting the
>> > integration testing to ensure that there are no surprises - with/without
>> > credentials, accessing GCS, S3 etc as well.
>> > When we get enough confidence and test coverage, let's merge this in.
>> > Does that sound like a reasonable path forward?
>>
>> I think it's beneficial to separate this into two separate things as
>> far as discussion goes:
>>
>> - using spark-submit: the code should definitely be starting the
>> driver using spark-submit, and potentially the executor using
>> spark-class.
>>
>> - separately, we can decide on whether to keep or remove init containers.
>>
>> Unfortunately, code-wise, those are not separate. If you get rid of
>> init containers, my current p.o.c. has most of the needed changes
>> (only lightly tested).
>>
>> But if you keep init containers, you'll need to mess with the
>> configuration so that spark-submit never sees spark.jars /
>> spark.files, so it doesn't trigger its dependency download code. (YARN
>> does something similar, btw.) That will surely mean different changes
>> in the current k8s code (which I wanted to double check anyway because
>> I remember seeing some oddities related to those configs in the logs).
>>
>> To comment on one point made by Andrew:
>> > there's almost a parallel here with spark.yarn.archive, where that
>> > configures the cluster (YARN) to do distribution pre-runtime
>>
>> That's more of a parallel to the docker image; spark.yarn.archive
>> points to a jar file with Spark jars in it so that YARN can make Spark
>> available to the driver / executors running in the cluster.
>>
>> Like the docker image, you could include other stuff that is not
>> really part of standard Spark in that archive too, or even not have
>> Spark at all there, if you want things to just fail. :-)
>>
>> --
>> Marcelo
>
>
>
>
> --
> Anirudh Ramanathan
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Anirudh Ramanathan <ra...@google.com.INVALID>.
If we can separate concerns those out, that might make sense in the short
term IMO.
There are several benefits to reusing spark-submit and spark-class as you
pointed out previously,
so, we should be looking to leverage those irrespective of how we do
dependency management -
in the interest of conformance with the other cluster managers.
I like the idea of passing arguments through in a way that it doesn't
trigger the dependency management code for now.
In the interest of time for 2.3, if we could target the just that (and
revisit the init containers afterwards),
there should be enough time to make the change, test and release with
confidence.
On Wed, Jan 10, 2018 at 3:45 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Wed, Jan 10, 2018 at 3:00 PM, Anirudh Ramanathan
> <ra...@google.com> wrote:
> > We can start by getting a PR going perhaps, and start augmenting the
> > integration testing to ensure that there are no surprises - with/without
> > credentials, accessing GCS, S3 etc as well.
> > When we get enough confidence and test coverage, let's merge this in.
> > Does that sound like a reasonable path forward?
>
> I think it's beneficial to separate this into two separate things as
> far as discussion goes:
>
> - using spark-submit: the code should definitely be starting the
> driver using spark-submit, and potentially the executor using
> spark-class.
>
> - separately, we can decide on whether to keep or remove init containers.
>
> Unfortunately, code-wise, those are not separate. If you get rid of
> init containers, my current p.o.c. has most of the needed changes
> (only lightly tested).
>
> But if you keep init containers, you'll need to mess with the
> configuration so that spark-submit never sees spark.jars /
> spark.files, so it doesn't trigger its dependency download code. (YARN
> does something similar, btw.) That will surely mean different changes
> in the current k8s code (which I wanted to double check anyway because
> I remember seeing some oddities related to those configs in the logs).
>
> To comment on one point made by Andrew:
> > there's almost a parallel here with spark.yarn.archive, where that
> configures the cluster (YARN) to do distribution pre-runtime
>
> That's more of a parallel to the docker image; spark.yarn.archive
> points to a jar file with Spark jars in it so that YARN can make Spark
> available to the driver / executors running in the cluster.
>
> Like the docker image, you could include other stuff that is not
> really part of standard Spark in that archive too, or even not have
> Spark at all there, if you want things to just fail. :-)
>
> --
> Marcelo
>
--
Anirudh Ramanathan
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Wed, Jan 10, 2018 at 3:00 PM, Anirudh Ramanathan
<ra...@google.com> wrote:
> We can start by getting a PR going perhaps, and start augmenting the
> integration testing to ensure that there are no surprises - with/without
> credentials, accessing GCS, S3 etc as well.
> When we get enough confidence and test coverage, let's merge this in.
> Does that sound like a reasonable path forward?
I think it's beneficial to separate this into two separate things as
far as discussion goes:
- using spark-submit: the code should definitely be starting the
driver using spark-submit, and potentially the executor using
spark-class.
- separately, we can decide on whether to keep or remove init containers.
Unfortunately, code-wise, those are not separate. If you get rid of
init containers, my current p.o.c. has most of the needed changes
(only lightly tested).
But if you keep init containers, you'll need to mess with the
configuration so that spark-submit never sees spark.jars /
spark.files, so it doesn't trigger its dependency download code. (YARN
does something similar, btw.) That will surely mean different changes
in the current k8s code (which I wanted to double check anyway because
I remember seeing some oddities related to those configs in the logs).
To comment on one point made by Andrew:
> there's almost a parallel here with spark.yarn.archive, where that configures the cluster (YARN) to do distribution pre-runtime
That's more of a parallel to the docker image; spark.yarn.archive
points to a jar file with Spark jars in it so that YARN can make Spark
available to the driver / executors running in the cluster.
Like the docker image, you could include other stuff that is not
really part of standard Spark in that archive too, or even not have
Spark at all there, if you want things to just fail. :-)
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Anirudh Ramanathan <ra...@google.com.INVALID>.
Thanks for this discussion everyone. It has been very useful in getting an
overall understanding here.
I think in general, consensus is that this change doesn't introduce
behavioral changes, and it's definitely an advantage to reuse the
constructs that Spark provides to us.
Moving on to a different question here - of pushing this through to Spark.
The init containers have been tested over the past two Spark releases by
external users and integration testing - and this would be a fundamental
change to that behavior.
We should work on getting enough test coverage and confidence here.
We can start by getting a PR going perhaps, and start augmenting the
integration testing to ensure that there are no surprises - with/without
credentials, accessing GCS, S3 etc as well.
When we get enough confidence and test coverage, let's merge this in.
Does that sound like a reasonable path forward?
On Wed, Jan 10, 2018 at 2:53 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Wed, Jan 10, 2018 at 2:51 PM, Matt Cheah <mc...@palantir.com> wrote:
> > those sidecars may perform side effects that are undesirable if the main
> Spark application failed because dependencies weren’t available
>
> If the contract is that the Spark driver pod does not have an init
> container, and the driver handles its own dependencies, then by
> definition that situation cannot exist.
>
> --
> Marcelo
>
--
Anirudh Ramanathan
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Wed, Jan 10, 2018 at 2:51 PM, Matt Cheah <mc...@palantir.com> wrote:
> those sidecars may perform side effects that are undesirable if the main Spark application failed because dependencies weren’t available
If the contract is that the Spark driver pod does not have an init
container, and the driver handles its own dependencies, then by
definition that situation cannot exist.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Matt Cheah <mc...@palantir.com>.
With regards to separation of concerns, there’s a fringe use case here – if more than one main container is on the pod, then none of them will run if the init-containers fail. A user can have a Pod Preset that attaches more sidecar containers to the driver and/or executors. In that case, those sidecars may perform side effects that are undesirable if the main Spark application failed because dependencies weren’t available. Using the init-container to localize the dependencies will prevent any of these sidecars from executing at all if the dependencies can’t be fetched.
It’s definitely a niche use case – I’m not sure how often pod presets are used in practice - but it’s an example to illustrate why the separation of concerns can be beneficial.
-Matt Cheah
On 1/10/18, 2:36 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
On Wed, Jan 10, 2018 at 2:30 PM, Yinan Li <li...@gmail.com> wrote:
> 1. Retries of init-containers are automatically supported by k8s through pod
> restart policies. For this point, sorry I'm not sure how spark-submit
> achieves this.
Great, add that feature to spark-submit, everybody benefits, not just k8s.
> 2. The ability to use credentials that are not shared with the main
> containers.
Not sure what that achieves.
> 3. Not only the user code, but Spark internal code like Executor won't be
> run if the init-container fails.
Not sure what that achieves. Executor will fail if dependency download
fails, Spark driver will recover (and start a new executor if needed).
> 4. Easier to build tooling around k8s events/status of the init-container in
> case of failures as it's doing exactly one thing: downloading dependencies.
Again, I don't see what is all this hoopla about fine grained control
of dependency downloads. Spark solved this years ago for Spark
applications. Don't reinvent the wheel.
--
Marcelo
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Fri, Jan 12, 2018 at 4:13 AM, Eric Charles <er...@apache.org> wrote:
>> Again, I don't see what is all this hoopla about fine grained control
>> of dependency downloads. Spark solved this years ago for Spark
>> applications. Don't reinvent the wheel.
>
> Init-containers are used today to download dependencies. I may be wrong and
> may open another facet of the discussion, but I see init container usage in
> a more generic way and not only restricted to dependencies download.
I'm not trying to discuss the general benefits of init containers as
it pertains to the kubernetes framework. I'm sure they added those for
a reason.
I'm trying to discuss them in the restricted scope of the spark-on-k8s
integration. And there, there is a single use for the single init
container that the Spark code itself injects into the pod: downloading
dependencies, which is something that spark-submit already does.
There's an option to override that one init container image with
another, where you can completely change its behavior. Given that
there is no contract currently that explains how these images should
behave, doing so is very, very risky and might break the application
completely (e.g. because dependencies are now not being downloaded, or
placed in the wrong location).
An you can do the exact same thing by overriding the main Spark image
itself. Just run the same code in your custom entry point before the
Spark-provided entry point runs. Same results and caveats as above
apply.
So again, the specific init container used by spark-on-k8s, as far as
I can see, seems to cause more problems than it solves.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Wed, Jan 10, 2018 at 2:30 PM, Yinan Li <li...@gmail.com> wrote:
> 1. Retries of init-containers are automatically supported by k8s through pod
> restart policies. For this point, sorry I'm not sure how spark-submit
> achieves this.
Great, add that feature to spark-submit, everybody benefits, not just k8s.
> 2. The ability to use credentials that are not shared with the main
> containers.
Not sure what that achieves.
> 3. Not only the user code, but Spark internal code like Executor won't be
> run if the init-container fails.
Not sure what that achieves. Executor will fail if dependency download
fails, Spark driver will recover (and start a new executor if needed).
> 4. Easier to build tooling around k8s events/status of the init-container in
> case of failures as it's doing exactly one thing: downloading dependencies.
Again, I don't see what is all this hoopla about fine grained control
of dependency downloads. Spark solved this years ago for Spark
applications. Don't reinvent the wheel.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Yinan Li <li...@gmail.com>.
> Sorry, but what are those again? So far all the benefits are already
> provided by spark-submit...
1. Retries of init-containers are automatically supported by k8s through
pod restart policies. For this point, sorry I'm not sure how spark-submit
achieves this.
2. The ability to use credentials that are not shared with the main
containers.
3. Not only the user code, but Spark internal code like Executor won't be
run if the init-container fails.
4. Easier to build tooling around k8s events/status of the init-container
in case of failures as it's doing exactly one thing: downloading
dependencies.
There could be others that I'm not aware of.
On Wed, Jan 10, 2018 at 2:21 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Wed, Jan 10, 2018 at 2:16 PM, Yinan Li <li...@gmail.com> wrote:
> > but we can not rule out the benefits init-containers bring either.
>
> Sorry, but what are those again? So far all the benefits are already
> provided by spark-submit...
>
> > Again, I would suggest we look at this more thoroughly post 2.3.
>
> Actually, one of the reasons why I brought this up is that we should
> remove init containers from 2.3 unless they're really required for
> something.
>
> Simplifying the code is not the only issue. The init container support
> introduces a whole lot of user-visible behavior - like config options
> and the execution of a completely separate container that the user can
> customize. If removed later, that could be considered a breaking
> change.
>
> So if we ship 2.3 without init containers and add them later if
> needed, it's a much better world than flipping that around.
>
> --
> Marcelo
>
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Wed, Jan 10, 2018 at 2:16 PM, Yinan Li <li...@gmail.com> wrote:
> but we can not rule out the benefits init-containers bring either.
Sorry, but what are those again? So far all the benefits are already
provided by spark-submit...
> Again, I would suggest we look at this more thoroughly post 2.3.
Actually, one of the reasons why I brought this up is that we should
remove init containers from 2.3 unless they're really required for
something.
Simplifying the code is not the only issue. The init container support
introduces a whole lot of user-visible behavior - like config options
and the execution of a completely separate container that the user can
customize. If removed later, that could be considered a breaking
change.
So if we ship 2.3 without init containers and add them later if
needed, it's a much better world than flipping that around.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Yinan Li <li...@gmail.com>.
> 1500 less lines of code trump all of the arguments given so far for
> what the init container might be a good idea.
We can also reduce the #lines of code by simply refactoring the code in
such as way that a lot of code can be shared between configuration of the
main container and that of the ini-container. Actually we have been
discussing this as one of the things to do right after the 2.3 release and
we do have a Jira ticket to track it. It's probably true that none of the
arguments we made are convincing enough, but we can not rule out the
benefits init-containers bring either.
Again, I would suggest we look at this more thoroughly post 2.3.
On Wed, Jan 10, 2018 at 2:06 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Wed, Jan 10, 2018 at 2:00 PM, Yinan Li <li...@gmail.com> wrote:
> > I want to re-iterate on one point, that the init-container achieves a
> clear
> > separation between preparing an application and actually running the
> > application. It's a guarantee provided by the K8s admission control and
> > scheduling components that if the init-container fails, the main
> container
> > won't be run. I think this is definitely positive to have. In the case
> of a
> > Spark application, the application code and driver/executor code won't
> even
> > be run if the init-container fails to localize any of the dependencies
>
> That is also the case with spark-submit... (can't download
> dependencies -> spark-submit fails before running user code).
>
> > Note that we are not blindly opposing getting rid of the init-container,
> > it's just that there's still valid reasons to keep it for now
>
> I'll flip that around: I'm not against having an init container if
> it's serving a needed purpose, it's just that nobody is able to tell
> me what that needed purpose is.
>
> 1500 less lines of code trump all of the arguments given so far for
> what the init container might be a good idea.
>
> --
> Marcelo
>
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Wed, Jan 10, 2018 at 2:00 PM, Yinan Li <li...@gmail.com> wrote:
> I want to re-iterate on one point, that the init-container achieves a clear
> separation between preparing an application and actually running the
> application. It's a guarantee provided by the K8s admission control and
> scheduling components that if the init-container fails, the main container
> won't be run. I think this is definitely positive to have. In the case of a
> Spark application, the application code and driver/executor code won't even
> be run if the init-container fails to localize any of the dependencies
That is also the case with spark-submit... (can't download
dependencies -> spark-submit fails before running user code).
> Note that we are not blindly opposing getting rid of the init-container,
> it's just that there's still valid reasons to keep it for now
I'll flip that around: I'm not against having an init container if
it's serving a needed purpose, it's just that nobody is able to tell
me what that needed purpose is.
1500 less lines of code trump all of the arguments given so far for
what the init container might be a good idea.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Yinan Li <li...@gmail.com>.
I want to re-iterate on one point, that the init-container achieves a clear
separation between preparing an application and actually running the
application. It's a guarantee provided by the K8s admission control and
scheduling components that if the init-container fails, the main container
won't be run. I think this is definitely positive to have. In the case of a
Spark application, the application code and driver/executor code won't even
be run if the init-container fails to localize any of the dependencies. The
result is that it's much easier for users to figure out what's wrong if
their applications fail to run: they can tell if the pods are initialized
or not and if not, simply check the status/logs of the init-container.
Another argument I want to make is we can easily make the init-container to
be able to exclusively use certain credentials for downloading dependencies
that are not appropriate to be visible in the main containers and therefore
should not be shared. This is not achievable using the Spark canonical way.
K8s has built-in support for dynamically injecting containers into pods
through the admission control process. One use case would be for cluster
operators to inject an init-container (e.g., through a admission webhook)
for downloading certain dependencies that require certain
access-restrictive credentials.
Note that we are not blindly opposing getting rid of the init-container,
it's just that there's still valid reasons to keep it for now, particularly
given that we don't have a solid around client mode yet. Also given that we
have been using it in our fork for over a year, we are definitely more
confident on the current way of handling remote dependencies as it's been
tested more thoroughly. Since getting rid of the init-container is such a
significant change, I would suggest that we defer making a decision on if
we should get rid of it to 2.4 so we have a more thorough understanding of
the pros and cons.
On Wed, Jan 10, 2018 at 1:48 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Wed, Jan 10, 2018 at 1:47 PM, Matt Cheah <mc...@palantir.com> wrote:
> >> With a config value set by the submission code, like what I'm doing to
> prevent client mode submission in my p.o.c.?
> >
> > The contract for what determines the appropriate scheduler backend to
> instantiate is then going to be different in Kubernetes versus the other
> cluster managers.
>
> There is no contract for how to pick the appropriate scheduler. That's
> a decision that is completely internal to the cluster manager code
>
> --
> Marcelo
>
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Wed, Jan 10, 2018 at 1:47 PM, Matt Cheah <mc...@palantir.com> wrote:
>> With a config value set by the submission code, like what I'm doing to prevent client mode submission in my p.o.c.?
>
> The contract for what determines the appropriate scheduler backend to instantiate is then going to be different in Kubernetes versus the other cluster managers.
There is no contract for how to pick the appropriate scheduler. That's
a decision that is completely internal to the cluster manager code
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Matt Cheah <mc...@palantir.com>.
> With a config value set by the submission code, like what I'm doing to prevent client mode submission in my p.o.c.?
The contract for what determines the appropriate scheduler backend to instantiate is then going to be different in Kubernetes versus the other cluster managers. The cluster manager typically only picks the scheduler backend implementation based on the master URL format plus the deploy mode. Perhaps this is an acceptable tradeoff for being able to leverage spark-submit in the cluster mode deployed driver container. Again though, any flag we expose in spark-submit is a user-facing option that can be set erroneously, which is a practice we shouldn’t be encouraging.
Taking a step back though, I think we want to use spark-submit’s internals without using spark-submit itself. Any flags we add to spark-submit are user-facing. We ideally would be able to extract the dependency download + run user main class subroutines from spark-submit, and invoke that in all of the cluster managers. Perhaps this calls for a refactor in spark-submit itself to make some parts reusable in other contexts. Just an idea.
On 1/10/18, 1:38 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
On Wed, Jan 10, 2018 at 1:33 PM, Matt Cheah <mc...@palantir.com> wrote:
> If we use spark-submit in client mode from the driver container, how do we handle needing to switch between a cluster-mode scheduler backend and a client-mode scheduler backend in the future?
With a config value set by the submission code, like what I'm doing to
prevent client mode submission in my p.o.c.?
There are plenty of solutions to that problem if that's what's worrying you.
> Something else re: client mode accessibility – if we make client mode accessible to users even if it’s behind a flag, that’s a very different contract from needing to recompile spark-submit to support client mode. The amount of effort required from the user to get to client mode is very different between the two cases
Yes. But if we say we don't support client mode, we don't support
client mode regardless of how easy it is for the user to fool Spark
into trying to run in that mode.
--
Marcelo
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Wed, Jan 10, 2018 at 1:33 PM, Matt Cheah <mc...@palantir.com> wrote:
> If we use spark-submit in client mode from the driver container, how do we handle needing to switch between a cluster-mode scheduler backend and a client-mode scheduler backend in the future?
With a config value set by the submission code, like what I'm doing to
prevent client mode submission in my p.o.c.?
There are plenty of solutions to that problem if that's what's worrying you.
> Something else re: client mode accessibility – if we make client mode accessible to users even if it’s behind a flag, that’s a very different contract from needing to recompile spark-submit to support client mode. The amount of effort required from the user to get to client mode is very different between the two cases
Yes. But if we say we don't support client mode, we don't support
client mode regardless of how easy it is for the user to fool Spark
into trying to run in that mode.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Matt Cheah <mc...@palantir.com>.
If we use spark-submit in client mode from the driver container, how do we handle needing to switch between a cluster-mode scheduler backend and a client-mode scheduler backend in the future?
Something else re: client mode accessibility – if we make client mode accessible to users even if it’s behind a flag, that’s a very different contract from needing to recompile spark-submit to support client mode. The amount of effort required from the user to get to client mode is very different between the two cases, and the contract is much clearer when client mode is forbidden in all circumstances, versus client mode being allowed with a specific flag. If we’re saying that we don’t support client mode, we should bias towards making client mode as difficult as possible to access, i.e. impossible with a standard Spark distribution.
-Matt Cheah
On 1/10/18, 1:24 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
On Wed, Jan 10, 2018 at 1:10 PM, Matt Cheah <mc...@palantir.com> wrote:
> I’d imagine this is a reason why YARN hasn’t went with using spark-submit from the application master...
I wouldn't use YARN as a template to follow when writing a new
backend. A lot of the reason why the YARN backend works the way it
does is because of backwards compatibility. IMO it would be much
better to change the YARN backend to use spark-submit, because it
would immensely simplify the code there. It was a nightmare to get
YARN to reach feature parity with other backends because it has to
pretty much reimplement everything.
But doing that would break pretty much every Spark-on-YARN deployment,
so it's not something we can do right now.
For the other backends the situation is sort of similar; it probably
wouldn't be hard to change standalone's DriverWrapper to also use
spark-submit. But that brings potential side effects for existing
users that don't exist with spark-on-k8s, because spark-on-k8s is new
(the current fork aside).
> But using init-containers makes it such that we don’t need to use spark-submit at all
Those are actually separate concerns. There are a whole bunch of
things that spark-submit provides you that you'd have to replicate in
the k8s backend if not using it. Thinks like properly handling special
characters in arguments, native library paths, "userClassPathFirst",
etc. You get them almost for free with spark-submit, and using an init
container does not solve any of those for you.
I'd say that using spark-submit is really not up for discussion here;
it saves you from re-implementing a whole bunch of code that you
shouldn't even be trying to re-implement.
Separately, if there is a legitimate need for an init container, then
it can be added. But I don't see that legitimate need right now, so I
don't see what it's bringing other than complexity.
(And no, "the k8s documentation mentions that init containers are
sometimes used to download dependencies" is not a legitimate need.)
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Wed, Jan 10, 2018 at 1:10 PM, Matt Cheah <mc...@palantir.com> wrote:
> I’d imagine this is a reason why YARN hasn’t went with using spark-submit from the application master...
I wouldn't use YARN as a template to follow when writing a new
backend. A lot of the reason why the YARN backend works the way it
does is because of backwards compatibility. IMO it would be much
better to change the YARN backend to use spark-submit, because it
would immensely simplify the code there. It was a nightmare to get
YARN to reach feature parity with other backends because it has to
pretty much reimplement everything.
But doing that would break pretty much every Spark-on-YARN deployment,
so it's not something we can do right now.
For the other backends the situation is sort of similar; it probably
wouldn't be hard to change standalone's DriverWrapper to also use
spark-submit. But that brings potential side effects for existing
users that don't exist with spark-on-k8s, because spark-on-k8s is new
(the current fork aside).
> But using init-containers makes it such that we don’t need to use spark-submit at all
Those are actually separate concerns. There are a whole bunch of
things that spark-submit provides you that you'd have to replicate in
the k8s backend if not using it. Thinks like properly handling special
characters in arguments, native library paths, "userClassPathFirst",
etc. You get them almost for free with spark-submit, and using an init
container does not solve any of those for you.
I'd say that using spark-submit is really not up for discussion here;
it saves you from re-implementing a whole bunch of code that you
shouldn't even be trying to re-implement.
Separately, if there is a legitimate need for an init container, then
it can be added. But I don't see that legitimate need right now, so I
don't see what it's bringing other than complexity.
(And no, "the k8s documentation mentions that init containers are
sometimes used to download dependencies" is not a legitimate need.)
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Matt Cheah <mc...@palantir.com>.
A crucial point here is considering whether we want to have a separate scheduler backend code path for client mode versus cluster mode. If we need such a separation in the code paths, it would be difficult to make it possible to run spark-submit in client mode from the driver container.
We discussed this already when we started to think about client mode. See https://github.com/apache-spark-on-k8s/spark/pull/456. In our initial designs for a client mode, we considered that there are some concepts that would only apply to cluster mode and not to client mode – see https://github.com/apache-spark-on-k8s/spark/pull/456#issuecomment-343007093. But we haven’t worked out all of the details yet. The situation may work out such that client mode is similar enough to cluster mode that we can consider the cluster mode as being a spark-submit in client mode from a container.
I’d imagine this is a reason why YARN hasn’t went with using spark-submit from the application master, because there are separate code paths for a YarnClientSchedulerBackend versus a YarnClusterSchedulerBackend, and the deploy mode serves as the switch between the two implementations. Though I am curious as to why Spark standalone isn’t using spark-submit – the DriverWrapper is manually fetching the user’s jars and putting them on a classloader before invoking the user’s main class with that classloader. But there’s only one scheduler backend for both client and cluster mode for standalone’s case.
The main idea here is that we need to understand if we need different code paths for a client mode scheduler backend versus a cluster mode scheduler backend, before we can know if we can use spark-submit in client mode from the driver container. But using init-containers makes it such that we don’t need to use spark-submit at all, meaning that the differences can more or less be ignored at least in this particular context.
-Matt Cheah
On 1/10/18, 8:40 AM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
On a side note, while it's great that you guys have meetings to
discuss things related to the project, it's general Apache practice to
discuss these things in the mailing list - or at the very list send
detailed info about what discussed in these meetings to the mailing
list. Not everybody can attend these meetings, and I'm not just
talking about people being busy, but there are people who live in
different time zones.
Now that this code is moving into Spark I'd recommend getting people
more involved with the Spark project to move things forward.
On Tue, Jan 9, 2018 at 8:23 PM, Anirudh Ramanathan
<ra...@google.com> wrote:
> Marcelo, I can see that we might be misunderstanding what this change
> implies for performance and some of the deeper implementation details here.
> We have a community meeting tomorrow (at 10am PT), and we'll be sure to
> explore this idea in detail, and understand the implications and then get
> back to you.
>
> Thanks for the detailed responses here, and for spending time with the idea.
> (Also, you're more than welcome to attend the meeting - there's a link here
> if you're around.)
>
> Cheers,
> Anirudh
>
>
> On Jan 9, 2018 8:05 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
>
> One thing I forgot in my previous e-mail is that if a resource is
> remote I'm pretty sure (but haven't double checked the code) that
> executors will download it directly from the remote server, and not
> from the driver. So there, distributed download without an init
> container.
>
> On Tue, Jan 9, 2018 at 7:15 PM, Yinan Li <li...@gmail.com> wrote:
>> The init-container is required for use with the resource staging server
>>
>> (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache-2Dspark-2Don-2Dk8s_userdocs_blob_master_src_jekyll_running-2Don-2Dkubernetes.md-23resource-2Dstaging-2Dserver&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=rQzoyVLMucfZdPLZAwNlE-PZ90ViBJzTQ49K1dzjr3c&s=HcCtT_KLkPi_05ojHei1nbXUpwoJomou8bitD-WkYmI&e=).
>
> If the staging server *requires* an init container you have already a
> design problem right there.
>
>> Additionally, the init-container is a Kubernetes
>> native way of making sure that the dependencies are localized
>
> Sorry, but the init container does not do anything by itself. You had
> to add a whole bunch of code to execute the existing Spark code in an
> init container, when not doing it would have achieved the exact same
> goal much more easily, in a way that is consistent with how Spark
> already does things.
>
> Matt:
>> the executors wouldn’t receive the jars on their class loader until after
>> the executor starts
>
> I actually consider that a benefit. It means spark-on-k8s application
> will behave more like all the other backends, where that is true also
> (application jars live in a separate class loader).
>
>> traditionally meant to prepare the environment for the application that is
>> to be run
>
> You guys are forcing this argument when it all depends on where you
> draw the line. Spark can be launched without downloading any of those
> dependencies, because Spark will download them for you. Forcing the
> "kubernetes way" just means you're writing a lot more code, and
> breaking the Spark app initialization into multiple container
> invocations, to achieve the same thing.
>
>> would make the SparkSubmit code inadvertently allow running client mode
>> Kubernetes applications as well
>
> Not necessarily. I have that in my patch; it doesn't allow client mode
> unless a property that only the cluster mode submission code sets is
> present. If some user wants to hack their way around that, more power
> to them; users can also compile their own Spark without the checks if
> they want to try out client mode in some way.
>
> Anirudh:
>> Telling users that they must rebuild images ... every time seems less
>> than convincing to me.
>
> Sure, I'm not proposing people use the docker image approach all the
> time. It would be a hassle while developing an app, as it is kind of a
> hassle today where the code doesn't upload local files to the k8s
> cluster.
>
> But it's perfectly reasonable for people to optimize a production app
> by bundling the app into a pre-built docker image to avoid
> re-downloading resources every time. Like they'd probably place the
> jar + dependencies on HDFS today with YARN, to get the benefits of the
> YARN cache.
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
Marcelo
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On a side note, while it's great that you guys have meetings to
discuss things related to the project, it's general Apache practice to
discuss these things in the mailing list - or at the very list send
detailed info about what discussed in these meetings to the mailing
list. Not everybody can attend these meetings, and I'm not just
talking about people being busy, but there are people who live in
different time zones.
Now that this code is moving into Spark I'd recommend getting people
more involved with the Spark project to move things forward.
On Tue, Jan 9, 2018 at 8:23 PM, Anirudh Ramanathan
<ra...@google.com> wrote:
> Marcelo, I can see that we might be misunderstanding what this change
> implies for performance and some of the deeper implementation details here.
> We have a community meeting tomorrow (at 10am PT), and we'll be sure to
> explore this idea in detail, and understand the implications and then get
> back to you.
>
> Thanks for the detailed responses here, and for spending time with the idea.
> (Also, you're more than welcome to attend the meeting - there's a link here
> if you're around.)
>
> Cheers,
> Anirudh
>
>
> On Jan 9, 2018 8:05 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
>
> One thing I forgot in my previous e-mail is that if a resource is
> remote I'm pretty sure (but haven't double checked the code) that
> executors will download it directly from the remote server, and not
> from the driver. So there, distributed download without an init
> container.
>
> On Tue, Jan 9, 2018 at 7:15 PM, Yinan Li <li...@gmail.com> wrote:
>> The init-container is required for use with the resource staging server
>>
>> (https://github.com/apache-spark-on-k8s/userdocs/blob/master/src/jekyll/running-on-kubernetes.md#resource-staging-server).
>
> If the staging server *requires* an init container you have already a
> design problem right there.
>
>> Additionally, the init-container is a Kubernetes
>> native way of making sure that the dependencies are localized
>
> Sorry, but the init container does not do anything by itself. You had
> to add a whole bunch of code to execute the existing Spark code in an
> init container, when not doing it would have achieved the exact same
> goal much more easily, in a way that is consistent with how Spark
> already does things.
>
> Matt:
>> the executors wouldn’t receive the jars on their class loader until after
>> the executor starts
>
> I actually consider that a benefit. It means spark-on-k8s application
> will behave more like all the other backends, where that is true also
> (application jars live in a separate class loader).
>
>> traditionally meant to prepare the environment for the application that is
>> to be run
>
> You guys are forcing this argument when it all depends on where you
> draw the line. Spark can be launched without downloading any of those
> dependencies, because Spark will download them for you. Forcing the
> "kubernetes way" just means you're writing a lot more code, and
> breaking the Spark app initialization into multiple container
> invocations, to achieve the same thing.
>
>> would make the SparkSubmit code inadvertently allow running client mode
>> Kubernetes applications as well
>
> Not necessarily. I have that in my patch; it doesn't allow client mode
> unless a property that only the cluster mode submission code sets is
> present. If some user wants to hack their way around that, more power
> to them; users can also compile their own Spark without the checks if
> they want to try out client mode in some way.
>
> Anirudh:
>> Telling users that they must rebuild images ... every time seems less
>> than convincing to me.
>
> Sure, I'm not proposing people use the docker image approach all the
> time. It would be a hassle while developing an app, as it is kind of a
> hassle today where the code doesn't upload local files to the k8s
> cluster.
>
> But it's perfectly reasonable for people to optimize a production app
> by bundling the app into a pre-built docker image to avoid
> re-downloading resources every time. Like they'd probably place the
> jar + dependencies on HDFS today with YARN, to get the benefits of the
> YARN cache.
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Anirudh Ramanathan <ra...@google.com.INVALID>.
Marcelo, I can see that we might be misunderstanding what this change
implies for performance and some of the deeper implementation details here.
We have a community meeting tomorrow (at 10am PT), and we'll be sure to
explore this idea in detail, and understand the implications and then get
back to you.
Thanks for the detailed responses here, and for spending time with the idea.
(Also, you're more than welcome to attend the meeting - there's a link
<https://github.com/kubernetes/community/tree/master/sig-big-data> here if
you're around.)
Cheers,
Anirudh
On Jan 9, 2018 8:05 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
One thing I forgot in my previous e-mail is that if a resource is
remote I'm pretty sure (but haven't double checked the code) that
executors will download it directly from the remote server, and not
from the driver. So there, distributed download without an init
container.
On Tue, Jan 9, 2018 at 7:15 PM, Yinan Li <li...@gmail.com> wrote:
> The init-container is required for use with the resource staging server
> (https://github.com/apache-spark-on-k8s/userdocs/blob/master
/src/jekyll/running-on-kubernetes.md#resource-staging-server).
If the staging server *requires* an init container you have already a
design problem right there.
> Additionally, the init-container is a Kubernetes
> native way of making sure that the dependencies are localized
Sorry, but the init container does not do anything by itself. You had
to add a whole bunch of code to execute the existing Spark code in an
init container, when not doing it would have achieved the exact same
goal much more easily, in a way that is consistent with how Spark
already does things.
Matt:
> the executors wouldn’t receive the jars on their class loader until after
the executor starts
I actually consider that a benefit. It means spark-on-k8s application
will behave more like all the other backends, where that is true also
(application jars live in a separate class loader).
> traditionally meant to prepare the environment for the application that
is to be run
You guys are forcing this argument when it all depends on where you
draw the line. Spark can be launched without downloading any of those
dependencies, because Spark will download them for you. Forcing the
"kubernetes way" just means you're writing a lot more code, and
breaking the Spark app initialization into multiple container
invocations, to achieve the same thing.
> would make the SparkSubmit code inadvertently allow running client mode
Kubernetes applications as well
Not necessarily. I have that in my patch; it doesn't allow client mode
unless a property that only the cluster mode submission code sets is
present. If some user wants to hack their way around that, more power
to them; users can also compile their own Spark without the checks if
they want to try out client mode in some way.
Anirudh:
> Telling users that they must rebuild images ... every time seems less
than convincing to me.
Sure, I'm not proposing people use the docker image approach all the
time. It would be a hassle while developing an app, as it is kind of a
hassle today where the code doesn't upload local files to the k8s
cluster.
But it's perfectly reasonable for people to optimize a production app
by bundling the app into a pre-built docker image to avoid
re-downloading resources every time. Like they'd probably place the
jar + dependencies on HDFS today with YARN, to get the benefits of the
YARN cache.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
One thing I forgot in my previous e-mail is that if a resource is
remote I'm pretty sure (but haven't double checked the code) that
executors will download it directly from the remote server, and not
from the driver. So there, distributed download without an init
container.
On Tue, Jan 9, 2018 at 7:15 PM, Yinan Li <li...@gmail.com> wrote:
> The init-container is required for use with the resource staging server
> (https://github.com/apache-spark-on-k8s/userdocs/blob/master/src/jekyll/running-on-kubernetes.md#resource-staging-server).
If the staging server *requires* an init container you have already a
design problem right there.
> Additionally, the init-container is a Kubernetes
> native way of making sure that the dependencies are localized
Sorry, but the init container does not do anything by itself. You had
to add a whole bunch of code to execute the existing Spark code in an
init container, when not doing it would have achieved the exact same
goal much more easily, in a way that is consistent with how Spark
already does things.
Matt:
> the executors wouldn’t receive the jars on their class loader until after the executor starts
I actually consider that a benefit. It means spark-on-k8s application
will behave more like all the other backends, where that is true also
(application jars live in a separate class loader).
> traditionally meant to prepare the environment for the application that is to be run
You guys are forcing this argument when it all depends on where you
draw the line. Spark can be launched without downloading any of those
dependencies, because Spark will download them for you. Forcing the
"kubernetes way" just means you're writing a lot more code, and
breaking the Spark app initialization into multiple container
invocations, to achieve the same thing.
> would make the SparkSubmit code inadvertently allow running client mode Kubernetes applications as well
Not necessarily. I have that in my patch; it doesn't allow client mode
unless a property that only the cluster mode submission code sets is
present. If some user wants to hack their way around that, more power
to them; users can also compile their own Spark without the checks if
they want to try out client mode in some way.
Anirudh:
> Telling users that they must rebuild images ... every time seems less than convincing to me.
Sure, I'm not proposing people use the docker image approach all the
time. It would be a hassle while developing an app, as it is kind of a
hassle today where the code doesn't upload local files to the k8s
cluster.
But it's perfectly reasonable for people to optimize a production app
by bundling the app into a pre-built docker image to avoid
re-downloading resources every time. Like they'd probably place the
jar + dependencies on HDFS today with YARN, to get the benefits of the
YARN cache.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Anirudh Ramanathan <ra...@google.com.INVALID>.
Marcelo, to address the points you raised:
> k8s uses docker images. Users can create docker images with all the
dependencies their app needs, and submit the app using that image.
The entire reason why we support additional methods of localizing
dependencies than baking everything into docker images is that
it's not a very good workflow fit for all use-cases. There are definitely
some users that will do that (and I've spoken to some),
and they build a versioned image in their registry every time they change
their code with a CD pipeline,
but a lot of people are looking for something lighter - and versioning
application code, not entire images.
Telling users that they must rebuild images and pay the cost of localizing
new images from the docker registry
(which is also not very well understood/measured in terms of performance)
every time seems less than convincing to me.
> - The original spark-on-k8s spec mentioned a "dependency server"
approach which sounded like a more generic version of the YARN
distributed cache, which I hope can be a different way of mitigating
that issue. With that work, we could build this functionality into
spark-submit itself and have other backends also benefit.
The resource staging server as was written was a non-HA fileserver for
staging dependencies within the cluster.
It's not distributed, and has no notion of locality, etc. I don't think we
had plans (yet) to invest in to make it more
like the distributed cache you mentioned, at least not until we heard
back from the community - so, that's unplanned work at this point. It's
also hard to imagine how we could
extend that to go beyond just K8s tbh. We should definitely have a JIRA
tracking this, if that's a
direction we want to explore in the future.
I understand the change you're proposing would simplify the code but a
decision here seems hard to make
until we get some real benchmarks/measurements, or user feedback.
On Tue, Jan 9, 2018 at 7:24 PM, Matt Cheah <mc...@palantir.com> wrote:
> A few reasons to prefer init-containers come to mind:
>
>
>
> Firstly, if we used spark-submit from within the driver container, the
> executors wouldn’t receive the jars on their class loader until after the
> executor starts because the executor has to launch first before localizing
> resources. It is certainly possible to make the class loader work with the
> user’s jars here, as is the case with all the client mode implementations,
> but, it seems cleaner to have the classpath include the user’s jars at
> executor launch time instead of needing to reason about the classloading
> order.
>
>
>
> We can also consider the idiomatic approach from the perspective of
> Kubernetes. Yinan touched on this already, but init-containers are
> traditionally meant to prepare the environment for the application that is
> to be run, which is exactly what we do here. This also makes it such that
> the localization process can be completely decoupled from the execution of
> the application itself. We can then for example detect the errors that
> happen on the resource localization layer, say when an HDFS cluster is
> down, before the application itself launches. The failure at the
> init-container stage is explicitly noted via the Kubernetes pod status API.
>
>
>
> Finally, running spark-submit from the container would make the
> SparkSubmit code inadvertently allow running client mode Kubernetes
> applications as well. We’re not quite ready to support that. Even if we
> were, it’s not entirely intuitive for the cluster mode code path to depend
> on the client mode code path. This isn’t entirely without precedent though,
> as Mesos has a similar dependency.
>
>
>
> Essentially the semantics seem neater and the contract is very explicit
> when using an init-container, even though the code does end up being more
> complex.
>
>
>
> *From: *Yinan Li <li...@gmail.com>
> *Date: *Tuesday, January 9, 2018 at 7:16 PM
> *To: *Nicholas Chammas <ni...@gmail.com>
> *Cc: *Anirudh Ramanathan <ra...@google.com.invalid>, Marcelo Vanzin
> <va...@cloudera.com>, Matt Cheah <mc...@palantir.com>, Kimoon Kim <
> kimoon@pepperdata.com>, dev <de...@spark.apache.org>
> *Subject: *Re: Kubernetes: why use init containers?
>
>
>
> The init-container is required for use with the resource staging server (
> https://github.com/apache-spark-on-k8s/userdocs/blob/
> master/src/jekyll/running-on-kubernetes.md#resource-
> staging-server[github.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache-2Dspark-2Don-2Dk8s_userdocs_blob_master_src_jekyll_running-2Don-2Dkubernetes.md-23resource-2Dstaging-2Dserver&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=q1Zla4QOVxYBCOe2Y8c9fm5ONO6hBMezCzXGoSL-714&s=gxXPcm4K56VS9gWykUuYmpdLJjYfkCc5HE6X4jD3US8&e=>).
> The resource staging server (RSS) is a spark-on-k8s component running in a
> Kubernetes cluster for staging submission client local dependencies to
> Spark pods. The init-container is responsible for downloading the
> dependencies from the RSS. We haven't upstream the RSS code yet, but this
> is a value add component for Spark on K8s as a way for users to use
> submission local dependencies without resorting to other mechanisms that
> are not immediately available on most Kubernetes clusters, e.g., HDFS. We
> do plan to upstream it in the 2.4 timeframe. Additionally, the
> init-container is a Kubernetes native way of making sure that the
> dependencies are localized before the main driver/executor containers are
> started. IMO, this guarantee is positive to have and it helps achieve
> separation of concerns. So IMO, I think the init-container is a valuable
> component and should be kept.
>
>
>
> On Tue, Jan 9, 2018 at 6:25 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
> I’d like to point out the output of “git show —stat” for that diff:
> 29 files changed, 130 insertions(+), 1560 deletions(-)
>
> +1 for that and generally for the idea of leveraging spark-submit.
>
> You can argue that executors downloading from
> external servers would be faster than downloading from the driver, but
> I’m not sure I’d agree - it can go both ways.
>
> On a tangentially related note, one of the main reasons
> spark-ec2[github.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_amplab_spark-2Dec2&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=q1Zla4QOVxYBCOe2Y8c9fm5ONO6hBMezCzXGoSL-714&s=SIHifGxsORvoKZq8pnnkNUm7JvzunHChW5Ovm65HAhw&e=>
> is so slow to launch clusters is that it distributes files like the Spark
> binaries to all the workers via the master. Because of that, the launch
> time scaled with the number of workers requested[issues.apache.org]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SPARK-2D5189&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=q1Zla4QOVxYBCOe2Y8c9fm5ONO6hBMezCzXGoSL-714&s=ybrBRVTuRMTPFSrPsyNrq4-HnjPZTFCdCIpsU7GM7CM&e=>
> .
>
> When I wrote Flintrock[github.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_nchammas_flintrock&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=q1Zla4QOVxYBCOe2Y8c9fm5ONO6hBMezCzXGoSL-714&s=prll8GmfnSeLdkiJI61HeEhSt_JIITBGES7-F7aAr94&e=>,
> I got a large improvement in launch time over spark-ec2 simply by having
> all the workers download the installation files in parallel from an
> external host (typically S3 or an Apache mirror). And launch time became
> largely independent of the cluster size.
>
> That may or may not say anything about the driver distributing application
> files vs. having init containers do it in parallel, but I’d be curious to
> hear more.
>
> Nick
>
>
>
>
>
> On Tue, Jan 9, 2018 at 9:08 PM Anirudh Ramanathan <ra...@google.com.invalid>
> wrote:
>
> We were running a change in our fork which was similar to this at one
> point early on. My biggest concerns off the top of my head with this change
> would be localization performance with large numbers of executors, and what
> we lose in terms of separation of concerns. Init containers are a standard
> construct in k8s for resource localization. Also how this approach affects
> the HDFS work would be interesting.
>
>
>
> +matt +kimoon
>
> Still thinking about the potential trade offs here. Adding Matt and Kimoon
> who would remember more about our reasoning at the time.
>
>
>
>
>
> On Jan 9, 2018 5:22 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
>
> Hello,
>
> Me again. I was playing some more with the kubernetes backend and the
> whole init container thing seemed unnecessary to me.
>
> Currently it's used to download remote jars and files, mount the
> volume into the driver / executor, and place those jars in the
> classpath / move the files to the working directory. This is all stuff
> that spark-submit already does without needing extra help.
>
> So I spent some time hacking stuff and removing the init container
> code, and launching the driver inside kubernetes using spark-submit
> (similar to how standalone and mesos cluster mode works):
>
> https://github.com/vanzin/spark/commit/k8s-no-init[github.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_vanzin_spark_commit_k8s-2Dno-2Dinit&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=q1Zla4QOVxYBCOe2Y8c9fm5ONO6hBMezCzXGoSL-714&s=EqsOxJaMMhHoyOA03szHpgjZffj7eX5sM9UY2B2972g&e=>
>
> I'd like to point out the output of "git show --stat" for that diff:
> 29 files changed, 130 insertions(+), 1560 deletions(-)
>
> You get massive code reuse by simply using spark-submit. The remote
> dependencies are downloaded in the driver, and the driver does the job
> of service them to executors.
>
> So I guess my question is: is there any advantage in using an init
> container?
>
> The current init container code can download stuff in parallel, but
> that's an easy improvement to make in spark-submit and that would
> benefit everybody. You can argue that executors downloading from
> external servers would be faster than downloading from the driver, but
> I'm not sure I'd agree - it can go both ways.
>
> Also the same idea could probably be applied to starting executors;
> Mesos starts executors using "spark-class" already, so doing that
> would both improve code sharing and potentially simplify some code in
> the k8s backend.
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
--
Anirudh Ramanathan
Re: Kubernetes: why use init containers?
Posted by Matt Cheah <mc...@palantir.com>.
A few reasons to prefer init-containers come to mind:
Firstly, if we used spark-submit from within the driver container, the executors wouldn’t receive the jars on their class loader until after the executor starts because the executor has to launch first before localizing resources. It is certainly possible to make the class loader work with the user’s jars here, as is the case with all the client mode implementations, but, it seems cleaner to have the classpath include the user’s jars at executor launch time instead of needing to reason about the classloading order.
We can also consider the idiomatic approach from the perspective of Kubernetes. Yinan touched on this already, but init-containers are traditionally meant to prepare the environment for the application that is to be run, which is exactly what we do here. This also makes it such that the localization process can be completely decoupled from the execution of the application itself. We can then for example detect the errors that happen on the resource localization layer, say when an HDFS cluster is down, before the application itself launches. The failure at the init-container stage is explicitly noted via the Kubernetes pod status API.
Finally, running spark-submit from the container would make the SparkSubmit code inadvertently allow running client mode Kubernetes applications as well. We’re not quite ready to support that. Even if we were, it’s not entirely intuitive for the cluster mode code path to depend on the client mode code path. This isn’t entirely without precedent though, as Mesos has a similar dependency.
Essentially the semantics seem neater and the contract is very explicit when using an init-container, even though the code does end up being more complex.
From: Yinan Li <li...@gmail.com>
Date: Tuesday, January 9, 2018 at 7:16 PM
To: Nicholas Chammas <ni...@gmail.com>
Cc: Anirudh Ramanathan <ra...@google.com.invalid>, Marcelo Vanzin <va...@cloudera.com>, Matt Cheah <mc...@palantir.com>, Kimoon Kim <ki...@pepperdata.com>, dev <de...@spark.apache.org>
Subject: Re: Kubernetes: why use init containers?
The init-container is required for use with the resource staging server (https://github.com/apache-spark-on-k8s/userdocs/blob/master/src/jekyll/running-on-kubernetes.md#resource-staging-server[github.com]). The resource staging server (RSS) is a spark-on-k8s component running in a Kubernetes cluster for staging submission client local dependencies to Spark pods. The init-container is responsible for downloading the dependencies from the RSS. We haven't upstream the RSS code yet, but this is a value add component for Spark on K8s as a way for users to use submission local dependencies without resorting to other mechanisms that are not immediately available on most Kubernetes clusters, e.g., HDFS. We do plan to upstream it in the 2.4 timeframe. Additionally, the init-container is a Kubernetes native way of making sure that the dependencies are localized before the main driver/executor containers are started. IMO, this guarantee is positive to have and it helps achieve separation of concerns. So IMO, I think the init-container is a valuable component and should be kept.
On Tue, Jan 9, 2018 at 6:25 PM, Nicholas Chammas <ni...@gmail.com> wrote:
I’d like to point out the output of “git show —stat” for that diff:
29 files changed, 130 insertions(+), 1560 deletions(-)
+1 for that and generally for the idea of leveraging spark-submit.
You can argue that executors downloading from
external servers would be faster than downloading from the driver, but
I’m not sure I’d agree - it can go both ways.
On a tangentially related note, one of the main reasons spark-ec2[github.com] is so slow to launch clusters is that it distributes files like the Spark binaries to all the workers via the master. Because of that, the launch time scaled with the number of workers requested[issues.apache.org].
When I wrote Flintrock[github.com], I got a large improvement in launch time over spark-ec2 simply by having all the workers download the installation files in parallel from an external host (typically S3 or an Apache mirror). And launch time became largely independent of the cluster size.
That may or may not say anything about the driver distributing application files vs. having init containers do it in parallel, but I’d be curious to hear more.
Nick
On Tue, Jan 9, 2018 at 9:08 PM Anirudh Ramanathan <ra...@google.com.invalid> wrote:
We were running a change in our fork which was similar to this at one point early on. My biggest concerns off the top of my head with this change would be localization performance with large numbers of executors, and what we lose in terms of separation of concerns. Init containers are a standard construct in k8s for resource localization. Also how this approach affects the HDFS work would be interesting.
+matt +kimoon
Still thinking about the potential trade offs here. Adding Matt and Kimoon who would remember more about our reasoning at the time.
On Jan 9, 2018 5:22 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
Hello,
Me again. I was playing some more with the kubernetes backend and the
whole init container thing seemed unnecessary to me.
Currently it's used to download remote jars and files, mount the
volume into the driver / executor, and place those jars in the
classpath / move the files to the working directory. This is all stuff
that spark-submit already does without needing extra help.
So I spent some time hacking stuff and removing the init container
code, and launching the driver inside kubernetes using spark-submit
(similar to how standalone and mesos cluster mode works):
https://github.com/vanzin/spark/commit/k8s-no-init[github.com]
I'd like to point out the output of "git show --stat" for that diff:
29 files changed, 130 insertions(+), 1560 deletions(-)
You get massive code reuse by simply using spark-submit. The remote
dependencies are downloaded in the driver, and the driver does the job
of service them to executors.
So I guess my question is: is there any advantage in using an init container?
The current init container code can download stuff in parallel, but
that's an easy improvement to make in spark-submit and that would
benefit everybody. You can argue that executors downloading from
external servers would be faster than downloading from the driver, but
I'm not sure I'd agree - it can go both ways.
Also the same idea could probably be applied to starting executors;
Mesos starts executors using "spark-class" already, so doing that
would both improve code sharing and potentially simplify some code in
the k8s backend.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Yinan Li <li...@gmail.com>.
The init-container is required for use with the resource staging server (
https://github.com/apache-spark-on-k8s/userdocs/blob/master/src/jekyll/running-on-kubernetes.md#resource-staging-server).
The resource staging server (RSS) is a spark-on-k8s component running in a
Kubernetes cluster for staging submission client local dependencies to
Spark pods. The init-container is responsible for downloading the
dependencies from the RSS. We haven't upstream the RSS code yet, but this
is a value add component for Spark on K8s as a way for users to use
submission local dependencies without resorting to other mechanisms that
are not immediately available on most Kubernetes clusters, e.g., HDFS. We
do plan to upstream it in the 2.4 timeframe. Additionally, the
init-container is a Kubernetes native way of making sure that the
dependencies are localized before the main driver/executor containers are
started. IMO, this guarantee is positive to have and it helps achieve
separation of concerns. So IMO, I think the init-container is a valuable
component and should be kept.
On Tue, Jan 9, 2018 at 6:25 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:
> I’d like to point out the output of “git show —stat” for that diff:
> 29 files changed, 130 insertions(+), 1560 deletions(-)
>
> +1 for that and generally for the idea of leveraging spark-submit.
>
> You can argue that executors downloading from
> external servers would be faster than downloading from the driver, but
> I’m not sure I’d agree - it can go both ways.
>
> On a tangentially related note, one of the main reasons spark-ec2
> <https://github.com/amplab/spark-ec2> is so slow to launch clusters is
> that it distributes files like the Spark binaries to all the workers via
> the master. Because of that, the launch time scaled with the number of
> workers requested <https://issues.apache.org/jira/browse/SPARK-5189>.
>
> When I wrote Flintrock <https://github.com/nchammas/flintrock>, I got a
> large improvement in launch time over spark-ec2 simply by having all the
> workers download the installation files in parallel from an external host
> (typically S3 or an Apache mirror). And launch time became largely
> independent of the cluster size.
>
> That may or may not say anything about the driver distributing application
> files vs. having init containers do it in parallel, but I’d be curious to
> hear more.
>
> Nick
>
>
> On Tue, Jan 9, 2018 at 9:08 PM Anirudh Ramanathan <ra...@google.com.invalid>
> wrote:
>
>> We were running a change in our fork which was similar to this at one
>> point early on. My biggest concerns off the top of my head with this change
>> would be localization performance with large numbers of executors, and what
>> we lose in terms of separation of concerns. Init containers are a standard
>> construct in k8s for resource localization. Also how this approach affects
>> the HDFS work would be interesting.
>>
>> +matt +kimoon
>> Still thinking about the potential trade offs here. Adding Matt and
>> Kimoon who would remember more about our reasoning at the time.
>>
>>
>> On Jan 9, 2018 5:22 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
>>
>>> Hello,
>>>
>>> Me again. I was playing some more with the kubernetes backend and the
>>> whole init container thing seemed unnecessary to me.
>>>
>>> Currently it's used to download remote jars and files, mount the
>>> volume into the driver / executor, and place those jars in the
>>> classpath / move the files to the working directory. This is all stuff
>>> that spark-submit already does without needing extra help.
>>>
>>> So I spent some time hacking stuff and removing the init container
>>> code, and launching the driver inside kubernetes using spark-submit
>>> (similar to how standalone and mesos cluster mode works):
>>>
>>> https://github.com/vanzin/spark/commit/k8s-no-init
>>>
>>> I'd like to point out the output of "git show --stat" for that diff:
>>> 29 files changed, 130 insertions(+), 1560 deletions(-)
>>>
>>> You get massive code reuse by simply using spark-submit. The remote
>>> dependencies are downloaded in the driver, and the driver does the job
>>> of service them to executors.
>>>
>>> So I guess my question is: is there any advantage in using an init
>>> container?
>>>
>>> The current init container code can download stuff in parallel, but
>>> that's an easy improvement to make in spark-submit and that would
>>> benefit everybody. You can argue that executors downloading from
>>> external servers would be faster than downloading from the driver, but
>>> I'm not sure I'd agree - it can go both ways.
>>>
>>> Also the same idea could probably be applied to starting executors;
>>> Mesos starts executors using "spark-class" already, so doing that
>>> would both improve code sharing and potentially simplify some code in
>>> the k8s backend.
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
Re: Kubernetes: why use init containers?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Tue, Jan 9, 2018 at 6:25 PM, Nicholas Chammas
<ni...@gmail.com> wrote:
> You can argue that executors downloading from
> external servers would be faster than downloading from the driver, but
> I’m not sure I’d agree - it can go both ways.
>
> On a tangentially related note, one of the main reasons spark-ec2 is so slow
> to launch clusters is that it distributes files like the Spark binaries to
> all the workers via the master. Because of that, the launch time scaled with
> the number of workers requested.
It's true that there are side effects. But there are two things that
can be used to mitigate this:
- k8s uses docker images. Users can create docker images with all the
dependencies their app needs, and submit the app using that image.
Spark doesn't have yet documentation on how to create these customized
images, but I'd rather invest time on that instead of supporting this
init container approach.
- The original spark-on-k8s spec mentioned a "dependency server"
approach which sounded like a more generic version of the YARN
distributed cache, which I hope can be a different way of mitigating
that issue. With that work, we could build this functionality into
spark-submit itself and have other backends also benefit.
In general, forcing the download of dependencies on every invocation
of an app should be avoided.
Anirudh:
> what we lose in terms of separation of concerns
1500 less lines of code lower my level of concern a lot more.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Kubernetes: why use init containers?
Posted by Nicholas Chammas <ni...@gmail.com>.
I’d like to point out the output of “git show —stat” for that diff:
29 files changed, 130 insertions(+), 1560 deletions(-)
+1 for that and generally for the idea of leveraging spark-submit.
You can argue that executors downloading from
external servers would be faster than downloading from the driver, but
I’m not sure I’d agree - it can go both ways.
On a tangentially related note, one of the main reasons spark-ec2
<https://github.com/amplab/spark-ec2> is so slow to launch clusters is that
it distributes files like the Spark binaries to all the workers via the
master. Because of that, the launch time scaled with the number of workers
requested <https://issues.apache.org/jira/browse/SPARK-5189>.
When I wrote Flintrock <https://github.com/nchammas/flintrock>, I got a
large improvement in launch time over spark-ec2 simply by having all the
workers download the installation files in parallel from an external host
(typically S3 or an Apache mirror). And launch time became largely
independent of the cluster size.
That may or may not say anything about the driver distributing application
files vs. having init containers do it in parallel, but I’d be curious to
hear more.
Nick
On Tue, Jan 9, 2018 at 9:08 PM Anirudh Ramanathan
<ra...@google.com.invalid> wrote:
> We were running a change in our fork which was similar to this at one
> point early on. My biggest concerns off the top of my head with this change
> would be localization performance with large numbers of executors, and what
> we lose in terms of separation of concerns. Init containers are a standard
> construct in k8s for resource localization. Also how this approach affects
> the HDFS work would be interesting.
>
> +matt +kimoon
> Still thinking about the potential trade offs here. Adding Matt and Kimoon
> who would remember more about our reasoning at the time.
>
>
> On Jan 9, 2018 5:22 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
>
>> Hello,
>>
>> Me again. I was playing some more with the kubernetes backend and the
>> whole init container thing seemed unnecessary to me.
>>
>> Currently it's used to download remote jars and files, mount the
>> volume into the driver / executor, and place those jars in the
>> classpath / move the files to the working directory. This is all stuff
>> that spark-submit already does without needing extra help.
>>
>> So I spent some time hacking stuff and removing the init container
>> code, and launching the driver inside kubernetes using spark-submit
>> (similar to how standalone and mesos cluster mode works):
>>
>> https://github.com/vanzin/spark/commit/k8s-no-init
>>
>> I'd like to point out the output of "git show --stat" for that diff:
>> 29 files changed, 130 insertions(+), 1560 deletions(-)
>>
>> You get massive code reuse by simply using spark-submit. The remote
>> dependencies are downloaded in the driver, and the driver does the job
>> of service them to executors.
>>
>> So I guess my question is: is there any advantage in using an init
>> container?
>>
>> The current init container code can download stuff in parallel, but
>> that's an easy improvement to make in spark-submit and that would
>> benefit everybody. You can argue that executors downloading from
>> external servers would be faster than downloading from the driver, but
>> I'm not sure I'd agree - it can go both ways.
>>
>> Also the same idea could probably be applied to starting executors;
>> Mesos starts executors using "spark-class" already, so doing that
>> would both improve code sharing and potentially simplify some code in
>> the k8s backend.
>>
>> --
>> Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: Kubernetes: why use init containers?
Posted by Anirudh Ramanathan <ra...@google.com.INVALID>.
We were running a change in our fork which was similar to this at one point
early on. My biggest concerns off the top of my head with this change would
be localization performance with large numbers of executors, and what we
lose in terms of separation of concerns. Init containers are a standard
construct in k8s for resource localization. Also how this approach affects
the HDFS work would be interesting.
+matt +kimoon
Still thinking about the potential trade offs here. Adding Matt and Kimoon
who would remember more about our reasoning at the time.
On Jan 9, 2018 5:22 PM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
> Hello,
>
> Me again. I was playing some more with the kubernetes backend and the
> whole init container thing seemed unnecessary to me.
>
> Currently it's used to download remote jars and files, mount the
> volume into the driver / executor, and place those jars in the
> classpath / move the files to the working directory. This is all stuff
> that spark-submit already does without needing extra help.
>
> So I spent some time hacking stuff and removing the init container
> code, and launching the driver inside kubernetes using spark-submit
> (similar to how standalone and mesos cluster mode works):
>
> https://github.com/vanzin/spark/commit/k8s-no-init
>
> I'd like to point out the output of "git show --stat" for that diff:
> 29 files changed, 130 insertions(+), 1560 deletions(-)
>
> You get massive code reuse by simply using spark-submit. The remote
> dependencies are downloaded in the driver, and the driver does the job
> of service them to executors.
>
> So I guess my question is: is there any advantage in using an init
> container?
>
> The current init container code can download stuff in parallel, but
> that's an easy improvement to make in spark-submit and that would
> benefit everybody. You can argue that executors downloading from
> external servers would be faster than downloading from the driver, but
> I'm not sure I'd agree - it can go both ways.
>
> Also the same idea could probably be applied to starting executors;
> Mesos starts executors using "spark-class" already, so doing that
> would both improve code sharing and potentially simplify some code in
> the k8s backend.
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>