You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Charles Allen <ch...@metamarkets.com> on 2017/04/03 21:16:39 UTC
Mesos checkpointing
As per https://issues.apache.org/jira/browse/SPARK-4899
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerUtils#createSchedulerDriver
allows
checkpointing, but
only org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler uses
it. Is there a reason for that?
Re: Mesos checkpointing
Posted by Michael Gummelt <mg...@mesosphere.io>.
Ah, then yea, checkpointing should solve your problem. Let's do that.
On Wed, May 24, 2017 at 11:19 AM, Charles Allen <
charles.allen@metamarkets.com> wrote:
> The issue on our side is we tend to roll out a bunch of agent updates at
> about the same time. So rolling an agent, then waiting for spark jobs to
> recover, then rolling another agent is not at all practical. It is a huge
> benefit if we can just update the agents in bulk (or even sequentially, but
> only waiting for the mesos agent to recover).
>
> On Wed, May 24, 2017 at 11:17 AM Michael Gummelt <mg...@mesosphere.io>
> wrote:
>
>> > We had investigated internally recently why restarting the mesos
>> agents failed the spark jobs (no real reason they should, right?) and came
>> across the data.
>>
>> Restarting the agent without checkpointing enabled will kill the
>> executor, but that still shouldn't cause the Spark job to fail, since Spark
>> jobs should tolerate executor failures.
>>
>> On Mon, Apr 3, 2017 at 2:26 PM, Timothy Chen <tn...@gmail.com> wrote:
>>
>>> Yes, adding the timeout config should be the only code change required.
>>>
>>> And just to clarify, this is for reconnecting with Mesos master (not
>>> agents) after failover.
>>>
>>> Tim
>>>
>>> On Mon, Apr 3, 2017 at 2:23 PM, Charles Allen
>>> <ch...@metamarkets.com> wrote:
>>> > We had investigated internally recently why restarting the mesos agents
>>> > failed the spark jobs (no real reason they should, right?) and came
>>> across
>>> > the data. The other conversation by Yu sparked trying to poke to get
>>> some of
>>> > the tickets updated to spread around any tribal knowledge that is
>>> floating
>>> > in the community.
>>> >
>>> > It sounds like the only thing keeping it from being enabled is a
>>> timeout
>>> > config and someone volunteering to do some testing?
>>> >
>>> >
>>> > On Mon, Apr 3, 2017 at 2:19 PM Timothy Chen <tn...@gmail.com> wrote:
>>> >>
>>> >> The only reason is that MesosClusterScheduler by design is long
>>> >> running so we really needed it to have failover configured correctly.
>>> >>
>>> >> I wanted to create a JIRA ticket to allow users to configure it for
>>> >> each Spark framework, but just didn't remember to do so.
>>> >>
>>> >> Per another question that came up in the mailing list, I believe we
>>> >> should add it as it's a fairly straight forward effort.
>>> >>
>>> >> Tim
>>> >>
>>> >> On Mon, Apr 3, 2017 at 2:16 PM, Charles Allen
>>> >> <ch...@metamarkets.com> wrote:
>>> >> > As per https://issues.apache.org/jira/browse/SPARK-4899
>>> >> >
>>> >> > org.apache.spark.scheduler.cluster.mesos.MesosSchedulerUtils#
>>> createSchedulerDriver
>>> >> > allows checkpointing, but only
>>> >> > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler
>>> uses it.
>>> >> > Is
>>> >> > there a reason for that?
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>
>>
>> --
>> Michael Gummelt
>> Software Engineer
>> Mesosphere
>>
>
--
Michael Gummelt
Software Engineer
Mesosphere
Re: Mesos checkpointing
Posted by Charles Allen <ch...@metamarkets.com>.
The issue on our side is we tend to roll out a bunch of agent updates at
about the same time. So rolling an agent, then waiting for spark jobs to
recover, then rolling another agent is not at all practical. It is a huge
benefit if we can just update the agents in bulk (or even sequentially, but
only waiting for the mesos agent to recover).
On Wed, May 24, 2017 at 11:17 AM Michael Gummelt <mg...@mesosphere.io>
wrote:
> > We had investigated internally recently why restarting the mesos agents
> failed the spark jobs (no real reason they should, right?) and came across
> the data.
>
> Restarting the agent without checkpointing enabled will kill the executor,
> but that still shouldn't cause the Spark job to fail, since Spark jobs
> should tolerate executor failures.
>
> On Mon, Apr 3, 2017 at 2:26 PM, Timothy Chen <tn...@gmail.com> wrote:
>
>> Yes, adding the timeout config should be the only code change required.
>>
>> And just to clarify, this is for reconnecting with Mesos master (not
>> agents) after failover.
>>
>> Tim
>>
>> On Mon, Apr 3, 2017 at 2:23 PM, Charles Allen
>> <ch...@metamarkets.com> wrote:
>> > We had investigated internally recently why restarting the mesos agents
>> > failed the spark jobs (no real reason they should, right?) and came
>> across
>> > the data. The other conversation by Yu sparked trying to poke to get
>> some of
>> > the tickets updated to spread around any tribal knowledge that is
>> floating
>> > in the community.
>> >
>> > It sounds like the only thing keeping it from being enabled is a timeout
>> > config and someone volunteering to do some testing?
>> >
>> >
>> > On Mon, Apr 3, 2017 at 2:19 PM Timothy Chen <tn...@gmail.com> wrote:
>> >>
>> >> The only reason is that MesosClusterScheduler by design is long
>> >> running so we really needed it to have failover configured correctly.
>> >>
>> >> I wanted to create a JIRA ticket to allow users to configure it for
>> >> each Spark framework, but just didn't remember to do so.
>> >>
>> >> Per another question that came up in the mailing list, I believe we
>> >> should add it as it's a fairly straight forward effort.
>> >>
>> >> Tim
>> >>
>> >> On Mon, Apr 3, 2017 at 2:16 PM, Charles Allen
>> >> <ch...@metamarkets.com> wrote:
>> >> > As per https://issues.apache.org/jira/browse/SPARK-4899
>> >> >
>> >> >
>> org.apache.spark.scheduler.cluster.mesos.MesosSchedulerUtils#createSchedulerDriver
>> >> > allows checkpointing, but only
>> >> > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler uses
>> it.
>> >> > Is
>> >> > there a reason for that?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere
>
Re: Mesos checkpointing
Posted by Michael Gummelt <mg...@mesosphere.io>.
> We had investigated internally recently why restarting the mesos agents
failed the spark jobs (no real reason they should, right?) and came across
the data.
Restarting the agent without checkpointing enabled will kill the executor,
but that still shouldn't cause the Spark job to fail, since Spark jobs
should tolerate executor failures.
On Mon, Apr 3, 2017 at 2:26 PM, Timothy Chen <tn...@gmail.com> wrote:
> Yes, adding the timeout config should be the only code change required.
>
> And just to clarify, this is for reconnecting with Mesos master (not
> agents) after failover.
>
> Tim
>
> On Mon, Apr 3, 2017 at 2:23 PM, Charles Allen
> <ch...@metamarkets.com> wrote:
> > We had investigated internally recently why restarting the mesos agents
> > failed the spark jobs (no real reason they should, right?) and came
> across
> > the data. The other conversation by Yu sparked trying to poke to get
> some of
> > the tickets updated to spread around any tribal knowledge that is
> floating
> > in the community.
> >
> > It sounds like the only thing keeping it from being enabled is a timeout
> > config and someone volunteering to do some testing?
> >
> >
> > On Mon, Apr 3, 2017 at 2:19 PM Timothy Chen <tn...@gmail.com> wrote:
> >>
> >> The only reason is that MesosClusterScheduler by design is long
> >> running so we really needed it to have failover configured correctly.
> >>
> >> I wanted to create a JIRA ticket to allow users to configure it for
> >> each Spark framework, but just didn't remember to do so.
> >>
> >> Per another question that came up in the mailing list, I believe we
> >> should add it as it's a fairly straight forward effort.
> >>
> >> Tim
> >>
> >> On Mon, Apr 3, 2017 at 2:16 PM, Charles Allen
> >> <ch...@metamarkets.com> wrote:
> >> > As per https://issues.apache.org/jira/browse/SPARK-4899
> >> >
> >> > org.apache.spark.scheduler.cluster.mesos.MesosSchedulerUtils#
> createSchedulerDriver
> >> > allows checkpointing, but only
> >> > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler uses
> it.
> >> > Is
> >> > there a reason for that?
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
Michael Gummelt
Software Engineer
Mesosphere
Re: Mesos checkpointing
Posted by Timothy Chen <tn...@gmail.com>.
Yes, adding the timeout config should be the only code change required.
And just to clarify, this is for reconnecting with Mesos master (not
agents) after failover.
Tim
On Mon, Apr 3, 2017 at 2:23 PM, Charles Allen
<ch...@metamarkets.com> wrote:
> We had investigated internally recently why restarting the mesos agents
> failed the spark jobs (no real reason they should, right?) and came across
> the data. The other conversation by Yu sparked trying to poke to get some of
> the tickets updated to spread around any tribal knowledge that is floating
> in the community.
>
> It sounds like the only thing keeping it from being enabled is a timeout
> config and someone volunteering to do some testing?
>
>
> On Mon, Apr 3, 2017 at 2:19 PM Timothy Chen <tn...@gmail.com> wrote:
>>
>> The only reason is that MesosClusterScheduler by design is long
>> running so we really needed it to have failover configured correctly.
>>
>> I wanted to create a JIRA ticket to allow users to configure it for
>> each Spark framework, but just didn't remember to do so.
>>
>> Per another question that came up in the mailing list, I believe we
>> should add it as it's a fairly straight forward effort.
>>
>> Tim
>>
>> On Mon, Apr 3, 2017 at 2:16 PM, Charles Allen
>> <ch...@metamarkets.com> wrote:
>> > As per https://issues.apache.org/jira/browse/SPARK-4899
>> >
>> > org.apache.spark.scheduler.cluster.mesos.MesosSchedulerUtils#createSchedulerDriver
>> > allows checkpointing, but only
>> > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler uses it.
>> > Is
>> > there a reason for that?
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Mesos checkpointing
Posted by Charles Allen <ch...@metamarkets.com>.
We had investigated internally recently why restarting the mesos agents
failed the spark jobs (no real reason they should, right?) and came across
the data. The other conversation by Yu sparked trying to poke to get some
of the tickets updated to spread around any tribal knowledge that is
floating in the community.
It sounds like the only thing keeping it from being enabled is a timeout
config and someone volunteering to do some testing?
On Mon, Apr 3, 2017 at 2:19 PM Timothy Chen <tn...@gmail.com> wrote:
> The only reason is that MesosClusterScheduler by design is long
> running so we really needed it to have failover configured correctly.
>
> I wanted to create a JIRA ticket to allow users to configure it for
> each Spark framework, but just didn't remember to do so.
>
> Per another question that came up in the mailing list, I believe we
> should add it as it's a fairly straight forward effort.
>
> Tim
>
> On Mon, Apr 3, 2017 at 2:16 PM, Charles Allen
> <ch...@metamarkets.com> wrote:
> > As per https://issues.apache.org/jira/browse/SPARK-4899
> >
> org.apache.spark.scheduler.cluster.mesos.MesosSchedulerUtils#createSchedulerDriver
> > allows checkpointing, but only
> > org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler uses it.
> Is
> > there a reason for that?
>
Re: Mesos checkpointing
Posted by Timothy Chen <tn...@gmail.com>.
The only reason is that MesosClusterScheduler by design is long
running so we really needed it to have failover configured correctly.
I wanted to create a JIRA ticket to allow users to configure it for
each Spark framework, but just didn't remember to do so.
Per another question that came up in the mailing list, I believe we
should add it as it's a fairly straight forward effort.
Tim
On Mon, Apr 3, 2017 at 2:16 PM, Charles Allen
<ch...@metamarkets.com> wrote:
> As per https://issues.apache.org/jira/browse/SPARK-4899
> org.apache.spark.scheduler.cluster.mesos.MesosSchedulerUtils#createSchedulerDriver
> allows checkpointing, but only
> org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler uses it. Is
> there a reason for that?
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org