Swapping of Spark Streaming job

[Ryan Chan <ry...@gmail.com>]

Hi,

Currently system like Storm only support kill-and-resubmit kind of job
swapping for a live system, which is not good as down time can be up to 10
minutes in our experience.

Is this issue better handled in Spark Streaming, such as with the help of
mesos?

Re: Swapping of Spark Streaming job

[Patrick Wendell <pw...@gmail.com>]

Hey Ryan,

I looked this up and apparently this is a storm term for replacing a
topology when the code changes.

The basic mechanics of doing this in Spark Streaming are the same as Storm.
You'll shut down the old streaming job and start up the new one. However,
this shouldn't take on the order of minutes. It can be resubmitted on the
order of seconds.

- Patrick


On Thu, Oct 24, 2013 at 10:22 PM, Patrick Wendell <pw...@gmail.com>wrote:

> Hey Ryan,
>
> Could you clarify what you mean by "swapping"?
>
> - Patrick
>
>
> On Thu, Oct 24, 2013 at 10:12 PM, Ryan Chan <ry...@gmail.com> wrote:
>
>> Hi,
>>
>> Currently system like Storm only support kill-and-resubmit kind of job
>> swapping for a live system, which is not good as down time can be up to 10
>> minutes in our experience.
>>
>> Is this issue better handled in Spark Streaming, such as with the help of
>> mesos?
>>
>>
>>
>

Re: Swapping of Spark Streaming job

[Patrick Wendell <pw...@gmail.com>]

Hey Ryan,

Could you clarify what you mean by "swapping"?

- Patrick


On Thu, Oct 24, 2013 at 10:12 PM, Ryan Chan <ry...@gmail.com> wrote:

> Hi,
>
> Currently system like Storm only support kill-and-resubmit kind of job
> swapping for a live system, which is not good as down time can be up to 10
> minutes in our experience.
>
> Is this issue better handled in Spark Streaming, such as with the help of
> mesos?
>
>
>