Cluster submit mode - only supported on Yarn?

[Chris Schneider <ch...@christopher-schneider.com>]

We are starting to use Spark, but we don't have any existing infrastructure
related to big-data, so we decided to setup the standalone cluster, rather
than mess around with Yarn or Mesos.

But it appears like the driver program has to stay up on the client for the
full duration of the job ("client mode").

What is the simplest way to setup "cluster" submission mode, to allow our
client boxes to submit jobs and then move on with the other work they need
to do without keeping a potentially long running java process up?

Thanks,
Chris

Re: Cluster submit mode - only supported on Yarn?

[Chris Schneider <ch...@christopher-schneider.com>]

Thanks Steve, but my goal is to hopefully avoid installing yet another
component into my environment.  Yarn is cool, but wouldn't be used for
anything but Spark. We have no hadoop in our ecosystem (or HDFS).  Ideally
I'd avoid having to learn about yet another tool.


On Wed, Jul 23, 2014 at 11:12 AM, Steve Nunez <sn...@hortonworks.com>
wrote:

> I’m also in early stages of setting up long running Spark jobs. Easiest
> way I’ve found is to set up a cluster and submit the job via YARN. Then I
> can come back and check in on progress when I need to. Seems the trick is
> tuning the queue priority and YARN preemption to get the job to run in a
> reasonable amount of time without disrupting the other jobs.
>
> - SteveN
>
>
> From: Chris Schneider <ch...@christopher-schneider.com>
> Reply-To: <us...@spark.apache.org>
> Date: Wednesday, July 23, 2014 at 7:39
> To: <us...@spark.apache.org>
> Subject: Cluster submit mode - only supported on Yarn?
>
> We are starting to use Spark, but we don't have any existing
> infrastructure related to big-data, so we decided to setup the standalone
> cluster, rather than mess around with Yarn or Mesos.
>
> But it appears like the driver program has to stay up on the client for
> the full duration of the job ("client mode").
>
> What is the simplest way to setup "cluster" submission mode, to allow our
> client boxes to submit jobs and then move on with the other work they need
> to do without keeping a potentially long running java process up?
>
> Thanks,
> Chris
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Cluster submit mode - only supported on Yarn?

[Steve Nunez <sn...@hortonworks.com>]

I¹m also in early stages of setting up long running Spark jobs. Easiest way
I¹ve found is to set up a cluster and submit the job via YARN. Then I can
come back and check in on progress when I need to. Seems the trick is tuning
the queue priority and YARN preemption to get the job to run in a reasonable
amount of time without disrupting the other jobs.

- SteveN


From:  Chris Schneider <ch...@christopher-schneider.com>
Reply-To:  <us...@spark.apache.org>
Date:  Wednesday, July 23, 2014 at 7:39
To:  <us...@spark.apache.org>
Subject:  Cluster submit mode - only supported on Yarn?

We are starting to use Spark, but we don't have any existing infrastructure
related to big-data, so we decided to setup the standalone cluster, rather
than mess around with Yarn or Mesos.

But it appears like the driver program has to stay up on the client for the
full duration of the job ("client mode").

What is the simplest way to setup "cluster" submission mode, to allow our
client boxes to submit jobs and then move on with the other work they need
to do without keeping a potentially long running java process up?

Thanks,
Chris






-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.