You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by cs user <ac...@gmail.com> on 2015/11/27 08:36:18 UTC
Spark on yarn vs spark standalone
Hi All,
Apologies if this question has been asked before. I'd like to know if there
are any downsides to running spark over yarn with the --master yarn-cluster
option vs having a separate spark standalone cluster to execute jobs?
We're looking at installing a hdfs/hadoop cluster with Ambari and
submitting jobs to the cluster using yarn, or having an Ambari cluster and
a separate standalone spark cluster, which will run the spark jobs on data
within hdfs.
With yarn, will we still get all the benefits of spark?
Will it be possible to process streaming data?
Many thanks in advance for any responses.
Cheers!
Re: Spark on yarn vs spark standalone
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Mark,
I said I've only managed to develop a limited understanding of how
Spark works in the different deploy modes ;-)
But somehow I thought that cluster in spark standalone is not
supported. I think I've seen a JIRA with a change quite recently where
it was said or something similar. Can't find it now :(
Pozdrawiam,
Jacek
--
Jacek Laskowski | https://medium.com/@jaceklaskowski/ |
http://blog.jaceklaskowski.pl
Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
Follow me at https://twitter.com/jaceklaskowski
Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
On Mon, Nov 30, 2015 at 6:58 PM, Mark Hamstra <ma...@clearstorydata.com> wrote:
> Standalone mode also supports running the driver on a cluster node. See
> "cluster" mode in
> http://spark.apache.org/docs/latest/spark-standalone.html#launching-spark-applications
> . Also,
> http://spark.apache.org/docs/latest/spark-standalone.html#high-availability
>
> On Mon, Nov 30, 2015 at 9:47 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>>
>> Hi,
>>
>> My understanding of Spark on YARN and even Spark in general is very
>> limited so keep that in mind.
>>
>> I'm not sure why you compare yarn-cluster and spark standalone? In
>> yarn-cluster a driver runs on a node in the YARN cluster while spark
>> standalone keeps the driver on the machine you launched a Spark
>> application. Also, YARN cluster supports retrying applications while
>> standalone doesn't. There's also support for rack locality preference
>> (but dunno if that's used and where in Spark).
>>
>> My limited understanding suggests me to use Spark on YARN if you're
>> considering to use Hadoop/HDFS and submitting jobs using YARN.
>> Standalone's an entry option where throwing in YARN could kill
>> introducing Spark to organizations without Hadoop YARN.
>>
>> Just my two cents.
>>
>> Pozdrawiam,
>> Jacek
>>
>> --
>> Jacek Laskowski | https://medium.com/@jaceklaskowski/ |
>> http://blog.jaceklaskowski.pl
>> Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
>> Follow me at https://twitter.com/jaceklaskowski
>> Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
>>
>>
>> On Fri, Nov 27, 2015 at 8:36 AM, cs user <ac...@gmail.com> wrote:
>> > Hi All,
>> >
>> > Apologies if this question has been asked before. I'd like to know if
>> > there
>> > are any downsides to running spark over yarn with the --master
>> > yarn-cluster
>> > option vs having a separate spark standalone cluster to execute jobs?
>> >
>> > We're looking at installing a hdfs/hadoop cluster with Ambari and
>> > submitting
>> > jobs to the cluster using yarn, or having an Ambari cluster and a
>> > separate
>> > standalone spark cluster, which will run the spark jobs on data within
>> > hdfs.
>> >
>> > With yarn, will we still get all the benefits of spark?
>> >
>> > Will it be possible to process streaming data?
>> >
>> > Many thanks in advance for any responses.
>> >
>> > Cheers!
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark on yarn vs spark standalone
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Standalone mode also supports running the driver on a cluster node. See
"cluster" mode in
http://spark.apache.org/docs/latest/spark-standalone.html#launching-spark-applications
. Also,
http://spark.apache.org/docs/latest/spark-standalone.html#high-availability
On Mon, Nov 30, 2015 at 9:47 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> My understanding of Spark on YARN and even Spark in general is very
> limited so keep that in mind.
>
> I'm not sure why you compare yarn-cluster and spark standalone? In
> yarn-cluster a driver runs on a node in the YARN cluster while spark
> standalone keeps the driver on the machine you launched a Spark
> application. Also, YARN cluster supports retrying applications while
> standalone doesn't. There's also support for rack locality preference
> (but dunno if that's used and where in Spark).
>
> My limited understanding suggests me to use Spark on YARN if you're
> considering to use Hadoop/HDFS and submitting jobs using YARN.
> Standalone's an entry option where throwing in YARN could kill
> introducing Spark to organizations without Hadoop YARN.
>
> Just my two cents.
>
> Pozdrawiam,
> Jacek
>
> --
> Jacek Laskowski | https://medium.com/@jaceklaskowski/ |
> http://blog.jaceklaskowski.pl
> Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
> Follow me at https://twitter.com/jaceklaskowski
> Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
>
>
> On Fri, Nov 27, 2015 at 8:36 AM, cs user <ac...@gmail.com> wrote:
> > Hi All,
> >
> > Apologies if this question has been asked before. I'd like to know if
> there
> > are any downsides to running spark over yarn with the --master
> yarn-cluster
> > option vs having a separate spark standalone cluster to execute jobs?
> >
> > We're looking at installing a hdfs/hadoop cluster with Ambari and
> submitting
> > jobs to the cluster using yarn, or having an Ambari cluster and a
> separate
> > standalone spark cluster, which will run the spark jobs on data within
> hdfs.
> >
> > With yarn, will we still get all the benefits of spark?
> >
> > Will it be possible to process streaming data?
> >
> > Many thanks in advance for any responses.
> >
> > Cheers!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark on yarn vs spark standalone
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
My understanding of Spark on YARN and even Spark in general is very
limited so keep that in mind.
I'm not sure why you compare yarn-cluster and spark standalone? In
yarn-cluster a driver runs on a node in the YARN cluster while spark
standalone keeps the driver on the machine you launched a Spark
application. Also, YARN cluster supports retrying applications while
standalone doesn't. There's also support for rack locality preference
(but dunno if that's used and where in Spark).
My limited understanding suggests me to use Spark on YARN if you're
considering to use Hadoop/HDFS and submitting jobs using YARN.
Standalone's an entry option where throwing in YARN could kill
introducing Spark to organizations without Hadoop YARN.
Just my two cents.
Pozdrawiam,
Jacek
--
Jacek Laskowski | https://medium.com/@jaceklaskowski/ |
http://blog.jaceklaskowski.pl
Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
Follow me at https://twitter.com/jaceklaskowski
Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
On Fri, Nov 27, 2015 at 8:36 AM, cs user <ac...@gmail.com> wrote:
> Hi All,
>
> Apologies if this question has been asked before. I'd like to know if there
> are any downsides to running spark over yarn with the --master yarn-cluster
> option vs having a separate spark standalone cluster to execute jobs?
>
> We're looking at installing a hdfs/hadoop cluster with Ambari and submitting
> jobs to the cluster using yarn, or having an Ambari cluster and a separate
> standalone spark cluster, which will run the spark jobs on data within hdfs.
>
> With yarn, will we still get all the benefits of spark?
>
> Will it be possible to process streaming data?
>
> Many thanks in advance for any responses.
>
> Cheers!
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark on yarn vs spark standalone
Posted by Jeff Zhang <zj...@gmail.com>.
If your cluster is a dedicated spark cluster (only running spark job, no
other jobs like hive/pig/mr), then spark standalone would be fine.
Otherwise I think yarn would be a better option.
On Fri, Nov 27, 2015 at 3:36 PM, cs user <ac...@gmail.com> wrote:
> Hi All,
>
> Apologies if this question has been asked before. I'd like to know if
> there are any downsides to running spark over yarn with the --master
> yarn-cluster option vs having a separate spark standalone cluster to
> execute jobs?
>
> We're looking at installing a hdfs/hadoop cluster with Ambari and
> submitting jobs to the cluster using yarn, or having an Ambari cluster and
> a separate standalone spark cluster, which will run the spark jobs on data
> within hdfs.
>
> With yarn, will we still get all the benefits of spark?
>
> Will it be possible to process streaming data?
>
> Many thanks in advance for any responses.
>
> Cheers!
>
--
Best Regards
Jeff Zhang