Spark as standalone or with Hadoop stack.

[Shiv Kandavelu <sh...@riversand.com>]

Hi All,

We currently have a Hadoop cluster having Yarn as the resource manager.
We are planning to use HBase as the data store due to the C-P aspects of the CAP Theorem.
We now want to do extensive data processing both stored data in HBase as well as Steam processing from online website / API
We now want to use both Spark/Mapreduce on an existing Hadoop cluster.

One of the recommendation we got was to use Spark Cluster as a standalone with Mesos as a resource manager on top of it to Monitor and scale. The reason for this recommendation is that Standalone Spark with Mesos is 100x faster than the Spark/Yarn/Hadoop combination. It was also mentioned that building on Spark/Mesos can help automatically add spark nodes on the fly for processing to scale. Also, it is easy to switch the bottom data stack HBASE to Cassandra or something else if we use Spark.

We are in the process of evaluating which stack will work best and with the knowledge we have, it is getting tough to pick one versus the other b/c of our inexperience in these platforms.

Can you help us understand the pros and cons of having Spark as a Standalone cluster Vs running on top of Hadoop stack?

Thanks!

Re: Spark as standalone or with Hadoop stack.

[Sean Owen <so...@cloudera.com>]

Might be better for another list, but, I suspect it's more than HBase
is simply much more integrated with YARN, and because it's run with
other services that are as well.

On Wed, Sep 23, 2015 at 12:02 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> That sentence caught my attention. Could you explain the reasons for
> not running HBase on Mesos, i.e. what makes Mesos inappropriate for
> HBase?

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark as standalone or with Hadoop stack.

[Ted Yu <yu...@gmail.com>]

HDFS on Mesos framework is still being developed.
What I said previously reflected current deployment practice.

Things may change in the future.

On Tue, Sep 22, 2015 at 4:02 PM, Jacek Laskowski <ja...@japila.pl> wrote:

> On Tue, Sep 22, 2015 at 10:03 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > To my knowledge, no one runs HBase on top of Mesos.
>
> Hi,
>
> That sentence caught my attention. Could you explain the reasons for
> not running HBase on Mesos, i.e. what makes Mesos inappropriate for
> HBase?
>
> Jacek
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: Spark as standalone or with Hadoop stack.

[Jacek Laskowski <ja...@japila.pl>]

On Tue, Sep 22, 2015 at 10:03 PM, Ted Yu <yu...@gmail.com> wrote:

> To my knowledge, no one runs HBase on top of Mesos.

Hi,

That sentence caught my attention. Could you explain the reasons for
not running HBase on Mesos, i.e. what makes Mesos inappropriate for
HBase?

Jacek

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark as standalone or with Hadoop stack.

[Ted Yu <yu...@gmail.com>]

bq. it's relatively harder to use it with HBase

I agree with Sean.
I work on HBase. To my knowledge, no one runs HBase on top of Mesos.

On Tue, Sep 22, 2015 at 12:31 PM, Sean Owen <so...@cloudera.com> wrote:

> Who told you Mesos would make Spark 100x faster? does it make sense
> that just the resource manager could make that kind of difference?
> This sounds entirely wrong, or, maybe a mishearing.
>
> I don't know if Mesos is somehow easier to use with Cassandra, but
> it's relatively harder to use it with HBase, HDFS, etc. You probably
> want to use the Hadoop resource manager, YARN, if using Hadoop-ish
> stack components.
>
> As for Spark, the YARN integration actually has some advantages at the
> moment, like dynamic allocation. I think the security story is more
> complete too (? not sure).
>
> On Tue, Sep 22, 2015 at 8:25 PM, Shiv Kandavelu
> <sh...@riversand.com> wrote:
> >
> >
> > Hi All,
> >
> >
> >
> > We currently have a Hadoop cluster having Yarn as the resource manager.
> >
> > We are planning to use HBase as the data store due to the C-P aspects of
> the
> > CAP Theorem.
> >
> > We now want to do extensive data processing both stored data in HBase as
> > well as Steam processing from online website / API
> >
> > We now want to use both Spark/Mapreduce on an existing Hadoop cluster.
> >
> >
> >
> > One of the recommendation we got was to use Spark Cluster as a standalone
> > with Mesos as a resource manager on top of it to Monitor and scale. The
> > reason for this recommendation is that Standalone Spark with Mesos is
> 100x
> > faster than the Spark/Yarn/Hadoop combination. It was also mentioned that
> > building on Spark/Mesos can help automatically add spark nodes on the fly
> > for processing to scale. Also, it is easy to switch the bottom data stack
> > HBASE to Cassandra or something else if we use Spark.
> >
> >
> >
> > We are in the process of evaluating which stack will work best and with
> the
> > knowledge we have, it is getting tough to pick one versus the other b/c
> of
> > our inexperience in these platforms.
> >
> >
> >
> > Can you help us understand the pros and cons of having Spark as a
> Standalone
> > cluster Vs running on top of Hadoop stack?
> >
> >
> >
> > Thanks!
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: Spark as standalone or with Hadoop stack.

[Sean Owen <so...@cloudera.com>]

Who told you Mesos would make Spark 100x faster? does it make sense
that just the resource manager could make that kind of difference?
This sounds entirely wrong, or, maybe a mishearing.

I don't know if Mesos is somehow easier to use with Cassandra, but
it's relatively harder to use it with HBase, HDFS, etc. You probably
want to use the Hadoop resource manager, YARN, if using Hadoop-ish
stack components.

As for Spark, the YARN integration actually has some advantages at the
moment, like dynamic allocation. I think the security story is more
complete too (? not sure).

On Tue, Sep 22, 2015 at 8:25 PM, Shiv Kandavelu
<sh...@riversand.com> wrote:
>
>
> Hi All,
>
>
>
> We currently have a Hadoop cluster having Yarn as the resource manager.
>
> We are planning to use HBase as the data store due to the C-P aspects of the
> CAP Theorem.
>
> We now want to do extensive data processing both stored data in HBase as
> well as Steam processing from online website / API
>
> We now want to use both Spark/Mapreduce on an existing Hadoop cluster.
>
>
>
> One of the recommendation we got was to use Spark Cluster as a standalone
> with Mesos as a resource manager on top of it to Monitor and scale. The
> reason for this recommendation is that Standalone Spark with Mesos is 100x
> faster than the Spark/Yarn/Hadoop combination. It was also mentioned that
> building on Spark/Mesos can help automatically add spark nodes on the fly
> for processing to scale. Also, it is easy to switch the bottom data stack
> HBASE to Cassandra or something else if we use Spark.
>
>
>
> We are in the process of evaluating which stack will work best and with the
> knowledge we have, it is getting tough to pick one versus the other b/c of
> our inexperience in these platforms.
>
>
>
> Can you help us understand the pros and cons of having Spark as a Standalone
> cluster Vs running on top of Hadoop stack?
>
>
>
> Thanks!
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org