You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ashok Kumar <as...@yahoo.com.INVALID> on 2016/04/14 21:13:13 UTC

Spark replacing Hadoop

Hi,
I hear that some saying that Hadoop is getting old and out of date and will be replaced by Spark!
Does this make sense and if so how accurate is it?
Best

Re: Spark replacing Hadoop

Posted by Mich Talebzadeh <mi...@gmail.com>.
One can see from the responses that Big Data landscape is getting very
crowded with tools and there are dozens of alternatives offered. However,
as usual the laws of selection will gravitate towards solutions that are
scalable, reliable and more importantly cost effective.

To this end any commercial decision to acquire solutions as a technology
stack has to take into account the available skill sets in-house and the
stability of the products. I would concur with those that agree that a
smart solution will always require a good query engine, a mechanism to
organise the storage and the storage layer itself plus the resource
manager. The rests are icing on the cake.

To me Spark with Hive, HDFS and Yarn are winning combinations. Hadoop
encompasses HDFS and it is almost impossible to side step it without
finding a viable alternative as persistent storage. Also I take the point
that with already made investments in Hadoop, the exist barriers won't make
commercial sense. In other words one needs compelling arguments (besides
purely technical outlook) to replace Hadoop in this Financial climate that
technology dollars are a premium.

HTH

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 14 April 2016 at 21:35, Peyman Mohajerian <mo...@gmail.com> wrote:

> Cloud adds another dimension:
> The fact that in cloud compute and storage is decoupled, s3-emr or
> blob-hdisight, means in cloud Hadoop ends up being more of a compute engine
> and a lot of the governance, security features are irrelevant or less
> important because data at rest is out of Hadoop.
> Currently the biggest reason to run Spark in Hadoop is Yarn (in cloud),
>  but if you decide to use Mesos/Standalone then again you may not need
> Hadoop. Databrick adds another dimension to this in cloud which I won't
> comment on.
>
> But on-premise I think you can argue that HDFS is here to stay in many
> forms, e.g. Isilon, object stores and other storage types not just local
> disk. HDFS API actually works over Azure's Data Lake Store completely
> independent of Hadoop!
>
> On Thu, Apr 14, 2016 at 1:29 PM, Cody Koeninger <co...@koeninger.org>
> wrote:
>
>> I've been using spark for years and have (thankfully) been able to
>> avoid needing HDFS, aside from one contract where it was already in
>> use.
>>
>> At this point, many of the people I know would consider Kafka to be
>> more important than HDFS.
>>
>> On Thu, Apr 14, 2016 at 3:11 PM, Jörn Franke <jo...@gmail.com>
>> wrote:
>> > I do not think so. Hadoop provides an ecosystem in which you can deploy
>> > different engines, such as MR, HBase, TEZ, Spark, Flink, titandb, hive,
>> > solr... I observe also that commercial analytical tools use one or more
>> of
>> > these engines to execute their code in a distributed fashion. You  need
>> this
>> > flexibility to have an ecosystem suitable for your needs -especially In
>> the
>> > area of security. HDFS is one key element for the storage and locality.
>> > Spark itself cannot provide such a complete ecosystem but is part of
>> > ecosystems.
>> >
>> > On 14 Apr 2016, at 21:13, Ashok Kumar <as...@yahoo.com.INVALID>
>> wrote:
>> >
>> > Hi,
>> >
>> > I hear that some saying that Hadoop is getting old and out of date and
>> will
>> > be replaced by Spark!
>> >
>> > Does this make sense and if so how accurate is it?
>> >
>> > Best
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Re: Spark replacing Hadoop

Posted by Peyman Mohajerian <mo...@gmail.com>.
Cloud adds another dimension:
The fact that in cloud compute and storage is decoupled, s3-emr or
blob-hdisight, means in cloud Hadoop ends up being more of a compute engine
and a lot of the governance, security features are irrelevant or less
important because data at rest is out of Hadoop.
Currently the biggest reason to run Spark in Hadoop is Yarn (in cloud),
 but if you decide to use Mesos/Standalone then again you may not need
Hadoop. Databrick adds another dimension to this in cloud which I won't
comment on.

But on-premise I think you can argue that HDFS is here to stay in many
forms, e.g. Isilon, object stores and other storage types not just local
disk. HDFS API actually works over Azure's Data Lake Store completely
independent of Hadoop!

On Thu, Apr 14, 2016 at 1:29 PM, Cody Koeninger <co...@koeninger.org> wrote:

> I've been using spark for years and have (thankfully) been able to
> avoid needing HDFS, aside from one contract where it was already in
> use.
>
> At this point, many of the people I know would consider Kafka to be
> more important than HDFS.
>
> On Thu, Apr 14, 2016 at 3:11 PM, Jörn Franke <jo...@gmail.com> wrote:
> > I do not think so. Hadoop provides an ecosystem in which you can deploy
> > different engines, such as MR, HBase, TEZ, Spark, Flink, titandb, hive,
> > solr... I observe also that commercial analytical tools use one or more
> of
> > these engines to execute their code in a distributed fashion. You  need
> this
> > flexibility to have an ecosystem suitable for your needs -especially In
> the
> > area of security. HDFS is one key element for the storage and locality.
> > Spark itself cannot provide such a complete ecosystem but is part of
> > ecosystems.
> >
> > On 14 Apr 2016, at 21:13, Ashok Kumar <as...@yahoo.com.INVALID>
> wrote:
> >
> > Hi,
> >
> > I hear that some saying that Hadoop is getting old and out of date and
> will
> > be replaced by Spark!
> >
> > Does this make sense and if so how accurate is it?
> >
> > Best
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: Spark replacing Hadoop

Posted by Cody Koeninger <co...@koeninger.org>.
I've been using spark for years and have (thankfully) been able to
avoid needing HDFS, aside from one contract where it was already in
use.

At this point, many of the people I know would consider Kafka to be
more important than HDFS.

On Thu, Apr 14, 2016 at 3:11 PM, Jörn Franke <jo...@gmail.com> wrote:
> I do not think so. Hadoop provides an ecosystem in which you can deploy
> different engines, such as MR, HBase, TEZ, Spark, Flink, titandb, hive,
> solr... I observe also that commercial analytical tools use one or more of
> these engines to execute their code in a distributed fashion. You  need this
> flexibility to have an ecosystem suitable for your needs -especially In the
> area of security. HDFS is one key element for the storage and locality.
> Spark itself cannot provide such a complete ecosystem but is part of
> ecosystems.
>
> On 14 Apr 2016, at 21:13, Ashok Kumar <as...@yahoo.com.INVALID> wrote:
>
> Hi,
>
> I hear that some saying that Hadoop is getting old and out of date and will
> be replaced by Spark!
>
> Does this make sense and if so how accurate is it?
>
> Best

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark replacing Hadoop

Posted by Jörn Franke <jo...@gmail.com>.
I do not think so. Hadoop provides an ecosystem in which you can deploy different engines, such as MR, HBase, TEZ, Spark, Flink, titandb, hive, solr... I observe also that commercial analytical tools use one or more of these engines to execute their code in a distributed fashion. You  need this flexibility to have an ecosystem suitable for your needs -especially In the area of security. HDFS is one key element for the storage and locality. 
Spark itself cannot provide such a complete ecosystem but is part of ecosystems.

> On 14 Apr 2016, at 21:13, Ashok Kumar <as...@yahoo.com.INVALID> wrote:
> 
> Hi,
> 
> I hear that some saying that Hadoop is getting old and out of date and will be replaced by Spark!
> 
> Does this make sense and if so how accurate is it?
> 
> Best

Re: Spark replacing Hadoop

Posted by Felipe Gustavo <fe...@gmail.com>.
Hi Ashok,

In my opinion, we should look at Hadoop as a general purpose Framework that
supports multiple models and we should look at Spark as an alternative to
Hadoop MapReduce rather than a replacement to Hadoop ecosystem (for
instance, Spark is not replacing Zookeper, HDFS, etc)

Regards

On Thu, Apr 14, 2016 at 4:22 PM, Andy Davidson <
Andy@santacruzintegration.com> wrote:

> Hi Ashok
>
> In general if I was starting a new project and had not invested heavily in
> hadoop (i.e. Had a large staff that was trained on hadoop, had a lot of
> existing projects implemented on hadoop, …) I would probably start using
> spark. Its faster and easier to use
>
> Your mileage may vary
>
> Andy
>
> From: Ashok Kumar <as...@yahoo.com.INVALID>
> Reply-To: Ashok Kumar <as...@yahoo.com>
> Date: Thursday, April 14, 2016 at 12:13 PM
> To: "user @spark" <us...@spark.apache.org>
> Subject: Spark replacing Hadoop
>
> Hi,
>
> I hear that some saying that Hadoop is getting old and out of date and
> will be replaced by Spark!
>
> Does this make sense and if so how accurate is it?
>
> Best
>
>

Re: Spark replacing Hadoop

Posted by Sean Owen <so...@cloudera.com>.
Depends indeed on what you mean by "Hadoop". The core Hadoop project
is MapReduce, YARN and HDFS. MapReduce is still in use as a workhorse
but superseded by engines like Spark (or perhaps Flink).  (Tez maps
loosely to Spark Core really, and is not really a MapReduce
replacement.)

"Hadoop" can also be a catch-all term for projects typically used
together in conjunction with core Hadoop. That can be Spark, Kafka,
HBase, ZK, Solr, Parquet, Impala, Hive, etc.

If you mean the former -- mostly no, Spark needs a storage layer like
HDFS for persistent storage, and needs to integrate with a cluster
manager like YARN in order to share resources with other apps, but
replaces MapReduce.

If you mean the latter -- no, Spark is a big piece of the broader
picture and replaces several pieces (Mahout, maybe Crunch in some
ways, Giraph, arguably takes on some of Hive's workloads), but doesn't
replace most of them.

Really, there's no reason to expect that one project will do
everything. Core Hadoop mostly certainly wasn't enough to handle all
the "Hadoop" workloads today. It's a false choice. You can use Spark
*and* Hadoop-related projects and that's the best of all.

On Thu, Apr 14, 2016 at 8:40 PM, Mich Talebzadeh
<mi...@gmail.com> wrote:
> Hi,
>
> My two cents here.
>
> Hadoop as I understand has two components namely HDFS (Hadoop Distributed
> File System) and MapReduce.
>
> Whatever we use I still think we need to store data on HDFS (excluding
> standalones like MongoDB etc.). Now moving to MapReduce as the execution
> engine that is replaced by TEZ (basically MapReduce with DAG) or with Spark
> which uses in memory capabilities and DAG. MapReduce is the one moving
> sideways.
>
> To me Spark besides being versatile is a powerful tool. Remember tools are
> just tools, not solutions so we can discuss this all day. Effectively I
> would argue that with Spark as the front end tool with Hive and its
> organisation for metadata plus HDFS as the storage layer, you have all three
> components to create a powerful solution.
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
>
> On 14 April 2016 at 20:22, Andy Davidson <An...@santacruzintegration.com>
> wrote:
>>
>> Hi Ashok
>>
>> In general if I was starting a new project and had not invested heavily in
>> hadoop (i.e. Had a large staff that was trained on hadoop, had a lot of
>> existing projects implemented on hadoop, …) I would probably start using
>> spark. Its faster and easier to use
>>
>> Your mileage may vary
>>
>> Andy
>>
>> From: Ashok Kumar <as...@yahoo.com.INVALID>
>> Reply-To: Ashok Kumar <as...@yahoo.com>
>> Date: Thursday, April 14, 2016 at 12:13 PM
>> To: "user @spark" <us...@spark.apache.org>
>> Subject: Spark replacing Hadoop
>>
>> Hi,
>>
>> I hear that some saying that Hadoop is getting old and out of date and
>> will be replaced by Spark!
>>
>> Does this make sense and if so how accurate is it?
>>
>> Best
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark replacing Hadoop

Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,

My two cents here.

Hadoop as I understand has two components namely HDFS (Hadoop Distributed
File System) and MapReduce.

Whatever we use I still think we need to store data on HDFS (excluding
standalones like MongoDB etc.). Now moving to MapReduce as the execution
engine that is replaced by TEZ (basically MapReduce with DAG) or with Spark
which uses in memory capabilities and DAG. MapReduce is the one moving
sideways.

To me Spark besides being versatile is a powerful tool. Remember tools are
just tools, not solutions so we can discuss this all day. Effectively I
would argue that with Spark as the front end tool with Hive and its
organisation for metadata plus HDFS as the storage layer, you have all
three components to create a powerful solution.

HTH


Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 14 April 2016 at 20:22, Andy Davidson <An...@santacruzintegration.com>
wrote:

> Hi Ashok
>
> In general if I was starting a new project and had not invested heavily in
> hadoop (i.e. Had a large staff that was trained on hadoop, had a lot of
> existing projects implemented on hadoop, …) I would probably start using
> spark. Its faster and easier to use
>
> Your mileage may vary
>
> Andy
>
> From: Ashok Kumar <as...@yahoo.com.INVALID>
> Reply-To: Ashok Kumar <as...@yahoo.com>
> Date: Thursday, April 14, 2016 at 12:13 PM
> To: "user @spark" <us...@spark.apache.org>
> Subject: Spark replacing Hadoop
>
> Hi,
>
> I hear that some saying that Hadoop is getting old and out of date and
> will be replaced by Spark!
>
> Does this make sense and if so how accurate is it?
>
> Best
>
>

Re: Spark replacing Hadoop

Posted by Andy Davidson <An...@SantaCruzIntegration.com>.
Hi Ashok

In general if I was starting a new project and had not invested heavily in
hadoop (i.e. Had a large staff that was trained on hadoop, had a lot of
existing projects implemented on hadoop, Š) I would probably start using
spark. Its faster and easier to use

Your mileage may vary

Andy

From:  Ashok Kumar <as...@yahoo.com.INVALID>
Reply-To:  Ashok Kumar <as...@yahoo.com>
Date:  Thursday, April 14, 2016 at 12:13 PM
To:  "user @spark" <us...@spark.apache.org>
Subject:  Spark replacing Hadoop

> Hi,
> 
> I hear that some saying that Hadoop is getting old and out of date and will be
> replaced by Spark!
> 
> Does this make sense and if so how accurate is it?
> 
> Best



Re: Spark replacing Hadoop

Posted by Arunkumar Chandrasekar <ar...@outlook.com>.
Hello,

I would stand in side of Spark. Spark provides numerous add-ons like Spark SQL, Spark MLIB that are possibly something hard to set it up with Map Reduce. 

Thank You.



> On Apr 15, 2016, at 1:16 AM, Ashok Kumar <as...@yahoo.com.INVALID> wrote:
> 
> Hello,
> 
> Well, Sounds like Andy is implying that Spark can replace Hadoop whereas Mich still believes that HDFS is a keeper?
> 
> thanks
> 
> 
> 
> 
> On Thursday, 14 April 2016, 20:40, David Newberger <da...@wandcorp.com> wrote:
> 
> 
> Can we assume your question is “Will Spark replace Hadoop MapReduce?” or do you literally mean replacing the whole of Hadoop?
>  
> David
>  
> From: Ashok Kumar [mailto:ashok34668@yahoo.com.INVALID] 
> Sent: Thursday, April 14, 2016 2:13 PM
> To: User
> Subject: Spark replacing Hadoop
>  
> Hi,
>  
> I hear that some saying that Hadoop is getting old and out of date and will be replaced by Spark!
>  
> Does this make sense and if so how accurate is it?
>  
> Best
> 
> 

Re: Spark replacing Hadoop

Posted by Ashok Kumar <as...@yahoo.com.INVALID>.
Hello,
Well, Sounds like Andy is implying that Spark can replace Hadoop whereas Mich still believes that HDFS is a keeper?
thanks

 

    On Thursday, 14 April 2016, 20:40, David Newberger <da...@wandcorp.com> wrote:
 

 #yiv4514430231 #yiv4514430231 -- _filtered #yiv4514430231 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv4514430231 {font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;}#yiv4514430231 #yiv4514430231 p.yiv4514430231MsoNormal, #yiv4514430231 li.yiv4514430231MsoNormal, #yiv4514430231 div.yiv4514430231MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv4514430231 a:link, #yiv4514430231 span.yiv4514430231MsoHyperlink {color:blue;text-decoration:underline;}#yiv4514430231 a:visited, #yiv4514430231 span.yiv4514430231MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv4514430231 span.yiv4514430231EmailStyle17 {color:#1F497D;}#yiv4514430231 .yiv4514430231MsoChpDefault {font-size:10.0pt;} _filtered #yiv4514430231 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv4514430231 div.yiv4514430231WordSection1 {}#yiv4514430231 Can we assume your question is “Will Spark replace Hadoop MapReduce?” or do you literally mean replacing the whole of Hadoop?    David    From: Ashok Kumar [mailto:ashok34668@yahoo.com.INVALID]
Sent: Thursday, April 14, 2016 2:13 PM
To: User
Subject: Spark replacing Hadoop    Hi,    I hear that some saying that Hadoop is getting old and out of date and will be replaced by Spark!    Does this make sense and if so how accurate is it?    Best 

  

RE: Spark replacing Hadoop

Posted by David Newberger <da...@wandcorp.com>.
Can we assume your question is “Will Spark replace Hadoop MapReduce?” or do you literally mean replacing the whole of Hadoop?

David

From: Ashok Kumar [mailto:ashok34668@yahoo.com.INVALID]
Sent: Thursday, April 14, 2016 2:13 PM
To: User
Subject: Spark replacing Hadoop

Hi,

I hear that some saying that Hadoop is getting old and out of date and will be replaced by Spark!

Does this make sense and if so how accurate is it?

Best