You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by jamal sasha <ja...@gmail.com> on 2013/11/18 06:38:44 UTC

interesting question on quora

I found this interesting question on quora.. and thought of sharing here.
https://www.quora.com/Apache-Hadoop/Will-spark-ever-overtake-hadoop
So.. is spark missing any capabilty?

Re: interesting question on quora

Posted by Koert Kuipers <ko...@tresata.com>.

the core of hadoop is currently hdfs + mapreduce. the more appropriate
question is if it will become hdfs + spark. so will spark overtake
mapreduce as the dominant computational engine? its a very serious
candidate for that i think. it can do many things mapreduce cannot do, and
has an awesome api.

it's missing a few things to truly replace mapreduce:
* handling data that does not fit in memory per key/reducer
* security support (integrate with hdfs authorization/authentication)
* scalability??? (has spark been tested on 1000 machines)

On Mon, Nov 18, 2013 at 12:38 AM, jamal sasha <ja...@gmail.com> wrote:

> I found this interesting question on quora.. and thought of sharing here.
> https://www.quora.com/Apache-Hadoop/Will-spark-ever-overtake-hadoop
> So.. is spark missing any capabilty?
>
>