You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by 林武康 <vb...@gmail.com> on 2014/02/21 08:16:28 UTC

Can spark-streaming work with spark-on-yarn mode?

hi all,
I am a very newbie of apache spark, recently I have tried spark on yarn, it works for batch process. Now we want to try streaming process using spark-streaming, and still, use yarn for resource scheduler as we want to manager all the resource of the cluster used for computing tasks in a unified way.  Can this works? 
any suggestions are welcome!
 

--------------------------------------------------------------------------------

Best Regards！

Re: Can spark-streaming work with spark-on-yarn mode?

Posted by Tathagata Das <ta...@gmail.com>.

Yes, Spark and Spark Streaming programs can be deployed on YARN. Here is
the documentation<http://spark.incubator.apache.org/docs/latest/running-on-yarn.html>
.

TD

On Thu, Feb 20, 2014 at 11:16 PM, 林武康 <vb...@gmail.com> wrote:

>  hi all,
> I am a very newbie of apache spark, recently I have tried spark on yarn,
> it works for batch process. Now we want to try streaming process using
> spark-streaming, and still, use yarn for resource scheduler as we want to
> manager all the resource of the cluster used for computing tasks in a
> unified way.  Can this works?
> any suggestions are welcome!
>
>
>
> --------------------------------------------------------------------------------
>
> Best Regards！
>