You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by Wandong Wu <wu...@husky.neu.edu> on 2018/07/13 03:28:22 UTC

Some questions about RscDriver

Hi:

I have some questions about How Livy communicate with Spark, and I am not very familiar with Spark architecture. 

Within an interactive session, Livy communicate with Spark through RPC. According to some architecture diagrams, LivyServer has a RscClient, and Spark has a RscDriver. My understanding is that RscDriver is one of components belonging to Spark, and RscDriver has existed before the Livy project was created. Is that right? 

I want to know how LivyServer communicates with Spark cluster 


Yours
Wandong

Re: Some questions about RscDriver

Posted by Wandong Wu <wu...@husky.neu.edu>.
Hi:

	Thank you for your response. I want to know how this component communicate with each other. So, RscDriver communicate with Spark Driver through RPC? Right? 
	Is there a protocol between RscClient and RscDriver?


Thanks
Wandong

> On Jul 17, 2018, at 08:49, Meisam Fathi <me...@gmail.com> wrote:
> 
> 
> Within an interactive session, Livy communicate with Spark through RPC. According to some architecture diagrams, LivyServer has a RscClient, and Spark has a RscDriver. My understanding is that RscDriver is one of components belonging to Spark, and RscDriver has existed before the Livy project was created. Is that right? 
> 
> No. RscDriver and RscClient are both part of Livy. RscDriver starts the Spark Driver. RscDriver stars a Spark context, gets the statements from RscClient, runs them on SparkContext, and returns the results to RscClient.
>  
> 
> Thanks,
> Meisam

Re: Some questions about RscDriver

Posted by Meisam Fathi <me...@gmail.com>.
>
>
> Within an interactive session, Livy communicate with Spark through RPC.
> According to some architecture diagrams, LivyServer has a RscClient, and
> Spark has a RscDriver. My understanding is that RscDriver is one of
> components belonging to Spark, and RscDriver has existed before the Livy
> project was created. Is that right?
>
> No. RscDriver and RscClient are both part of Livy. RscDriver starts the
Spark Driver. RscDriver stars a Spark context, gets the statements from
RscClient, runs them on SparkContext, and returns the results to RscClient.


Thanks,
Meisam

>