Reply: Any best practice for hardware configuration for the masterserver in standalone cluster mode?

[guxiaobo1982 <gu...@qq.com>]

Thanks for your reply, I am new hand at Spark, does driver mean the server where user applications are commit?






------------------ Original ------------------
Sender: "Mayur Rustagi"<ma...@gmail.com>;
Send time: Tuesday, Dec 31, 2013 9:55 PM
To: "user"<us...@spark.incubator.apache.org>; 

Subject: Re: Any best practice for hardware configuration for the masterserver in standalone cluster mode?



Master server needs to be a little beefy as the driver runs on it. We ran into some issues around scaling due to master servers. You can offload the drivers to workers or other machines then the master server can be smaller.  Regards
Mayur


Mayur Rustagi
Ph: +919632149971http://www.sigmoidanalytics.com
 https://twitter.com/mayur_rustagi





 

On Tue, Dec 31, 2013 at 6:48 PM, guxiaobo1982 <gu...@qq.com> wrote:
 Him


I read the following article regarding to hardware configurations for the worker servers in the standalone cluster mode, but what about the master server?


http://spark.incubator.apache.org/docs/latest/hardware-provisioning.html
 



Regards,


Xiaobo Gu

Re: Reply: Any best practice for hardware configuration for the masterserver in standalone cluster mode?

[Mayur Rustagi <ma...@gmail.com>]

Driver is the process that manages the execution across the cluster. So say
your application is a "sql query" then the system spawns a shark-cli-driver
that uses spark framework, hdfs etc to execute the query and deliver
result. All this happens automatically so you dont need to worry about it
as a user of spark/shark framework. Just go for a bigger machine with a
master.


Mayur Rustagi
Ph: +919632149971
h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
https://twitter.com/mayur_rustagi



On Tue, Dec 31, 2013 at 7:01 PM, guxiaobo1982 <gu...@qq.com> wrote:

> Thanks for your reply, I am new hand at Spark, does driver mean the server
> where user applications are commit?
>
>
>
> ------------------ Original ------------------
> *Sender:* "Mayur Rustagi"<ma...@gmail.com>;
> *Send time:* Tuesday, Dec 31, 2013 9:55 PM
> *To:* "user"<us...@spark.incubator.apache.org>;
> *Subject:* Re: Any best practice for hardware configuration for the
> masterserver in standalone cluster mode?
>
> Master server needs to be a little beefy as the driver runs on it. We ran
> into some issues around scaling due to master servers. You can offload the
> drivers to workers or other machines then the master server can be smaller.
> Regards
> Mayur
>
> Mayur Rustagi
> Ph: +919632149971
> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
> https://twitter.com/mayur_rustagi
>
>
>
> On Tue, Dec 31, 2013 at 6:48 PM, guxiaobo1982 <gu...@qq.com> wrote:
>
>> Him
>>
>> I read the following article regarding to hardware configurations for the
>> worker servers in the standalone cluster mode, but what about the master
>> server?
>>
>> http://spark.incubator.apache.org/docs/latest/hardware-provisioning.html
>>
>>
>> Regards,
>>
>> Xiaobo Gu
>>
>>
>