You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shuo Xiang <sh...@gmail.com> on 2014/06/20 22:30:52 UTC
Set the number/memory of workers under mesos
Hi, just wondering anybody knows how to set up the number of workers (and
the amount of memory) in mesos, while lauching spark-shell? I was trying to
edit conf/spark-env.sh and it looks like that the environment variables are
for YARN of standalone. Thanks!
Re: Set the number/memory of workers under mesos
Posted by Mayur Rustagi <ma...@gmail.com>.
You can do that after as well, it changes application wide settings for
subsequent task.
On 20 Jun 2014 17:05, "Shuo Xiang" <sh...@gmail.com> wrote:
> Hi Mayur,
> Are you referring to overriding the default sc in sparkshell? Is there
> any way to do that before running the shell?
>
>
> On Fri, Jun 20, 2014 at 1:40 PM, Mayur Rustagi <ma...@gmail.com>
> wrote:
>
>> You should be able to configure in spark context in Spark shell.
>> spark.cores.max & memory.
>> Regards
>> Mayur
>>
>> Mayur Rustagi
>> Ph: +1 (760) 203 3257
>> http://www.sigmoidanalytics.com
>> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>>
>>
>>
>> On Fri, Jun 20, 2014 at 4:30 PM, Shuo Xiang <sh...@gmail.com>
>> wrote:
>>
>>> Hi, just wondering anybody knows how to set up the number of workers
>>> (and the amount of memory) in mesos, while lauching spark-shell? I was
>>> trying to edit conf/spark-env.sh and it looks like that the environment
>>> variables are for YARN of standalone. Thanks!
>>>
>>>
>>>
>>>
>>
>
Re: Set the number/memory of workers under mesos
Posted by Shuo Xiang <sh...@gmail.com>.
Hi Mayur,
Are you referring to overriding the default sc in sparkshell? Is there
any way to do that before running the shell?
On Fri, Jun 20, 2014 at 1:40 PM, Mayur Rustagi <ma...@gmail.com>
wrote:
> You should be able to configure in spark context in Spark shell.
> spark.cores.max & memory.
> Regards
> Mayur
>
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>
>
>
> On Fri, Jun 20, 2014 at 4:30 PM, Shuo Xiang <sh...@gmail.com>
> wrote:
>
>> Hi, just wondering anybody knows how to set up the number of workers (and
>> the amount of memory) in mesos, while lauching spark-shell? I was trying to
>> edit conf/spark-env.sh and it looks like that the environment variables are
>> for YARN of standalone. Thanks!
>>
>>
>>
>>
>
Re: Set the number/memory of workers under mesos
Posted by Mayur Rustagi <ma...@gmail.com>.
You should be able to configure in spark context in Spark shell.
spark.cores.max & memory.
Regards
Mayur
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>
On Fri, Jun 20, 2014 at 4:30 PM, Shuo Xiang <sh...@gmail.com> wrote:
> Hi, just wondering anybody knows how to set up the number of workers (and
> the amount of memory) in mesos, while lauching spark-shell? I was trying to
> edit conf/spark-env.sh and it looks like that the environment variables are
> for YARN of standalone. Thanks!
>
>
>
>