You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Fabian Böhnlein <fa...@gmail.com> on 2017/03/28 12:24:27 UTC

Spark Interpreter: Change default scheduler pool

Hi all,

how can I change (globally, for Zeppelin) the default scheduler pool which
SparkInterpreter submits jobs to. Currently all jobs go into the pool
'default' but I want them to go into the pool 'fair'.
We use "Per Note" and "scoped" processes for best resource sharing.

"spark.scheduler.pool"="fair" in Interpreter Settings does not work, should
it?

What works is
sc.setLocalProperty("spark.scheduler.pool","fair")
but it's required in every *note* (not just notebook) since it's on thread
level.

Is there a possibility to globally/per notebook set the 'fair' pool as the
default pool?

Zeppelin brings two (hardcoded?) sheduler pools 'default' and 'fair'.
Between them, the scheduling is FAIR. 'default' is FIFO, 'fair' is FAIR.

This is awesome and together with dynamicAllocation allows for super
flexible usage for multiple users but above behavior is a bit complicated.

Thanks,
Fabian

Re: Spark Interpreter: Change default scheduler pool

Posted by Fabian Böhnlein <fa...@gmail.com>.
Hi moon,

exactly, thanks for the pointer.

Added the issue: https://issues.apache.org/jira/browse/ZEPPELIN-2413

Best,
Fabian


On Tue, 28 Mar 2017 at 15:48 moon soo Lee <mo...@apache.org> wrote:

> Hi Fabian,
>
> Thanks for sharing the issue.
> SparkSqlInterpreter set scheduler to "fair" depends on interpreter
> property [1]. I think we can do the similar for SparkInterpreter.
> Do you mind file a new JIRA issue for it?
>
> Regards,
> moon
>
> [1]
> https://github.com/apache/zeppelin/blob/0e1964877654c56c72473ad07dac1de6f9646816/spark/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java#L98
>
>
> On Tue, Mar 28, 2017 at 5:24 AM Fabian Böhnlein <
> fabian.boehnlein@gmail.com> wrote:
>
>> Hi all,
>>
>> how can I change (globally, for Zeppelin) the default scheduler pool
>> which SparkInterpreter submits jobs to. Currently all jobs go into the pool
>> 'default' but I want them to go into the pool 'fair'.
>> We use "Per Note" and "scoped" processes for best resource sharing.
>>
>> "spark.scheduler.pool"="fair" in Interpreter Settings does not work,
>> should it?
>>
>> What works is
>> sc.setLocalProperty("spark.scheduler.pool","fair")
>> but it's required in every *note* (not just notebook) since it's on
>> thread level.
>>
>> Is there a possibility to globally/per notebook set the 'fair' pool as
>> the default pool?
>>
>> Zeppelin brings two (hardcoded?) sheduler pools 'default' and 'fair'.
>> Between them, the scheduling is FAIR. 'default' is FIFO, 'fair' is FAIR.
>>
>> This is awesome and together with dynamicAllocation allows for super
>> flexible usage for multiple users but above behavior is a bit complicated.
>>
>> Thanks,
>> Fabian
>>
>>
>>
>>

Re: Spark Interpreter: Change default scheduler pool

Posted by moon soo Lee <mo...@apache.org>.
Hi Fabian,

Thanks for sharing the issue.
SparkSqlInterpreter set scheduler to "fair" depends on interpreter property
[1]. I think we can do the similar for SparkInterpreter.
Do you mind file a new JIRA issue for it?

Regards,
moon

[1]
https://github.com/apache/zeppelin/blob/0e1964877654c56c72473ad07dac1de6f9646816/spark/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java#L98


On Tue, Mar 28, 2017 at 5:24 AM Fabian Böhnlein <fa...@gmail.com>
wrote:

> Hi all,
>
> how can I change (globally, for Zeppelin) the default scheduler pool which
> SparkInterpreter submits jobs to. Currently all jobs go into the pool
> 'default' but I want them to go into the pool 'fair'.
> We use "Per Note" and "scoped" processes for best resource sharing.
>
> "spark.scheduler.pool"="fair" in Interpreter Settings does not work,
> should it?
>
> What works is
> sc.setLocalProperty("spark.scheduler.pool","fair")
> but it's required in every *note* (not just notebook) since it's on
> thread level.
>
> Is there a possibility to globally/per notebook set the 'fair' pool as the
> default pool?
>
> Zeppelin brings two (hardcoded?) sheduler pools 'default' and 'fair'.
> Between them, the scheduling is FAIR. 'default' is FIFO, 'fair' is FAIR.
>
> This is awesome and together with dynamicAllocation allows for super
> flexible usage for multiple users but above behavior is a bit complicated.
>
> Thanks,
> Fabian
>
>
>
>