Re: Spark Interpreter: Change default scheduler pool

[Fabian Böhnlein <fa...@gmail.com>]

Hi moon,

exactly, thanks for the pointer.

Added the issue: https://issues.apache.org/jira/browse/ZEPPELIN-2413

Best,
Fabian


On Tue, 28 Mar 2017 at 15:48 moon soo Lee <mo...@apache.org> wrote:

> Hi Fabian,
>
> Thanks for sharing the issue.
> SparkSqlInterpreter set scheduler to "fair" depends on interpreter
> property [1]. I think we can do the similar for SparkInterpreter.
> Do you mind file a new JIRA issue for it?
>
> Regards,
> moon
>
> [1]
> https://github.com/apache/zeppelin/blob/0e1964877654c56c72473ad07dac1de6f9646816/spark/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java#L98
>
>
> On Tue, Mar 28, 2017 at 5:24 AM Fabian Böhnlein <
> fabian.boehnlein@gmail.com> wrote:
>
>> Hi all,
>>
>> how can I change (globally, for Zeppelin) the default scheduler pool
>> which SparkInterpreter submits jobs to. Currently all jobs go into the pool
>> 'default' but I want them to go into the pool 'fair'.
>> We use "Per Note" and "scoped" processes for best resource sharing.
>>
>> "spark.scheduler.pool"="fair" in Interpreter Settings does not work,
>> should it?
>>
>> What works is
>> sc.setLocalProperty("spark.scheduler.pool","fair")
>> but it's required in every *note* (not just notebook) since it's on
>> thread level.
>>
>> Is there a possibility to globally/per notebook set the 'fair' pool as
>> the default pool?
>>
>> Zeppelin brings two (hardcoded?) sheduler pools 'default' and 'fair'.
>> Between them, the scheduling is FAIR. 'default' is FIFO, 'fair' is FAIR.
>>
>> This is awesome and together with dynamicAllocation allows for super
>> flexible usage for multiple users but above behavior is a bit complicated.
>>
>> Thanks,
>> Fabian
>>
>>
>>
>>