Running local and yarn using the same livy server

[Ravi Shankar <ra...@unifisoftware.com>]

Hey guys,
Is there a way to start different kind of spark sessions using the same
livy server ? For instance, i want my application to tell livy whether the
new session being requested should be started as a "local" session or a
"yarn" session on the cluster.

Ravi

Re: Running local and yarn using the same livy server

[Ryan Skraba <ry...@skraba.com>]

Hello!  We patched livy with this functionality -- to bypass the global
config to permit the "occasional" job to run in local mode.  We used an
attribute in the REST URL query string to control it.

It isn't very difficult to do, but (at the time) my impression was this
isn't a particularly _secure_ thing to allow.  It's pretty easy to
overwhelm Livy with may spark-local jobs, and if it's sitting on an edge
node serving many users... well, you probably don't want it to be easily
killed!  Our use case was particular, where we had a bit more control over
when and how jobs are scheduled before arriving at Livy.

Let me know if you think this is something that might be interesting to
push back into the community!

Best regards, Ryan



On Sat, Feb 1, 2020 at 9:13 AM Saisai Shao <sa...@gmail.com> wrote:

> I don't think current Livy support such behavior, the cluster manager
> specified in the conf file is a global configuration which affects all the
> created sessions.
>
> Thanks
> Saisai
>
> Ravi Shankar <ra...@unifisoftware.com> 于2020年1月28日周二 上午4:02写道：
>
>> Hey guys,
>> Is there a way to start different kind of spark sessions using the same
>> livy server ? For instance, i want my application to tell livy whether the
>> new session being requested should be started as a "local" session or a
>> "yarn" session on the cluster.
>>
>> Ravi
>>
>

Re: Running local and yarn using the same livy server

[Saisai Shao <sa...@gmail.com>]

I don't think current Livy support such behavior, the cluster manager
specified in the conf file is a global configuration which affects all the
created sessions.

Thanks
Saisai

Ravi Shankar <ra...@unifisoftware.com> 于2020年1月28日周二 上午4:02写道：

> Hey guys,
> Is there a way to start different kind of spark sessions using the same
> livy server ? For instance, i want my application to tell livy whether the
> new session being requested should be started as a "local" session or a
> "yarn" session on the cluster.
>
> Ravi
>