How Livy maintains SparkSession under the cover?

[Jobinesh Purushothaman <jo...@gmail.com>]

Hi
I am new to Apache Livy and trying to understand the offerings better.

A couple of queries here:

1. I understand that Livy REST APIs let me pass the same session id, if I
need to execute the new request in the context of an
already existing SparkSession. Can someone help me to understand how
the above works under the hood?
2. We have a usease to invoke Spark directly from a microservice(direct
RPC) . Is it possible/feasible to pick certain classes from Apache Livy to
call Spark directly(i.e direct RPC, without going through REST API route)

Any help is appreciated. Thanks for your time !

Thanks
Jobinesh

Re: How Livy maintains SparkSession under the cover?

[Andras Beni <an...@cloudera.com>]

Hi Jobinesh,

1. When you start an interactive session, Livy submits a Spark application
whose driver acts as a server itself. Every time you send new statements to
the Livy REST server to execute in the session, it sends the statements to
the server running in the driver. The driver then interprets and executes
them. You can see a schematic diagram on the projects page about this.
2. If you need something really simple, you could call Spark's spark-submit
script or SparkLauncher class directly. If you want something more, start
off with InteractiveSession
<https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala>
or
BatchSession to see if they bring you any closer to your goal. But I think
Livy was never meant to be a library, so it might be inconvenient to use
classes from Livy in your services. And in the end, you will probably build
features around your solution, that Livy already has. Not to mention that
most Livy code is not considered an API and can change between releases and
you will have a huge maintenance burden on your service team. I really
recommend using Livy through the REST API or through its Java/Scala
wrappers.

Hope this helps.
Best regards,
Andras

On Fri, Jun 26, 2020 at 12:11 AM Jobinesh Purushothaman <jo...@gmail.com>
wrote:

> Hi
> I am new to Apache Livy and trying to understand the offerings better.
>
> A couple of queries here:
>
> 1. I understand that Livy REST APIs let me pass the same session id, if I
> need to execute the new request in the context of an
> already existing SparkSession. Can someone help me to understand how
> the above works under the hood?
> 2. We have a usease to invoke Spark directly from a microservice(direct
> RPC) . Is it possible/feasible to pick certain classes from Apache Livy to
> call Spark directly(i.e direct RPC, without going through REST API route)
>
> Any help is appreciated. Thanks for your time !
>
> Thanks
> Jobinesh
>