You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@livy.apache.org by Malveeka Bhandari <ma...@gmail.com> on 2018/05/21 06:57:32 UTC

Fwd: Livy session

Hi everyone.

We are using Livy for our custom use case where a user can submit a batch
request. We send the user code in spark.jars configuration. But we are
having to make a new session everytime. Is there any way that there could
be one running session for a user and we could keep submitting new jobs but
with a new dependent jar each time.


Thanks

RE: Livy session

Posted by "Ghosh, Parthapratim " <pa...@citi.com>.

Hi,

                Not sure of your exact use case. I was also mentioning different user code in the same job’s codebase. My use case is different input while calling the same job multiple times.

Thanks and Regards
Partha Pratim Ghosh

From: Malveeka Bhandari [mailto:malveeka.bhandari@gmail.com]
Sent: Monday, May 21, 2018 3:43 AM
To: user@livy.incubator.apache.org
Subject: Re: Livy session

Hi,

Thanks for the idea. However, my use case is a little bit complicated. The job code remains constant, but the user code goes in as a dependent class. For e.g.  the job might be to query data, filter for some constraints and run a simulation (multiple transformations and actions) on the data. This job is constant. One of the functions inside the simulation would need the user code. So I would need to add the user code as a jar as opposed to a job class.

Is that possible in any way ?

Thanks !

On Mon, 21 May 2018 at 12:46 PM, Ghosh, Parthapratim <pa...@citi.com>> wrote:
Hi Malveeka,

                I am another user in the forum. However, your requirement may be met without creating a new jar. You may put the user code in a defined job as an instance level variable and use it everytime you use the client to submit a job. To call this, create a LivyClient and submit the job everytime with a new code as required, instantiating the job with the client code in overloaded constructor.

Thanks and Regards
Partha Pratim Ghosh

From: Malveeka Bhandari [mailto:malveeka.bhandari@gmail.com<ma...@gmail.com>]
Sent: Monday, May 21, 2018 2:58 AM
To: user@livy.incubator.apache.org<ma...@livy.incubator.apache.org>
Subject: Fwd: Livy session

Hi everyone.

We are using Livy for our custom use case where a user can submit a batch request. We send the user code in spark.jars configuration. But we are having to make a new session everytime. Is there any way that there could be one running session for a user and we could keep submitting new jobs but with a new dependent jar each time.

Thanks

Re: Livy session

Posted by Malveeka Bhandari <ma...@gmail.com>.

Hi,

Thanks for the idea. However, my use case is a little bit complicated. The
job code remains constant, but the user code goes in as a dependent class.
For e.g.  the job might be to query data, filter for some constraints and
run a simulation (multiple transformations and actions) on the data. This
job is constant. One of the functions inside the simulation would need the
user code. So I would need to add the user code as a jar as opposed to a
job class.

Is that possible in any way ?

Thanks !

On Mon, 21 May 2018 at 12:46 PM, Ghosh, Parthapratim <
parthapratim.ghosh@citi.com> wrote:

> Hi Malveeka,
>
>
>
>                 I am another user in the forum. However, your requirement
> may be met without creating a new jar. You may put the user code in a
> defined job as an instance level variable and use it everytime you use the
> client to submit a job. To call this, create a LivyClient and submit the
> job everytime with a new code as required, instantiating the job with the
> client code in overloaded constructor.
>
>
>
> *Thanks and Regards*
>
> *Partha Pratim Ghosh*
>
>
>
> *From:* Malveeka Bhandari [mailto:malveeka.bhandari@gmail.com]
> *Sent:* Monday, May 21, 2018 2:58 AM
> *To:* user@livy.incubator.apache.org
> *Subject:* Fwd: Livy session
>
>
>
>
>
>
>
> Hi everyone.
>
>
>
> We are using Livy for our custom use case where a user can submit a batch
> request. We send the user code in spark.jars configuration. But we are
> having to make a new session everytime. Is there any way that there could
> be one running session for a user and we could keep submitting new jobs but
> with a new dependent jar each time.
>
>
>
>
>
> Thanks
>

RE: Livy session

Posted by "Ghosh, Parthapratim " <pa...@citi.com>.

Hi Malveeka,

                I am another user in the forum. However, your requirement may be met without creating a new jar. You may put the user code in a defined job as an instance level variable and use it everytime you use the client to submit a job. To call this, create a LivyClient and submit the job everytime with a new code as required, instantiating the job with the client code in overloaded constructor.

Thanks and Regards
Partha Pratim Ghosh

From: Malveeka Bhandari [mailto:malveeka.bhandari@gmail.com]
Sent: Monday, May 21, 2018 2:58 AM
To: user@livy.incubator.apache.org
Subject: Fwd: Livy session

Hi everyone.

We are using Livy for our custom use case where a user can submit a batch request. We send the user code in spark.jars configuration. But we are having to make a new session everytime. Is there any way that there could be one running session for a user and we could keep submitting new jobs but with a new dependent jar each time.

Thanks