[jira] [Commented] (LIVY-471) New session creation API set to support resource uploading

["Alex Bozarth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481011#comment-16481011 ] 

Alex Bozarth commented on LIVY-471:
-----------------------------------

Continuing from my comments on the [PR|https://github.com/apache/incubator-livy/pull/91], it seems we have two API options with each consisting of three steps and step two is essentially the same:

 

Option 1:
 # Create Session but don't Start it
 # Upload files
 # Start Session

 

Option 2:
 # Create Session Id
 # Upload files
 # Create Session with Id

 

I see pros and cons for both, Option 1 can be done within our current API as seen in the PR, but the delay flag is a bit hacky. Option 2 will require a new API but would be similar to YARN.

 

I like option 1 because it fits with our current API and Sessions states. Creating a session but not starting it leaves it in a NotStarted state, which already exists and is clear to me in it's meaning. I also recognize that it puts the cart before the horse a bit in that the entire session is created before it is needed. My issues with option two is that it would require an update to the API and the potential session states (what would a state would a session be in if it's just an id?).

 

I'm willing to help review and clean up the current PR, which I have a handful of technical issues with, but I would also like to see how Option 2 would look in code because a solid, clean implementation could easily sway me. In summary I prefer moving forward with the current PR unless I see a really solid implementation of option 2 to convince me otherwise.

> New session creation API set to support resource uploading
> ----------------------------------------------------------
>
>                 Key: LIVY-471
>                 URL: https://issues.apache.org/jira/browse/LIVY-471
>             Project: Livy
>          Issue Type: Improvement
>          Components: Server
>    Affects Versions: 0.5.0
>            Reporter: Saisai Shao
>            Priority: Major
>
> Already post in mail list.
> In our current API design to create interactive / batch session, we assume end user should upload jars, pyFiles and related dependencies to HDFS before creating the session, and we use one POST request to create session. But usually end user may not have the permission to access the HDFS in their submission machine, so it makes them hard to create new sessions. So the requirement here is that if Livy could offer APIs to upload resources during session creation. One implementation is proposed in
> [https://github.com/apache/incubator-livy/pull/91|https://github.com/apache/incubator-livy/pull/91.]
> This add a field in session creation request to delay the session creation, then adding a bunch of APIs to support resource upload, finally adding an API to start creating the session. This seems a feasible solution, but also a little hack to support such scenario. So I was thinking if we could a set of new APIs to support such scenarios, rather than hack the existing APIs.
> To borrow the concept from yarn application submission, we can have 3 APIs to create session.
>  * requesting a new session id from Livy Server.
>  * uploading resources associate with this session id.
>  * submitting request to create session.
> This is similar to YARN's process to submit application, and we can bump the supported API version for newly added APIs.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)