You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Saisai Shao (JIRA)" <ji...@apache.org> on 2018/05/18 02:52:00 UTC
[jira] [Updated] (LIVY-471) New session creation API set to support
resource uploading
[ https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Saisai Shao updated LIVY-471:
-----------------------------
Description:
Already post in mail list.
In our current API design to create interactive / batch session, we assume end user should upload jars, pyFiles and related dependencies to HDFS before creating the session, and we use one POST request to create session. But usually end user may not have the permission to access the HDFS in their submission machine, so it makes them hard to create new sessions. So the requirement here is that if Livy could offer APIs to upload resources during session creation. One implementation is proposed in
[https://github.com/apache/incubator-livy/pull/91.]
This add a field in session creation request to delay the session creation, then adding a bunch of APIs to support resource upload, finally adding an API to start creating the session. This seems a feasible solution, but also a little hack to support such scenario. So I was thinking if we could a set of new APIs to support such scenarios, rather than hack the existing APIs.
To borrow the concept from yarn application submission, we can have 3 APIs to create session.
* requesting a new session id from Livy Server.
* uploading resources associate with this session id.
* submitting request to create session.
This is similar to YARN's process to submit application, and we can bump the supported API version for newly added APIs.
was:
Already post in mail list.
In our current API design to create interactive / batch session, we assume end user should upload jars, pyFiles and related dependencies to HDFS before creating the session, and we use one POST request to create session. But usually end user may not have the permission to access the HDFS in their submission machine, so it makes them hard to create new sessions. So the requirement here is that if Livy could offer APIs to upload resources during session creation. One implementation is proposed in
[https://github.com/apache/incubator-livy/pull/91.]
This add a field in session creation request to delay the session creation, then adding a bunch of APIs to support resource upload, finally adding an API to start creating the session. This seems a feasible solution, but also a little hack to support such scenario. So I was thinking if we could a set of new APIs to support such scenarios, rather than hack the existing APIs.
To borrow the concept from yarn application submission, we can have 3 APIs to create session.
* requesting a new session id from Livy Server.
* uploading resources associate with this session id.
* submitting request to create session. This is similar to YARN's process to submit application, and we can bump the supported API version for newly added APIs.
> New session creation API set to support resource uploading
> ----------------------------------------------------------
>
> Key: LIVY-471
> URL: https://issues.apache.org/jira/browse/LIVY-471
> Project: Livy
> Issue Type: Improvement
> Components: Server
> Affects Versions: 0.5.0
> Reporter: Saisai Shao
> Priority: Major
>
> Already post in mail list.
> In our current API design to create interactive / batch session, we assume end user should upload jars, pyFiles and related dependencies to HDFS before creating the session, and we use one POST request to create session. But usually end user may not have the permission to access the HDFS in their submission machine, so it makes them hard to create new sessions. So the requirement here is that if Livy could offer APIs to upload resources during session creation. One implementation is proposed in
> [https://github.com/apache/incubator-livy/pull/91.]
> This add a field in session creation request to delay the session creation, then adding a bunch of APIs to support resource upload, finally adding an API to start creating the session. This seems a feasible solution, but also a little hack to support such scenario. So I was thinking if we could a set of new APIs to support such scenarios, rather than hack the existing APIs.
> To borrow the concept from yarn application submission, we can have 3 APIs to create session.
> * requesting a new session id from Livy Server.
> * uploading resources associate with this session id.
> * submitting request to create session.
> This is similar to YARN's process to submit application, and we can bump the supported API version for newly added APIs.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)