You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Sandeep Kumar Mamidi (Jira)" <ji...@apache.org> on 2022/01/29 21:20:00 UTC
[jira] [Updated] (LIVY-882) Livy doesnot have an endpoint that would support execution of multiple SQL files in a single Interactive Session.
[ https://issues.apache.org/jira/browse/LIVY-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sandeep Kumar Mamidi updated LIVY-882:
--------------------------------------
Description:
*Problem Description :-*
Our requirement is that: in a single Livy Interactive Session, User wants to execute "x" number of SQL Files.User wants to explicitly pass these SQL files to the interactive session and perform execution. These SQL Files consists of contain multiple DDL, DML statements.
Below is how the User wants to run multiple SQLs using an interactive session:-
1) Open a Interactive Session with a predefined session name.
/usr/bin/curl --silent --negotiate -u:$USER --max-time 30 ${LIVY_SERVER}${uri} -X POST -H "X-Requested-By: $USER" -H 'Content-Type: application/json' -d "${json_data}"
2) Using the session name opened in the above step, User wants to perform execution of multiple SQL Files.
/usr/bin/curl -v --negotiate -u:$USER --max-time 30 "${LIVY_SERVER}${uri}/upload-statements?outputPath=$output_file" -H "X-Requested-By: $USER" -X POST -F file=@${code_file} -H 'Content-Type: multipart/form-data'
In the above curl command, code_file is the SQL File which the User has passed for execution.
*Suggested Solution :-*
The following code would support the new endpoint (upload-statements) for Livy Rest API:-
Below code needs to be added in "server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala"
{noformat}
post("/:id/upload-statements") {
withModifyAccessSession { session =>
fileParams.get("file") match {
case Some(file) =>
val fileContent = new String(file.get) val outputPath = params.get("outputPath")
val req = new ExecuteRequest(fileContent, None, outputPath)
val statement = session.executeStatement(req)
Created(statement,
headers = Map(
"Location" -> url(getStatement,
"id" -> session.id.toString,
"statementId" -> statement.id.toString)))
case None =>
log("Could not get the content of the file")
}{noformat}
was:
*Problem Description :-*
Our requirement is that: in a single Livy Interactive Session, User wants to execute "x" number of SQL Files.User wants to explicitly pass these SQL files to the interactive session and perform execution. These SQL Files consists of contain multiple DDL, DML statements.
Below is how the User wants to run multiple SQLs using an interactive session:-
1) Open a Interactive Session with a predefined session name.
/usr/bin/curl --silent --negotiate -u:$USER --max-time 30 ${LIVY_SERVER}${uri} -X POST -H "X-Requested-By: $USER" -H 'Content-Type: application/json' -d "${json_data}"
2) Using the session name opened in the above step, User wants to perform execution of multiple SQL Files.
/usr/bin/curl -v --negotiate -u:$USER --max-time 30 "${LIVY_SERVER}${uri}/upload-statements?outputPath=$output_file" -H "X-Requested-By: $USER" -X POST -F file=@${code_file} -H 'Content-Type: multipart/form-data'
In the above curl command, code_file is the SQL File which the User has passed for execution.
*Suggested Solution :-*
The following code would support the new endpoint (upload-statements) for Livy Rest API:-
Below code needs to be added in "server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala"
post("/:id/upload-statements") {
withModifyAccessSession { session =>
fileParams.get("file") match {
case Some(file) =>
val fileContent = new String(file.get)
val outputPath = params.get("outputPath")
val req = new ExecuteRequest(fileContent, None, outputPath)
val statement = session.executeStatement(req)
Created(statement,
headers = Map(
"Location" -> url(getStatement,
"id" -> session.id.toString,
"statementId" -> statement.id.toString)))
case None =>
log("Could not get the content of the file")
}
> Livy doesnot have an endpoint that would support execution of multiple SQL files in a single Interactive Session.
> -----------------------------------------------------------------------------------------------------------------
>
> Key: LIVY-882
> URL: https://issues.apache.org/jira/browse/LIVY-882
> Project: Livy
> Issue Type: New Feature
> Components: API
> Affects Versions: 0.7.0
> Reporter: Sandeep Kumar Mamidi
> Priority: Major
>
> *Problem Description :-*
> Our requirement is that: in a single Livy Interactive Session, User wants to execute "x" number of SQL Files.User wants to explicitly pass these SQL files to the interactive session and perform execution. These SQL Files consists of contain multiple DDL, DML statements.
> Below is how the User wants to run multiple SQLs using an interactive session:-
> 1) Open a Interactive Session with a predefined session name.
> /usr/bin/curl --silent --negotiate -u:$USER --max-time 30 ${LIVY_SERVER}${uri} -X POST -H "X-Requested-By: $USER" -H 'Content-Type: application/json' -d "${json_data}"
> 2) Using the session name opened in the above step, User wants to perform execution of multiple SQL Files.
> /usr/bin/curl -v --negotiate -u:$USER --max-time 30 "${LIVY_SERVER}${uri}/upload-statements?outputPath=$output_file" -H "X-Requested-By: $USER" -X POST -F file=@${code_file} -H 'Content-Type: multipart/form-data'
> In the above curl command, code_file is the SQL File which the User has passed for execution.
> *Suggested Solution :-*
> The following code would support the new endpoint (upload-statements) for Livy Rest API:-
> Below code needs to be added in "server/src/main/scala/org/apache/livy/server/interactive/InteractiveSessionServlet.scala"
> {noformat}
> post("/:id/upload-statements") {
> withModifyAccessSession { session =>
> fileParams.get("file") match {
> case Some(file) =>
> val fileContent = new String(file.get) val outputPath = params.get("outputPath")
> val req = new ExecuteRequest(fileContent, None, outputPath)
> val statement = session.executeStatement(req)
> Created(statement,
> headers = Map(
> "Location" -> url(getStatement,
> "id" -> session.id.toString,
> "statementId" -> statement.id.toString)))
> case None =>
> log("Could not get the content of the file")
> }{noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)