You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by fe...@apache.org on 2016/09/01 06:56:45 UTC
zeppelin git commit: [ZEPPELIN-699] Add new synchronous paragraph run
REST API
Repository: zeppelin
Updated Branches:
refs/heads/master 9dc9c7512 -> 7e2a1b5d4
[ZEPPELIN-699] Add new synchronous paragraph run REST API
### What is this PR for?
Right now, when calling the REST API `http://<ip>:<port>/api/notebook/job/<note_id>/<paragraph_id>` Zeppelin always returns **OK** as shown by this source code: https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java#L477
This ticket will update the behavior so that Zeppelin also return the result of the paragraph execution
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Code Review
* [ ] - Simple Test
### Is there a relevant Jira issue?
**[ZEPPELIN-699]**
### How should this be tested?
* `git fetch origin pull/746/head:ParagraphExecutionRESTAPI`
* `git checkout ParagraphExecutionRESTAPI`
* `mvn clean package -DskipTests`
* `bin/zeppelin-daemon.sh restart`
* Create a new note
* In the first paragraph, put the following code
```scala
%sh
echo "Current time = "`date +"%T"
```
* Retrieve the current note id in the URL
* Retrieve the current paragraph id
* Use a REST Client like **[POSTman]** to create a HTTP POST query `http://<ip>:<port>/api/notebook/run/<note_id>/<paragraph_id>`
* You should receive something similar as follow for answer
```
{
"status": "OK",
"body": {
"code": "SUCCESS",
"type": "TEXT",
"msg": "Current time = 16:14:18\n"
}
}
```
### Screenshots (if appropriate)
API Documentation update
**Existing asynchronous API**
**New synchronous API**
### Questions:
* Does the licenses files need update? --> **No**
* Is there breaking changes for older versions? --> **No**
* Does this needs documentation? --> **Yes**
[ZEPPELIN-699]: https://issues.apache.org/jira/browse/ZEPPELIN-699
[POSTman]: https://www.getpostman.com/
Author: DuyHai DOAN <do...@gmail.com>
Closes #746 from doanduyhai/ZEPPELIN-699 and squashes the following commits:
fb0570c [DuyHai DOAN] [ZEPPELIN-699] Update Notebook REST API documentation
8367acf [DuyHai DOAN] [ZEPPELIN-699] Add new synchronous paragraph run REST API
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/7e2a1b5d
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/7e2a1b5d
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/7e2a1b5d
Branch: refs/heads/master
Commit: 7e2a1b5d4f134489428a24c7c5502f189224b506
Parents: 9dc9c75
Author: DuyHai DOAN <do...@gmail.com>
Authored: Thu Jun 2 16:24:38 2016 +0200
Committer: Felix Cheung <fe...@apache.org>
Committed: Wed Aug 31 23:56:39 2016 -0700
----------------------------------------------------------------------
docs/rest-api/rest-notebook.md | 55 ++++++++++++-
.../apache/zeppelin/rest/NotebookRestApi.java | 81 ++++++++++++++++----
.../java/org/apache/zeppelin/notebook/Note.java | 26 ++++++-
3 files changed, 145 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7e2a1b5d/docs/rest-api/rest-notebook.md
----------------------------------------------------------------------
diff --git a/docs/rest-api/rest-notebook.md b/docs/rest-api/rest-notebook.md
index c7e17ea..ed39a4f 100644
--- a/docs/rest-api/rest-notebook.md
+++ b/docs/rest-api/rest-notebook.md
@@ -450,12 +450,12 @@ If you work with Apache Zeppelin and find a need for an additional REST API, ple
</table>
<br/>
-### Run a paragraph
+### Run a paragraph asynchronously
<table class="table-configuration">
<col width="200">
<tr>
<td>Description</td>
- <td>This ```POST``` method runs the paragraph by given notebook and paragraph id.
+ <td>This ```POST``` method runs the paragraph asynchronously by given notebook and paragraph id. This API always return SUCCESS even if the execution of the paragraph fails later because the API is asynchronous
</td>
</tr>
<tr>
@@ -488,6 +488,56 @@ If you work with Apache Zeppelin and find a need for an additional REST API, ple
</table>
<br/>
+### Run a paragraph synchronously
+ <table class="table-configuration">
+ <col width="200">
+ <tr>
+ <td>Description</td>
+ <td> This ```POST``` method runs the paragraph synchronously by given notebook and paragraph id. This API can return SUCCESS or ERROR depending on the outcome of the paragraph execution
+ </td>
+ </tr>
+ <tr>
+ <td>URL</td>
+ <td>```http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[notebookId]/[paragraphId]```</td>
+ </tr>
+ <tr>
+ <td>Success code</td>
+ <td>200</td>
+ </tr>
+ <tr>
+ <td> Fail code</td>
+ <td> 500 </td>
+ </tr>
+ <tr>
+ <td> sample JSON input (optional, only needed when if you want to update dynamic form's value) </td>
+ <td><pre>
+{
+ "name": "name of new notebook",
+ "params": {
+ "formLabel1": "value1",
+ "formLabel2": "value2"
+ }
+}</pre></td>
+ </tr>
+ <tr>
+ <td> sample JSON response </td>
+ <td><pre>{"status": "OK"}</pre></td>
+ </tr>
+ <tr>
+ <td> sample JSON error </td>
+ <td><pre>
+{
+ "status": "INTERNAL\_SERVER\_ERROR",
+ "body": {
+ "code": "ERROR",
+ "type": "TEXT",
+ "msg": "bash: -c: line 0: unexpected EOF while looking for matching ``'\nbash: -c: line 1: syntax error: unexpected end of file\nExitValue: 2"
+ }
+}</pre></td>
+ </tr>
+ </table>
+
+<br/>
### Stop a paragraph
<table class="table-configuration">
<col width="200">
@@ -922,4 +972,3 @@ If you work with Apache Zeppelin and find a need for an additional REST API, ple
</tr>
</tr>
</table>
-
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7e2a1b5d/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java
----------------------------------------------------------------------
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java
index db0cbec..17ec74f 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java
@@ -570,7 +570,7 @@ public class NotebookRestApi {
}
/**
- * Run paragraph job REST API
+ * Run asynchronously paragraph job REST API
*
* @param message - JSON with params if user wants to update dynamic form's value
* null, empty string, empty json if user doesn't want to update
@@ -583,7 +583,7 @@ public class NotebookRestApi {
public Response runParagraph(@PathParam("notebookId") String notebookId,
@PathParam("paragraphId") String paragraphId, String message)
throws IOException, IllegalArgumentException {
- LOG.info("run paragraph job {} {} {}", notebookId, paragraphId, message);
+ LOG.info("run paragraph job asynchronously {} {} {}", notebookId, paragraphId, message);
Note note = notebook.getNote(notebookId);
if (note == null) {
@@ -596,22 +596,60 @@ public class NotebookRestApi {
}
// handle params if presented
- if (!StringUtils.isEmpty(message)) {
- RunParagraphWithParametersRequest request =
- gson.fromJson(message, RunParagraphWithParametersRequest.class);
- Map<String, Object> paramsForUpdating = request.getParams();
- if (paramsForUpdating != null) {
- paragraph.settings.getParams().putAll(paramsForUpdating);
- AuthenticationInfo subject = new AuthenticationInfo(SecurityUtils.getPrincipal());
- note.setLastReplName(paragraph.getId());
- note.persist(subject);
- }
- }
+ handleParagraphParams(message, note, paragraph);
note.run(paragraph.getId());
return new JsonResponse<>(Status.OK).build();
}
+/**
+ * Run synchronously a paragraph REST API
+ *
+ * @param noteId - noteId
+ * @param paragraphId - paragraphId
+ * @param message - JSON with params if user wants to update dynamic form's value
+ * null, empty string, empty json if user doesn't want to update
+ *
+ * @return JSON with status.OK
+ * @throws IOException, IllegalArgumentException
+ */
+ @POST
+ @Path("run/{notebookId}/{paragraphId}")
+ @ZeppelinApi
+ public Response runParagraphSynchronously(@PathParam("notebookId") String noteId,
+ @PathParam("paragraphId") String paragraphId,
+ String message) throws
+ IOException, IllegalArgumentException {
+ LOG.info("run paragraph synchronously {} {} {}", noteId, paragraphId, message);
+
+ Note note = notebook.getNote(noteId);
+ if (note == null) {
+ return new JsonResponse<>(Status.NOT_FOUND, "note not found.").build();
+ }
+
+ Paragraph paragraph = note.getParagraph(paragraphId);
+ if (paragraph == null) {
+ return new JsonResponse<>(Status.NOT_FOUND, "paragraph not found.").build();
+ }
+
+ // handle params if presented
+ handleParagraphParams(message, note, paragraph);
+
+ if (paragraph.getListener() == null) {
+ note.initializeJobListenerForParagraph(paragraph);
+ }
+
+ paragraph.run();
+
+ final InterpreterResult result = paragraph.getResult();
+
+ if (result.code() == InterpreterResult.Code.SUCCESS) {
+ return new JsonResponse<>(Status.OK, result).build();
+ } else {
+ return new JsonResponse<>(Status.INTERNAL_SERVER_ERROR, result).build();
+ }
+ }
+
/**
* Stop(delete) paragraph job REST API
*
@@ -803,4 +841,21 @@ public class NotebookRestApi {
return new JsonResponse<>(Status.OK, notebooksFound).build();
}
+
+ private void handleParagraphParams(String message, Note note, Paragraph paragraph)
+ throws IOException {
+ // handle params if presented
+ if (!StringUtils.isEmpty(message)) {
+ RunParagraphWithParametersRequest request =
+ gson.fromJson(message, RunParagraphWithParametersRequest.class);
+ Map<String, Object> paramsForUpdating = request.getParams();
+ if (paramsForUpdating != null) {
+ paragraph.settings.getParams().putAll(paramsForUpdating);
+ AuthenticationInfo subject = new AuthenticationInfo(SecurityUtils.getPrincipal());
+ note.setLastReplName(paragraph.getId());
+ note.persist(subject);
+ }
+ }
+ }
+
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7e2a1b5d/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
index 2b89524..0b2b4c6 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
@@ -17,6 +17,8 @@
package org.apache.zeppelin.notebook;
+import static java.lang.String.format;
+
import java.io.IOException;
import java.io.Serializable;
import java.util.HashMap;
@@ -168,6 +170,28 @@ public class Note implements Serializable, ParagraphJobListener {
}
}
+ public void initializeJobListenerForParagraph(Paragraph paragraph) {
+ final Note paragraphNote = paragraph.getNote();
+ if (paragraphNote.getId().equals(this.getId())) {
+ throw new IllegalArgumentException(format("The paragraph %s from note %s " +
+ "does not belong to note %s", paragraph.getId(), paragraphNote.getId(),
+ this.getId()));
+ }
+
+ boolean foundParagraph = false;
+ for (Paragraph ownParagraph : paragraphs) {
+ if (paragraph.getId().equals(ownParagraph.getId())) {
+ paragraph.setListener(this.jobListenerFactory.getParagraphJobListener(this));
+ foundParagraph = true;
+ }
+ }
+
+ if (!foundParagraph) {
+ throw new IllegalArgumentException(format("Cannot find paragraph %s " +
+ "from note %s", paragraph.getId(), paragraphNote.getId()));
+ }
+ }
+
void setJobListenerFactory(JobListenerFactory jobListenerFactory) {
this.jobListenerFactory = jobListenerFactory;
}
@@ -480,7 +504,7 @@ public class Note implements Serializable, ParagraphJobListener {
logger.debug("New paragraph: {}", pText);
p.setEffectiveText(pText);
} else {
- String intpExceptionMsg = String.format("%s",
+ String intpExceptionMsg = format("%s",
p.getJobName()
+ "'s Interpreter "
+ requiredReplName + " not found"