You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by mo...@apache.org on 2016/04/28 16:29:12 UTC
incubator-zeppelin git commit: [ZEPPELIN-183] Interpreter process per
note
Repository: incubator-zeppelin
Updated Branches:
refs/heads/master bd4eaf8a3 -> 391ceab07
[ZEPPELIN-183] Interpreter process per note
### What is this PR for?
This PR give an option to create interpreter process per notebook.
### What type of PR is it?
Feature
### Todos
* [x] - Interpreter process per note
* [x] - Interpreter menu gui change
* [x] - unittest
* [x] - doc update
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-183
### How should this be tested?
Change interpreter setting option to 'process' and then create two notebooks.
Try to use both notebook with the same interpreter setting.
You'll see each notebook creates different interpreter process.
For example spark interpreter, if you run
```
%spark
sc
```
in two notebooks, then you'll see different object id.
### Screenshots (if appropriate)
Interpreter setting option gui looks like
If you hover the pointer on each option, you'll see tooltip help about each mode.
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
Author: Lee moon soo <mo...@apache.org>
Closes #795 from Leemoonsoo/ZEPPELIN-183 and squashes the following commits:
84f2201 [Lee moon soo] Provide correct localRepoPath
e5d9464 [Lee moon soo] Merge branch 'master' into ZEPPELIN-183
0bc3a1e [Lee moon soo] instance -> scoped, process -> isolated
382e2c2 [Lee moon soo] Add experimental
15c9ce5 [Lee moon soo] Fix style
777b1b6 [Lee moon soo] Update docs
103737f [Lee moon soo] fix close interpreter
d0b8394 [Lee moon soo] Interpreter setting page update
f14f3a3 [Lee moon soo] Per note interpreter process
Project: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/commit/391ceab0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/tree/391ceab0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/diff/391ceab0
Branch: refs/heads/master
Commit: 391ceab074c068d9deaeea80e357e739fd193fac
Parents: bd4eaf8
Author: Lee moon soo <mo...@apache.org>
Authored: Tue Apr 26 18:28:23 2016 -0700
Committer: Lee moon soo <mo...@apache.org>
Committed: Thu Apr 28 07:29:11 2016 -0700
----------------------------------------------------------------------
.../img/screenshots/interpreter_persession.png | Bin 37885 -> 43465 bytes
docs/interpreter/spark.md | 5 +-
docs/manual/interpreters.md | 4 +-
.../zeppelin/rest/InterpreterRestApi.java | 7 +-
.../apache/zeppelin/socket/NotebookServer.java | 18 +--
.../zeppelin/rest/AbstractTestRestApi.java | 4 +-
.../zeppelin/socket/NotebookServerTest.java | 2 +-
.../interpreter-create/interpreter-create.html | 34 +++++-
.../app/interpreter/interpreter.controller.js | 46 +++++++-
.../src/app/interpreter/interpreter.html | 41 +++++--
.../interpreter/InterpreterFactory.java | 113 +++++++++----------
.../interpreter/InterpreterGroupFactory.java | 26 +++++
.../zeppelin/interpreter/InterpreterOption.java | 9 ++
.../interpreter/InterpreterSetting.java | 73 ++++++++++--
.../java/org/apache/zeppelin/notebook/Note.java | 4 +-
.../notebook/NoteInterpreterLoader.java | 18 +--
.../org/apache/zeppelin/notebook/Notebook.java | 4 +-
.../org/apache/zeppelin/notebook/Paragraph.java | 4 +-
.../interpreter/InterpreterFactoryTest.java | 6 +-
.../notebook/NoteInterpreterLoaderTest.java | 58 ++++++++--
.../apache/zeppelin/notebook/NotebookTest.java | 8 +-
21 files changed, 352 insertions(+), 132 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/docs/assets/themes/zeppelin/img/screenshots/interpreter_persession.png
----------------------------------------------------------------------
diff --git a/docs/assets/themes/zeppelin/img/screenshots/interpreter_persession.png b/docs/assets/themes/zeppelin/img/screenshots/interpreter_persession.png
index bb81ed3..aae354d 100644
Binary files a/docs/assets/themes/zeppelin/img/screenshots/interpreter_persession.png and b/docs/assets/themes/zeppelin/img/screenshots/interpreter_persession.png differ
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/docs/interpreter/spark.md
----------------------------------------------------------------------
diff --git a/docs/interpreter/spark.md b/docs/interpreter/spark.md
index ce178e7..73590e1 100644
--- a/docs/interpreter/spark.md
+++ b/docs/interpreter/spark.md
@@ -334,9 +334,10 @@ select * from ${table=defaultTableName} where text like '%${search}%'
To learn more about dynamic form, checkout [Dynamic Form](../manual/dynamicform.html).
-### Separate Interpreter for each note
+### Interpreter setting option.
+
+Interpreter setting can choose one of 'shared', 'scoped', 'isolated' option. Spark interpreter creates separate scala compiler per each notebook but share a single SparkContext in 'scoped' mode (experimental). It creates separate SparkContext per each notebook in 'isolated' mode.
-In 'Separate Interpreter for each note' mode, SparkInterpreter creates scala compiler per each notebook. However it still shares the single SparkContext.
## Setting up Zeppelin with Kerberos
Logical setup with Zeppelin, Kerberos Key Distribution Center (KDC), and Spark on YARN:
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/docs/manual/interpreters.md
----------------------------------------------------------------------
diff --git a/docs/manual/interpreters.md b/docs/manual/interpreters.md
index 692001a..0441679 100644
--- a/docs/manual/interpreters.md
+++ b/docs/manual/interpreters.md
@@ -56,7 +56,7 @@ Each interpreters is belonged to a single group and registered together. All of
## Interpreter binding mode
-Each Interpreter Setting can choose one of two different interpreter binding mode.
-Shared mode (default) and 'Separate Interpreter for each note' mode. In shared mode, every notebook bound to the Interpreter Setting will share the single Interpreter instance. In 'Separate Interpreter for each note' mode, each notebook will create new Interpreter instance. Therefore each notebook will have fresh new Interpreter environment.
+Each Interpreter Setting can choose one of 'shared', 'scoped', 'isolated' interpreter binding mode.
+In 'shared' mode, every notebook bound to the Interpreter Setting will share the single Interpreter instance. In 'scoped' mode, each notebook will create new Interpreter instance in the same interpreter process. In 'isolated' mode, each notebook will create new Interpreter process.
<img src="/assets/themes/zeppelin/img/screenshots/interpreter_persession.png" width="400px">
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-server/src/main/java/org/apache/zeppelin/rest/InterpreterRestApi.java
----------------------------------------------------------------------
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/InterpreterRestApi.java b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/InterpreterRestApi.java
index 5bf42b4..6430973 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/InterpreterRestApi.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/InterpreterRestApi.java
@@ -94,14 +94,13 @@ public class InterpreterRestApi {
NewInterpreterSettingRequest.class);
Properties p = new Properties();
p.putAll(request.getProperties());
- InterpreterGroup interpreterGroup = interpreterFactory.add(request.getName(),
+ InterpreterSetting interpreterSetting = interpreterFactory.add(request.getName(),
request.getGroup(),
request.getDependencies(),
request.getOption(),
p);
- InterpreterSetting setting = interpreterFactory.get(interpreterGroup.getId());
- logger.info("new setting created with {}", setting.id());
- return new JsonResponse(Status.CREATED, "", setting).build();
+ logger.info("new setting created with {}", interpreterSetting.id());
+ return new JsonResponse(Status.CREATED, "", interpreterSetting).build();
} catch (InterpreterException e) {
logger.error("Exception in InterpreterRestApi while creating ", e);
return new JsonResponse(
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
----------------------------------------------------------------------
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
index 6ad3537..3fa3d8d 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
@@ -664,12 +664,12 @@ public class NotebookServer extends WebSocketServlet implements
List<InterpreterSetting> settings = note.getNoteReplLoader()
.getInterpreterSettings();
for (InterpreterSetting setting : settings) {
- if (setting.getInterpreterGroup() == null) {
+ if (setting.getInterpreterGroup(note.id()) == null) {
continue;
}
- if (interpreterGroupId.equals(setting.getInterpreterGroup().getId())) {
+ if (interpreterGroupId.equals(setting.getInterpreterGroup(note.id()).getId())) {
AngularObjectRegistry angularObjectRegistry = setting
- .getInterpreterGroup().getAngularObjectRegistry();
+ .getInterpreterGroup(note.id()).getAngularObjectRegistry();
// first trying to get local registry
ao = angularObjectRegistry.get(varName, noteId, paragraphId);
if (ao == null) {
@@ -705,12 +705,12 @@ public class NotebookServer extends WebSocketServlet implements
List<InterpreterSetting> settings = note.getNoteReplLoader()
.getInterpreterSettings();
for (InterpreterSetting setting : settings) {
- if (setting.getInterpreterGroup() == null) {
+ if (setting.getInterpreterGroup(n.id()) == null) {
continue;
}
- if (interpreterGroupId.equals(setting.getInterpreterGroup().getId())) {
+ if (interpreterGroupId.equals(setting.getInterpreterGroup(n.id()).getId())) {
AngularObjectRegistry angularObjectRegistry = setting
- .getInterpreterGroup().getAngularObjectRegistry();
+ .getInterpreterGroup(n.id()).getAngularObjectRegistry();
this.broadcastExcept(
n.id(),
new Message(OP.ANGULAR_OBJECT_UPDATE).put("angularObject", ao)
@@ -1146,14 +1146,14 @@ public class NotebookServer extends WebSocketServlet implements
}
for (InterpreterSetting intpSetting : settings) {
- AngularObjectRegistry registry = intpSetting.getInterpreterGroup()
+ AngularObjectRegistry registry = intpSetting.getInterpreterGroup(note.id())
.getAngularObjectRegistry();
List<AngularObject> objects = registry.getAllWithGlobal(note.id());
for (AngularObject object : objects) {
conn.send(serializeMessage(new Message(OP.ANGULAR_OBJECT_UPDATE)
.put("angularObject", object)
.put("interpreterGroupId",
- intpSetting.getInterpreterGroup().getId())
+ intpSetting.getInterpreterGroup(note.id()).getId())
.put("noteId", note.id())
.put("paragraphId", object.getParagraphId())
));
@@ -1184,7 +1184,7 @@ public class NotebookServer extends WebSocketServlet implements
if (intpSettings.isEmpty())
continue;
for (InterpreterSetting setting : intpSettings) {
- if (setting.getInterpreterGroup().getId().equals(interpreterGroupId)) {
+ if (setting.getInterpreterGroup(note.id()).getId().equals(interpreterGroupId)) {
broadcast(
note.id(),
new Message(OP.ANGULAR_OBJECT_UPDATE)
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java
----------------------------------------------------------------------
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java
index 4c5cc5e..5ea3c09 100644
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java
+++ b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java
@@ -374,13 +374,13 @@ public abstract class AbstractTestRestApi {
//Create new Setting and return Setting ID
protected String createTempSetting(String tempName)
throws IOException, RepositoryException {
- InterpreterGroup interpreterGroup = ZeppelinServer.notebook.getInterpreterFactory()
+ InterpreterSetting setting = ZeppelinServer.notebook.getInterpreterFactory()
.add(tempName,
"newGroup",
new LinkedList<Dependency>(),
new InterpreterOption(false),
new Properties());
- return interpreterGroup.getId();
+ return setting.id();
}
protected TypeSafeMatcher<? super JsonElement> hasRootElementNamed(final String memberName) {
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
----------------------------------------------------------------------
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java b/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
index 98895de..4cf57e3 100644
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
+++ b/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
@@ -95,7 +95,7 @@ public class NotebookServerTest extends AbstractTestRestApi {
List<InterpreterSetting> settings = note1.getNoteReplLoader().getInterpreterSettings();
for (InterpreterSetting setting : settings) {
if (setting.getName().equals("md")) {
- interpreterGroup = setting.getInterpreterGroup();
+ interpreterGroup = setting.getInterpreterGroup("sharedProcess");
break;
}
}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-web/src/app/interpreter/interpreter-create/interpreter-create.html
----------------------------------------------------------------------
diff --git a/zeppelin-web/src/app/interpreter/interpreter-create/interpreter-create.html b/zeppelin-web/src/app/interpreter/interpreter-create/interpreter-create.html
index f7769b4..fdc216c 100644
--- a/zeppelin-web/src/app/interpreter/interpreter-create/interpreter-create.html
+++ b/zeppelin-web/src/app/interpreter/interpreter-create/interpreter-create.html
@@ -36,9 +36,39 @@ limitations under the License.
</div>
<b>Option</b>
- <div class="checkbox">
- <label><input type="checkbox" style="top:-5px" ng-model="newInterpreterSetting.option.perNoteSession">Separate Interpreter for each note</input></label>
+ <div>
+ <span class="btn-group">
+ <button type="button" class="btn btn-default btn-xs dropdown-toggle"
+ data-toggle="dropdown">
+ {{getSessionOption(setting.id)}} <span class="caret"></span>
+ </button>
+ <ul class="dropdown-menu" role="menu">
+ <li>
+ <a style="cursor:pointer"
+ tooltip="Single interpreter instance are shared across notes"
+ ng-click="setSessionOption(setting.id, 'shared')">
+ shared
+ </a>
+ </li>
+ <li>
+ <a style="cursor:pointer"
+ tooltip="Separate Interpreter instance for each note"
+ ng-click="setSessionOption(setting.id, 'scoped')">
+ scoped
+ </a>
+ </li>
+ <li>
+ <a style="cursor:pointer"
+ tooltip="Separate Interpreter process for each note"
+ ng-click="setSessionOption(setting.id, 'isolated')">
+ isolated
+ </a>
+ </li>
+ </ul>
+ </span>
+ <span>Interpreter for note</span>
</div>
+ <br />
<b>Properties</b>
<table class="table table-striped properties">
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-web/src/app/interpreter/interpreter.controller.js
----------------------------------------------------------------------
diff --git a/zeppelin-web/src/app/interpreter/interpreter.controller.js b/zeppelin-web/src/app/interpreter/interpreter.controller.js
index ee72c3f..58b2d06 100644
--- a/zeppelin-web/src/app/interpreter/interpreter.controller.js
+++ b/zeppelin-web/src/app/interpreter/interpreter.controller.js
@@ -60,7 +60,48 @@ angular.module('zeppelinWebApp').controller('InterpreterCtrl', function($scope,
interpreterSettingsTmp[index] = angular.copy($scope.interpreterSettings[index]);
};
- $scope.updateInterpreterSetting = function (form, settingId) {
+ $scope.setSessionOption = function(settingId, sessionOption) {
+ var option;
+ if (settingId === undefined) {
+ option = $scope.newInterpreterSetting.option;
+ } else {
+ var index = _.findIndex($scope.interpreterSettings, {'id': settingId});
+ var setting = $scope.interpreterSettings[index];
+ option = setting.option;
+ }
+
+ if (sessionOption === 'isolated') {
+ option.perNoteSession = false;
+ option.perNoteProcess = true;
+ } else if (sessionOption === 'scoped') {
+ option.perNoteSession = true;
+ option.perNoteProcess = false;
+ } else {
+ option.perNoteSession = false;
+ option.perNoteProcess = false;
+ }
+ };
+
+ $scope.getSessionOption = function(settingId) {
+ var option;
+ if (settingId === undefined) {
+ option = $scope.newInterpreterSetting.option;
+ } else {
+ var index = _.findIndex($scope.interpreterSettings, {'id': settingId});
+ var setting = $scope.interpreterSettings[index];
+ option = setting.option;
+ }
+
+ if (option.perNoteSession) {
+ return 'scoped';
+ } else if (option.perNoteProcess) {
+ return 'isolated';
+ } else {
+ return 'shared';
+ }
+ };
+
+ $scope.updateInterpreterSetting = function(form, settingId) {
BootstrapDialog.confirm({
closable: true,
title: '',
@@ -231,7 +272,8 @@ angular.module('zeppelinWebApp').controller('InterpreterCtrl', function($scope,
dependencies: [],
option: {
remote: true,
- perNoteSession: false
+ perNoteSession: false,
+ perNoteProcess: false
}
};
emptyNewProperty($scope.newInterpreterSetting);
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-web/src/app/interpreter/interpreter.html
----------------------------------------------------------------------
diff --git a/zeppelin-web/src/app/interpreter/interpreter.html b/zeppelin-web/src/app/interpreter/interpreter.html
index c30ad61..bfdc7c7 100644
--- a/zeppelin-web/src/app/interpreter/interpreter.html
+++ b/zeppelin-web/src/app/interpreter/interpreter.html
@@ -111,16 +111,39 @@ limitations under the License.
<div class="row interpreter">
<div class="col-md-12">
<h5>Option</h5>
- <div class="checkbox">
- <label>
- <input type="checkbox"
- style="top:-5px"
- ng-disabled="!valueform.$visible"
- ng-model="setting.option.perNoteSession">
- Separate Interpreter for each note</input>
- </label>
- </div>
+ <span class="btn-group">
+ <button type="button" class="btn btn-default btn-xs dropdown-toggle"
+ data-toggle="dropdown"
+ ng-disabled="!valueform.$visible">
+ {{getSessionOption(setting.id)}} <span class="caret"></span>
+ </button>
+ <ul class="dropdown-menu" role="menu">
+ <li>
+ <a style="cursor:pointer"
+ tooltip="Single interpreter instance are shared across notes"
+ ng-click="setSessionOption(setting.id, 'shared')">
+ shared
+ </a>
+ </li>
+ <li>
+ <a style="cursor:pointer"
+ tooltip="Separate Interpreter instance for each note"
+ ng-click="setSessionOption(setting.id, 'scoped')">
+ scoped
+ </a>
+ </li>
+ <li>
+ <a style="cursor:pointer"
+ tooltip="Separate Interpreter process for each note"
+ ng-click="setSessionOption(setting.id, 'isolated')">
+ isolated
+ </a>
+ </li>
+ </ul>
+ </span>
+ <span>Interpreter for note</span>
</div>
+
<div ng-show="_.isEmpty(setting.properties) && _.isEmpty(setting.dependencies) || valueform.$hidden" class="col-md-12 gray40-message">
<em>Currently there are no properties and dependencies set for this interpreter</em>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java
index 269b54a..08e6465 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java
@@ -33,6 +33,7 @@ import org.apache.zeppelin.interpreter.Interpreter.RegisteredInterpreter;
import org.apache.zeppelin.interpreter.remote.RemoteAngularObjectRegistry;
import org.apache.zeppelin.interpreter.remote.RemoteInterpreter;
import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcessListener;
+import org.apache.zeppelin.notebook.NoteInterpreterLoader;
import org.apache.zeppelin.scheduler.Job;
import org.apache.zeppelin.scheduler.Job.Status;
import org.slf4j.Logger;
@@ -52,7 +53,7 @@ import java.util.*;
/**
* Manage interpreters.
*/
-public class InterpreterFactory {
+public class InterpreterFactory implements InterpreterGroupFactory {
Logger logger = LoggerFactory.getLogger(InterpreterFactory.class);
private Map<String, URLClassLoader> cleanCl = Collections
@@ -244,9 +245,7 @@ public class InterpreterFactory {
setting.getDependencies(),
setting.getOption());
- InterpreterGroup interpreterGroup = createInterpreterGroup(setting.id(), setting.getOption());
- intpSetting.setInterpreterGroup(interpreterGroup);
-
+ intpSetting.setInterpreterGroupFactory(this);
interpreterSettings.put(k, intpSetting);
}
@@ -370,7 +369,7 @@ public class InterpreterFactory {
* @throws InterpreterException
* @throws IOException
*/
- public InterpreterGroup add(String name, String groupName,
+ public InterpreterSetting add(String name, String groupName,
List<Dependency> dependencies,
InterpreterOption option, Properties properties)
throws InterpreterException, IOException, RepositoryException {
@@ -404,17 +403,15 @@ public class InterpreterFactory {
loadInterpreterDependencies(intpSetting);
}
- InterpreterGroup interpreterGroup = createInterpreterGroup(intpSetting.id(), option);
-
- intpSetting.setInterpreterGroup(interpreterGroup);
-
+ intpSetting.setInterpreterGroupFactory(this);
interpreterSettings.put(intpSetting.id(), intpSetting);
saveToFile();
- return interpreterGroup;
+ return intpSetting;
}
}
- private InterpreterGroup createInterpreterGroup(String id, InterpreterOption option)
+ @Override
+ public InterpreterGroup createInterpreterGroup(String id, InterpreterOption option)
throws InterpreterException, NullArgumentException {
//When called from REST API without option we receive NPE
@@ -444,25 +441,28 @@ public class InterpreterFactory {
public void removeInterpretersForNote(InterpreterSetting interpreterSetting,
String noteId) {
- if (!interpreterSetting.getOption().isPerNoteSession()) {
- return;
- }
- InterpreterGroup interpreterGroup = interpreterSetting.getInterpreterGroup();
- interpreterGroup.close(noteId);
- interpreterGroup.destroy(noteId);
- synchronized (interpreterGroup) {
- interpreterGroup.remove(noteId);
- interpreterGroup.notifyAll(); // notify createInterpreterForNote()
+ if (interpreterSetting.getOption().isPerNoteProcess()) {
+ interpreterSetting.closeAndRemoveInterpreterGroup(noteId);
+ } else if (interpreterSetting.getOption().isPerNoteSession()) {
+ InterpreterGroup interpreterGroup = interpreterSetting.getInterpreterGroup(noteId);
+
+ interpreterGroup.close(noteId);
+ interpreterGroup.destroy(noteId);
+ synchronized (interpreterGroup) {
+ interpreterGroup.remove(noteId);
+ interpreterGroup.notifyAll(); // notify createInterpreterForNote()
+ }
+ logger.info("Interpreter instance {} for note {} is removed",
+ interpreterSetting.getName(),
+ noteId);
}
- logger.info("Interpreter instance {} for note {} is removed",
- interpreterSetting.getName(),
- noteId);
}
public void createInterpretersForNote(
InterpreterSetting interpreterSetting,
- String noteId) {
- InterpreterGroup interpreterGroup = interpreterSetting.getInterpreterGroup();
+ String noteId,
+ String key) {
+ InterpreterGroup interpreterGroup = interpreterSetting.getInterpreterGroup(noteId);
String groupName = interpreterSetting.getGroup();
InterpreterOption option = interpreterSetting.getOption();
Properties properties = interpreterSetting.getProperties();
@@ -473,10 +473,12 @@ public class InterpreterFactory {
// interpreter process supposed to be terminated by RemoteInterpreterProcess.dereference()
// in ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT msec. However, if termination of the process and
// removal from interpreter group take too long, throw an error.
- long minTimeout = 10 * 1000 * 1000000; // 10 sec
+ long minTimeout = 10L * 1000 * 1000000; // 10 sec
long interpreterRemovalWaitTimeout =
- Math.max(minTimeout, conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT) * 2);
- while (interpreterGroup.containsKey(noteId)) {
+ Math.max(
+ minTimeout,
+ conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT) * 1000000L * 2);
+ while (interpreterGroup.containsKey(key)) {
if (System.nanoTime() - interpreterRemovalWaitStart > interpreterRemovalWaitTimeout) {
throw new InterpreterException("Can not create interpreter");
}
@@ -500,10 +502,10 @@ public class InterpreterFactory {
if (option.isRemote()) {
intp = createRemoteRepl(info.getPath(),
- noteId,
+ key,
info.getClassName(),
properties,
- interpreterGroup.id);
+ interpreterSetting.id());
} else {
intp = createRepl(info.getPath(),
info.getClassName(),
@@ -511,10 +513,10 @@ public class InterpreterFactory {
}
synchronized (interpreterGroup) {
- List<Interpreter> interpreters = interpreterGroup.get(noteId);
+ List<Interpreter> interpreters = interpreterGroup.get(key);
if (interpreters == null) {
interpreters = new LinkedList<Interpreter>();
- interpreterGroup.put(noteId, interpreters);
+ interpreterGroup.put(key, interpreters);
}
interpreters.add(intp);
}
@@ -532,8 +534,7 @@ public class InterpreterFactory {
synchronized (interpreterSettings) {
if (interpreterSettings.containsKey(id)) {
InterpreterSetting intp = interpreterSettings.get(id);
- intp.getInterpreterGroup().close();
- intp.getInterpreterGroup().destroy();
+ intp.closeAndRmoveAllInterpreterGroups();
interpreterSettings.remove(id);
for (List<String> settings : interpreterBindings.values()) {
@@ -663,16 +664,12 @@ public class InterpreterFactory {
stopJobAllInterpreter(intpsetting);
- intpsetting.getInterpreterGroup().close();
- intpsetting.getInterpreterGroup().destroy();
+ intpsetting.closeAndRmoveAllInterpreterGroups();
intpsetting.setOption(option);
intpsetting.setProperties(properties);
intpsetting.setDependencies(dependencies);
- InterpreterGroup interpreterGroup = createInterpreterGroup(intpsetting.id(), option);
- intpsetting.setInterpreterGroup(interpreterGroup);
-
loadInterpreterDependencies(intpsetting);
saveToFile();
} else {
@@ -689,13 +686,8 @@ public class InterpreterFactory {
stopJobAllInterpreter(intpsetting);
- intpsetting.getInterpreterGroup().close();
- intpsetting.getInterpreterGroup().destroy();
+ intpsetting.closeAndRmoveAllInterpreterGroups();
- InterpreterGroup interpreterGroup = createInterpreterGroup(
- intpsetting.id(),
- intpsetting.getOption());
- intpsetting.setInterpreterGroup(interpreterGroup);
} else {
throw new InterpreterException("Interpreter setting id " + id
+ " not found");
@@ -705,17 +697,19 @@ public class InterpreterFactory {
private void stopJobAllInterpreter(InterpreterSetting intpsetting) {
if (intpsetting != null) {
- for (List<Interpreter> interpreters : intpsetting.getInterpreterGroup().values()) {
- for (Interpreter intp : interpreters) {
- for (Job job : intp.getScheduler().getJobsRunning()) {
- job.abort();
- job.setStatus(Status.ABORT);
- logger.info("Job " + job.getJobName() + " aborted ");
- }
- for (Job job : intp.getScheduler().getJobsWaiting()) {
- job.abort();
- job.setStatus(Status.ABORT);
- logger.info("Job " + job.getJobName() + " aborted ");
+ for (InterpreterGroup intpGroup : intpsetting.getAllInterpreterGroups()) {
+ for (List<Interpreter> interpreters : intpGroup.values()) {
+ for (Interpreter intp : interpreters) {
+ for (Job job : intp.getScheduler().getJobsRunning()) {
+ job.abort();
+ job.setStatus(Status.ABORT);
+ logger.info("Job " + job.getJobName() + " aborted ");
+ }
+ for (Job job : intp.getScheduler().getJobsWaiting()) {
+ job.abort();
+ job.setStatus(Status.ABORT);
+ logger.info("Job " + job.getJobName() + " aborted ");
+ }
}
}
}
@@ -729,8 +723,7 @@ public class InterpreterFactory {
for (final InterpreterSetting intpsetting : intpsettings) {
Thread t = new Thread() {
public void run() {
- intpsetting.getInterpreterGroup().close();
- intpsetting.getInterpreterGroup().destroy();
+ intpsetting.closeAndRmoveAllInterpreterGroups();
}
};
t.start();
@@ -809,9 +802,9 @@ public class InterpreterFactory {
private Interpreter createRemoteRepl(String interpreterPath, String noteId, String className,
- Properties property, String interpreterId) {
+ Properties property, String interpreterSettingId) {
int connectTimeout = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT);
- String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" + interpreterId;
+ String localRepoPath = conf.getInterpreterLocalRepoPath() + "/" + interpreterSettingId;
int maxPoolSize = conf.getInt(ConfVars.ZEPPELIN_INTERPRETER_MAX_POOL_SIZE);
LazyOpenInterpreter intp = new LazyOpenInterpreter(new RemoteInterpreter(
property, noteId, className, conf.getInterpreterRemoteRunnerPath(),
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterGroupFactory.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterGroupFactory.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterGroupFactory.java
new file mode 100644
index 0000000..3b9be40
--- /dev/null
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterGroupFactory.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.interpreter;
+
+import org.apache.commons.lang.NullArgumentException;
+
+/**
+ * Created InterpreterGroup
+ */
+public interface InterpreterGroupFactory {
+ InterpreterGroup createInterpreterGroup(String interpreterGroupId, InterpreterOption option);
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterOption.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterOption.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterOption.java
index 29a4748..ba6f7b9 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterOption.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterOption.java
@@ -23,6 +23,7 @@ package org.apache.zeppelin.interpreter;
public class InterpreterOption {
boolean remote;
boolean perNoteSession;
+ boolean perNoteProcess;
public InterpreterOption() {
remote = false;
@@ -47,4 +48,12 @@ public class InterpreterOption {
public void setPerNoteSession(boolean perNoteSession) {
this.perNoteSession = perNoteSession;
}
+
+ public boolean isPerNoteProcess() {
+ return perNoteProcess;
+ }
+
+ public void setPerNoteProcess(boolean perNoteProcess) {
+ this.perNoteProcess = perNoteProcess;
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
index 96c91a9..91eb2c8 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
@@ -17,28 +17,32 @@
package org.apache.zeppelin.interpreter;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Properties;
-import java.util.Random;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.commons.lang.NullArgumentException;
import org.apache.zeppelin.dep.Dependency;
+import org.apache.zeppelin.display.AngularObjectRegistry;
+import org.apache.zeppelin.interpreter.remote.RemoteAngularObjectRegistry;
import org.apache.zeppelin.notebook.utility.IdHashes;
/**
* Interpreter settings
*/
public class InterpreterSetting {
+ private static final String SHARED_PROCESS = "shared_process";
private String id;
private String name;
private String group;
private String description;
private Properties properties;
+ private transient InterpreterGroupFactory interpreterGroupFactory;
// use 'interpreterGroup' as a field name to keep backward compatibility of
// conf/interpreter.json file format
private List<InterpreterInfo> interpreterGroup;
- private transient InterpreterGroup interpreterGroupRef;
+ private transient Map<String, InterpreterGroup> interpreterGroupRef =
+ new HashMap<String, InterpreterGroup>();
private List<Dependency> dependencies;
private InterpreterOption option;
@@ -56,6 +60,7 @@ public class InterpreterSetting {
this.properties = properties;
this.dependencies = dependencies;
this.option = option;
+ this.interpreterGroupFactory = interpreterGroupFactory;
}
public InterpreterSetting(String name,
@@ -117,12 +122,54 @@ public class InterpreterSetting {
return group;
}
- public InterpreterGroup getInterpreterGroup() {
- return interpreterGroupRef;
+
+ private String getInterpreterProcessKey(String noteId) {
+ if (getOption().isPerNoteProcess()) {
+ return noteId;
+ } else {
+ return SHARED_PROCESS;
+ }
+ }
+
+ public InterpreterGroup getInterpreterGroup(String noteId) {
+ String key = getInterpreterProcessKey(noteId);
+ synchronized (interpreterGroupRef) {
+ if (!interpreterGroupRef.containsKey(key)) {
+ String interpreterGroupId = id() + ":" + key;
+ InterpreterGroup intpGroup =
+ interpreterGroupFactory.createInterpreterGroup(interpreterGroupId, getOption());
+ interpreterGroupRef.put(key, intpGroup);
+ }
+ return interpreterGroupRef.get(key);
+ }
+ }
+
+ public Collection<InterpreterGroup> getAllInterpreterGroups() {
+ synchronized (interpreterGroupRef) {
+ return new LinkedList<InterpreterGroup>(interpreterGroupRef.values());
+ }
+ }
+
+ public void closeAndRemoveInterpreterGroup(String noteId) {
+ String key = getInterpreterProcessKey(noteId);
+ InterpreterGroup groupToRemove;
+ synchronized (interpreterGroupRef) {
+ groupToRemove = interpreterGroupRef.remove(key);
+ }
+
+ if (groupToRemove != null) {
+ groupToRemove.close();
+ groupToRemove.destroy();
+ }
}
- public void setInterpreterGroup(InterpreterGroup interpreterGroup) {
- this.interpreterGroupRef = interpreterGroup;
+ public void closeAndRmoveAllInterpreterGroups() {
+ synchronized (interpreterGroupRef) {
+ HashSet<String> groupsToRemove = new HashSet<String>(interpreterGroupRef.keySet());
+ for (String key : groupsToRemove) {
+ closeAndRemoveInterpreterGroup(key);
+ }
+ }
}
public Properties getProperties() {
@@ -159,4 +206,12 @@ public class InterpreterSetting {
public List<InterpreterInfo> getInterpreterInfos() {
return interpreterGroup;
}
+
+ public InterpreterGroupFactory getInterpreterGroupFactory() {
+ return interpreterGroupFactory;
+ }
+
+ public void setInterpreterGroupFactory(InterpreterGroupFactory interpreterGroupFactory) {
+ this.interpreterGroupFactory = interpreterGroupFactory;
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
index 5ef1171..f2bbe11 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
@@ -405,7 +405,7 @@ public class Note implements Serializable, JobListener {
}
for (InterpreterSetting setting : settings) {
- InterpreterGroup intpGroup = setting.getInterpreterGroup();
+ InterpreterGroup intpGroup = setting.getInterpreterGroup(id);
AngularObjectRegistry registry = intpGroup.getAngularObjectRegistry();
angularObjects.put(intpGroup.getId(), registry.getAllWithGlobal(id));
}
@@ -420,7 +420,7 @@ public class Note implements Serializable, JobListener {
}
for (InterpreterSetting setting : settings) {
- InterpreterGroup intpGroup = setting.getInterpreterGroup();
+ InterpreterGroup intpGroup = setting.getInterpreterGroup(id);
AngularObjectRegistry registry = intpGroup.getAngularObjectRegistry();
if (registry instanceof RemoteAngularObjectRegistry) {
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/NoteInterpreterLoader.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/NoteInterpreterLoader.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/NoteInterpreterLoader.java
index 4e7626b..7e22ca0 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/NoteInterpreterLoader.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/NoteInterpreterLoader.java
@@ -74,22 +74,23 @@ public class NoteInterpreterLoader {
return settings;
}
- private String getInterpreterGroupKey(InterpreterSetting setting) {
- if (!setting.getOption().isPerNoteSession()) {
- return SHARED_SESSION;
- } else {
+ private String getInterpreterInstanceKey(InterpreterSetting setting) {
+ if (setting.getOption().isPerNoteSession() || setting.getOption().isPerNoteProcess()) {
return noteId;
+ } else {
+ return SHARED_SESSION;
}
}
private List<Interpreter> createOrGetInterpreterList(InterpreterSetting setting) {
- InterpreterGroup interpreterGroup = setting.getInterpreterGroup();
+ InterpreterGroup interpreterGroup =
+ setting.getInterpreterGroup(noteId);
synchronized (interpreterGroup) {
- String key = getInterpreterGroupKey(setting);
+ String key = getInterpreterInstanceKey(setting);
if (!interpreterGroup.containsKey(key)) {
- factory.createInterpretersForNote(setting, key);
+ factory.createInterpretersForNote(setting, noteId, key);
}
- return interpreterGroup.get(getInterpreterGroupKey(setting));
+ return interpreterGroup.get(getInterpreterInstanceKey(setting));
}
}
@@ -100,6 +101,7 @@ public class NoteInterpreterLoader {
return;
}
+ System.err.println("close");
for (InterpreterSetting setting : settings) {
factory.removeInterpretersForNote(setting, noteId);
}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
index 4827bff..986753c 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
@@ -290,7 +290,7 @@ public class Notebook {
// remove from all interpreter instance's angular object registry
for (InterpreterSetting settings : replFactory.get()) {
- AngularObjectRegistry registry = settings.getInterpreterGroup().getAngularObjectRegistry();
+ AngularObjectRegistry registry = settings.getInterpreterGroup(id).getAngularObjectRegistry();
if (registry instanceof RemoteAngularObjectRegistry) {
// remove paragraph scope object
for (Paragraph p : note.getParagraphs()) {
@@ -380,7 +380,7 @@ public class Notebook {
SnapshotAngularObject snapshot = angularObjectSnapshot.get(name);
List<InterpreterSetting> settings = replFactory.get();
for (InterpreterSetting setting : settings) {
- InterpreterGroup intpGroup = setting.getInterpreterGroup();
+ InterpreterGroup intpGroup = setting.getInterpreterGroup(note.id());
if (intpGroup.getId().equals(snapshot.getIntpGroupId())) {
AngularObjectRegistry registry = intpGroup.getAngularObjectRegistry();
String noteId = snapshot.getAngularObject().getNoteId();
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
index ae0c410..6f2592b 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
@@ -320,8 +320,8 @@ public class Paragraph extends Job implements Serializable, Cloneable {
if (!getNoteReplLoader().getInterpreterSettings().isEmpty()) {
InterpreterSetting intpGroup = getNoteReplLoader().getInterpreterSettings().get(0);
- registry = intpGroup.getInterpreterGroup().getAngularObjectRegistry();
- resourcePool = intpGroup.getInterpreterGroup().getResourcePool();
+ registry = intpGroup.getInterpreterGroup(note.id()).getAngularObjectRegistry();
+ resourcePool = intpGroup.getInterpreterGroup(note.id()).getResourcePool();
}
List<InterpreterContextRunner> runners = new LinkedList<InterpreterContextRunner>();
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterFactoryTest.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterFactoryTest.java b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterFactoryTest.java
index d215e7c..742877a 100644
--- a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterFactoryTest.java
+++ b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterFactoryTest.java
@@ -87,8 +87,8 @@ public class InterpreterFactoryTest {
public void testBasic() {
List<String> all = factory.getDefaultInterpreterSettingList();
InterpreterSetting setting = factory.get(all.get(0));
- InterpreterGroup interpreterGroup = setting.getInterpreterGroup();
- factory.createInterpretersForNote(setting, "session");
+ InterpreterGroup interpreterGroup = setting.getInterpreterGroup("sharedProcess");
+ factory.createInterpretersForNote(setting, "sharedProcess", "session");
// get interpreter
Interpreter repl1 = interpreterGroup.get("session").get(0);
@@ -101,7 +101,7 @@ public class InterpreterFactoryTest {
// restart interpreter
factory.restart(all.get(0));
- assertNull(setting.getInterpreterGroup().get("session"));
+ assertNull(setting.getInterpreterGroup("sharedProcess").get("session"));
}
@Test
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NoteInterpreterLoaderTest.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NoteInterpreterLoaderTest.java b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NoteInterpreterLoaderTest.java
index aa98afd..f6abc9d 100644
--- a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NoteInterpreterLoaderTest.java
+++ b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NoteInterpreterLoaderTest.java
@@ -109,28 +109,68 @@ public class NoteInterpreterLoaderTest {
loaderB.getInterpreterSettings().get(0).getOption().setPerNoteSession(true);
// interpreters are not created before accessing it
- assertNull(loaderA.getInterpreterSettings().get(0).getInterpreterGroup().get("noteA"));
- assertNull(loaderB.getInterpreterSettings().get(0).getInterpreterGroup().get("noteB"));
+ assertNull(loaderA.getInterpreterSettings().get(0).getInterpreterGroup("shared_process").get("noteA"));
+ assertNull(loaderB.getInterpreterSettings().get(0).getInterpreterGroup("shared_process").get("noteB"));
+
+ loaderA.get(null).open();
+ loaderB.get(null).open();
- // per note session interpreter instance in the same interpreter process
assertTrue(
- loaderA.get(null).getInterpreterGroup().getRemoteInterpreterProcess() ==
- loaderB.get(null).getInterpreterGroup().getRemoteInterpreterProcess());
+ loaderA.get(null).getInterpreterGroup().getId().equals(
+ loaderB.get(null).getInterpreterGroup().getId()));
// interpreters are created after accessing it
- assertNotNull(loaderA.getInterpreterSettings().get(0).getInterpreterGroup().get("noteA"));
- assertNotNull(loaderB.getInterpreterSettings().get(0).getInterpreterGroup().get("noteB"));
+ assertNotNull(loaderA.getInterpreterSettings().get(0).getInterpreterGroup("shared_process").get("noteA"));
+ assertNotNull(loaderB.getInterpreterSettings().get(0).getInterpreterGroup("shared_process").get("noteB"));
// when
loaderA.close();
loaderB.close();
// interpreters are destroyed after close
- assertNull(loaderA.getInterpreterSettings().get(0).getInterpreterGroup().get("noteA"));
- assertNull(loaderB.getInterpreterSettings().get(0).getInterpreterGroup().get("noteB"));
+ assertNull(loaderA.getInterpreterSettings().get(0).getInterpreterGroup("shared_process").get("noteA"));
+ assertNull(loaderB.getInterpreterSettings().get(0).getInterpreterGroup("shared_process").get("noteB"));
+
+ }
+
+ @Test
+ public void testNotePerInterpreterProcess() throws IOException {
+ NoteInterpreterLoader loaderA = new NoteInterpreterLoader(factory);
+ loaderA.setNoteId("noteA");
+ loaderA.setInterpreters(factory.getDefaultInterpreterSettingList());
+ loaderA.getInterpreterSettings().get(0).getOption().setPerNoteProcess(true);
+
+ NoteInterpreterLoader loaderB = new NoteInterpreterLoader(factory);
+ loaderB.setNoteId("noteB");
+ loaderB.setInterpreters(factory.getDefaultInterpreterSettingList());
+ loaderB.getInterpreterSettings().get(0).getOption().setPerNoteProcess(true);
+ // interpreters are not created before accessing it
+ assertNull(loaderA.getInterpreterSettings().get(0).getInterpreterGroup("noteA").get("noteA"));
+ assertNull(loaderB.getInterpreterSettings().get(0).getInterpreterGroup("noteB").get("noteB"));
+
+ loaderA.get(null).open();
+ loaderB.get(null).open();
+
+ // per note interpreter process
+ assertFalse(
+ loaderA.get(null).getInterpreterGroup().getId().equals(
+ loaderB.get(null).getInterpreterGroup().getId()));
+
+ // interpreters are created after accessing it
+ assertNotNull(loaderA.getInterpreterSettings().get(0).getInterpreterGroup("noteA").get("noteA"));
+ assertNotNull(loaderB.getInterpreterSettings().get(0).getInterpreterGroup("noteB").get("noteB"));
+
+ // when
+ loaderA.close();
+ loaderB.close();
+
+ // interpreters are destroyed after close
+ assertNull(loaderA.getInterpreterSettings().get(0).getInterpreterGroup("noteA").get("noteA"));
+ assertNull(loaderB.getInterpreterSettings().get(0).getInterpreterGroup("noteB").get("noteB"));
}
+
private void delete(File file){
if(file.isFile()) file.delete();
else if(file.isDirectory()){
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/391ceab0/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
index a353328..c2c0338 100644
--- a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
+++ b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
@@ -364,7 +364,7 @@ public class NotebookTest implements JobListenerFactory{
note.getNoteReplLoader().setInterpreters(factory.getDefaultInterpreterSettingList());
AngularObjectRegistry registry = note.getNoteReplLoader()
- .getInterpreterSettings().get(0).getInterpreterGroup()
+ .getInterpreterSettings().get(0).getInterpreterGroup("sharedProcess")
.getAngularObjectRegistry();
Paragraph p1 = note.addParagraph();
@@ -397,7 +397,7 @@ public class NotebookTest implements JobListenerFactory{
note.getNoteReplLoader().setInterpreters(factory.getDefaultInterpreterSettingList());
AngularObjectRegistry registry = note.getNoteReplLoader()
- .getInterpreterSettings().get(0).getInterpreterGroup()
+ .getInterpreterSettings().get(0).getInterpreterGroup("sharedProcess")
.getAngularObjectRegistry();
Paragraph p1 = note.addParagraph();
@@ -430,7 +430,7 @@ public class NotebookTest implements JobListenerFactory{
note.getNoteReplLoader().setInterpreters(factory.getDefaultInterpreterSettingList());
AngularObjectRegistry registry = note.getNoteReplLoader()
- .getInterpreterSettings().get(0).getInterpreterGroup()
+ .getInterpreterSettings().get(0).getInterpreterGroup("sharedProcess")
.getAngularObjectRegistry();
// add local scope object
@@ -441,7 +441,7 @@ public class NotebookTest implements JobListenerFactory{
// restart interpreter
factory.restart(note.getNoteReplLoader().getInterpreterSettings().get(0).id());
registry = note.getNoteReplLoader()
- .getInterpreterSettings().get(0).getInterpreterGroup()
+ .getInterpreterSettings().get(0).getInterpreterGroup("sharedProcess")
.getAngularObjectRegistry();
// local and global scope object should be removed