You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by bz...@apache.org on 2016/05/31 14:34:28 UTC
incubator-zeppelin git commit: [ZEPPELIN-502] Python interpreter group
Repository: incubator-zeppelin
Updated Branches:
refs/heads/master 29723676b -> 34734b9c8
[ZEPPELIN-502] Python interpreter group
### What is this PR for?
Adding a python 2 &3 interpreter. It's a basic implementation (no py4j for example), with a java ProcessBuilder object used to instantiate a python REPL.
The interpreter doesn't bring it own python binary but uses the python specified by python.path configutation. Thus, you can still use your specific installed python modules (scikit-learn, matplotlib...) and the interpreter is able to work with python 2 & 3 without change.
I had a python helper function (zeppelin_show() ) to easily display matplotlib graph as SVG.
### What type of PR is it?
[Feature]
### Todos
* [x] - Code review
* [x] - Improve bootstrap.py : choose available helper functions and their names
* [x] - Unit / IT tests ?
* [x] documentation updates needed, that AhyoungRyu pointed out
* [X] LICENSE needs to be updated to include all non-apache licensed dependencies (i.e AFAIK Py4j is BSD ) in bin-license
* [x] double-check that code formatting conforms project style guide
* [x] the branch need to be rebased on latest master.
### What is the Jira issue?
[ZEPPELIN-502](https://issues.apache.org/jira/browse/ZEPPELIN-502?jql=project%20%3D%20ZEPPELIN%20AND%20text%20~%20%22python%22)
### How should this be tested?
1. In interpreter screen, in Python section, specify in python.path the python binary you want to use
2. In a paragraph, you can use the interpreter with **_%python_**. Calling help() will describe you the interpreter functionnalities.
3. Install py4j (pip install py4j) if you want to use input form
### Screenshots
### Questions:
* Does the licenses files need update? Yes, only bin-license (py4j)
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: Herv� RIVIERE <hr...@users.noreply.github.com>
Closes #869 from hriviere/PR_interpreter_python and squashes the following commits:
80b6e75 [Herv� RIVIERE] [ZEPPELIN-502] move BSD py4j license to zeppelin-distribution/src/bin_license/license
a4b82a5 [Herv� RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review
3252353 [Herv� RIVIERE] [ZEPPELIN-502] Formatting code to respect project convention
54ec4f1 [Herv� RIVIERE] [ZEPPELIN-502]Improving doc following @AhyoungRyu review
6a831bc [Herv� RIVIERE] [ZEPPELIN-502] Add BSD py4j license
11e1b9c [Herv� RIVIERE] [ZEPPELIN-502] minor changes in python.md
e5d0bdb [Herv� RIVIERE] [ZEPPELIN-502] change PYTHON_PATH to ZEPPELIN_PYTHON
c62ac98 [Herv� RIVIERE] [ZEPPELIN-502] Improve python.md
5008125 [Herv� RIVIERE] [ZEPPELIN-502] Improve python.md with features not yet supported and technical description
7d533e1 [Herv� RIVIERE] [ZEPPELIN-502] Add tests and reformating code to help tests writing
fecaf25 [Herv� RIVIERE] [ZEPPELIN-502] Rename python.path to python and default from /usr/bin/python to python
02d1320 [Herv� RIVIERE] [ZEPPELIN-502] Input form, change from simple input form to native (pyspark syntax)
60d2956 [Herv� RIVIERE] [ZEPPELIN-502] Indent as pep8 convention
9bdb192 [Herv� RIVIERE] [ZEPPELIN-502] Add python.md to _navigation.html
7142aa5 [Herv� RIVIERE] [ZEPPELIN-502] Catch exception in logger.error
1a86ad7 [Herv� RIVIERE] [ZEPPELIN-502] Python interpreter group
Project: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/commit/34734b9c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/tree/34734b9c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/diff/34734b9c
Branch: refs/heads/master
Commit: 34734b9c8ac6ba707a333356c5fb6b8baf127ca6
Parents: 2972367
Author: Herv� RIVIERE <hr...@users.noreply.github.com>
Authored: Mon May 30 22:07:26 2016 +0200
Committer: Alexander Bezzubov <bz...@apache.org>
Committed: Tue May 31 23:34:05 2016 +0900
----------------------------------------------------------------------
conf/zeppelin-site.xml.template | 2 +-
docs/_includes/themes/zeppelin/_navigation.html | 1 +
docs/interpreter/python.md | 117 +++++++++
.../screenshots/pythonMatplotlib.png | Bin 0 -> 20843 bytes
pom.xml | 1 +
python/pom.xml | 155 +++++++++++
.../zeppelin/python/PythonInterpreter.java | 262 +++++++++++++++++++
.../apache/zeppelin/python/PythonProcess.java | 133 ++++++++++
python/src/main/resources/bootstrap.py | 108 ++++++++
python/src/main/resources/bootstrap_input.py | 51 ++++
.../zeppelin/python/PythonInterpreterTest.java | 233 +++++++++++++++++
.../src/bin_license/licenses/LICENSE-py4j-0.9.2 | 26 ++
.../zeppelin/conf/ZeppelinConfiguration.java | 1 +
13 files changed, 1089 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/conf/zeppelin-site.xml.template
----------------------------------------------------------------------
diff --git a/conf/zeppelin-site.xml.template b/conf/zeppelin-site.xml.template
index bd61153..2d1fea1 100755
--- a/conf/zeppelin-site.xml.template
+++ b/conf/zeppelin-site.xml.template
@@ -178,7 +178,7 @@
<property>
<name>zeppelin.interpreters</name>
- <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.rinterpreter.RRepl,org.apache.zeppelin.rinterpreter.KnitR,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.file.HDFSFileInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.phoenix.PhoenixInterpreter,org.apache.zeppelin.k
ylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter,org.apache.zeppelin.alluxio.AlluxioInterpreter,org.apache.zeppelin.hbase.HbaseInterpreter,org.apache.zeppelin.livy.LivySparkInterpreter,org.apache.zeppelin.livy.LivyPySparkInterpreter,org.apache.zeppelin.livy.LivySparkRInterpreter,org.apache.zeppelin.livy.LivySparkSQLInterpreter</value>
+ <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.rinterpreter.RRepl,org.apache.zeppelin.rinterpreter.KnitR,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.file.HDFSFileInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,,org.apache.zeppelin.python.PythonInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.ph
oenix.PhoenixInterpreter,org.apache.zeppelin.kylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter,org.apache.zeppelin.alluxio.AlluxioInterpreter,org.apache.zeppelin.hbase.HbaseInterpreter,org.apache.zeppelin.livy.LivySparkInterpreter,org.apache.zeppelin.livy.LivyPySparkInterpreter,org.apache.zeppelin.livy.LivySparkRInterpreter,org.apache.zeppelin.livy.LivySparkSQLInterpreter</value>
<description>Comma separated interpreter configurations. First interpreter become a default</description>
</property>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/docs/_includes/themes/zeppelin/_navigation.html
----------------------------------------------------------------------
diff --git a/docs/_includes/themes/zeppelin/_navigation.html b/docs/_includes/themes/zeppelin/_navigation.html
index 8a46b74..5c84604 100644
--- a/docs/_includes/themes/zeppelin/_navigation.html
+++ b/docs/_includes/themes/zeppelin/_navigation.html
@@ -56,6 +56,7 @@
<li><a href="{{BASE_PATH}}/interpreter/lens.html">Lens</a></li>
<li><a href="{{BASE_PATH}}/interpreter/livy.html">Livy</a></li>
<li><a href="{{BASE_PATH}}/interpreter/markdown.html">Markdown</a></li>
+ <li><a href="{{BASE_PATH}}/interpreter/python.html">Python</a></li>
<li><a href="{{BASE_PATH}}/interpreter/postgresql.html">Postgresql, hawq</a></li>
<li><a href="{{BASE_PATH}}/interpreter/r.html">R</a></li>
<li><a href="{{BASE_PATH}}/interpreter/scalding.html">Scalding</a></li>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/docs/interpreter/python.md
----------------------------------------------------------------------
diff --git a/docs/interpreter/python.md b/docs/interpreter/python.md
new file mode 100644
index 0000000..c4fe756
--- /dev/null
+++ b/docs/interpreter/python.md
@@ -0,0 +1,117 @@
+---
+layout: page
+title: "Python Interpreter"
+description: "Python Interpreter"
+group: manual
+---
+{% include JB/setup %}
+
+## Python 2 & 3 Interpreter for Apache Zeppelin
+
+## Configuration
+<table class="table-configuration">
+ <tr>
+ <th>Property</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ <tr>
+ <td>python</td>
+ <td>python</td>
+ <td>Path of the already installed Python binary (could be python2 or python3).
+ If python is not in your $PATH you can set the absolute directory (example : /usr/bin/python)
+ </td>
+ </tr>
+</table>
+
+## Enabling Python Interpreter
+
+In a notebook, to enable the **Python** interpreter, click on the **Gear** icon and select **Python**
+
+## Using the Python Interpreter
+
+In a paragraph, use **_%python_** to select the **Python** interpreter and then input all commands.
+
+The interpreter can only work if you already have python installed (the interpreter doesn't bring it own python binaries).
+
+To access the help, type **help()**
+
+## Python modules
+The interpreter can use all modules already installed (with pip, easy_install...)
+
+## Apply Zeppelin Dynamic Forms
+You can leverage [Zeppelin Dynamic Form]({{BASE_PATH}}/manual/dynamicform.html) inside your Python code.
+
+**Zeppelin Dynamic Form can only be used if py4j Python library is installed in your system. If not, you can install it with `pip install py4j`.**
+
+Example :
+
+```python
+%python
+### Input form
+print (z.input("f1","defaultValue"))
+
+### Select form
+print (z.select("f1",[("o1","1"),("o2","2")],"2"))
+
+### Checkbox form
+print("".join(z.checkbox("f3", [("o1","1"), ("o2","2")],["1"])))
+```
+
+
+
+
+## Zeppelin features not fully supported by the Python Interpreter
+
+* Interrupt a paragraph execution (`cancel()` method) is currently only supported in Linux and MacOs. If interpreter runs in another operating system (for instance MS Windows) , interrupt a paragraph will close the whole interpreter. A JIRA ticket ([ZEPPELIN-893](https://issues.apache.org/jira/browse/ZEPPELIN-893)) is opened to implement this feature in a next release of the interpreter.
+* Progression bar in webUI (`getProgress()` method) is currently not implemented.
+* Code-completion is currently not implemented.
+
+## Matplotlib integration
+ The python interpreter can display matplotlib graph with the function **_zeppelin_show()_**
+ You need to already have matplotlib module installed and a running XServer to use this functionality !
+
+ ```python
+%python
+import matplotlib.pyplot as plt
+plt.figure()
+(.. ..)
+zeppelin_show(plt)
+plt.close()
+```
+zeppelin_show function can take optional parameters to adapt graph width and height
+
+ ```python
+%python
+zeppelin_show(plt,width='50px')
+zeppelin_show(plt,height='150px')
+```
+
+[](/docs/interpreter/screenshots/pythonMatplotlib.png)
+
+
+## Technical description - Interpreter architecture
+
+### Dev prerequisites
+
+* Python 2 and 3 installed with py4j (0.9.2) and matplotlib (1.31 or later) installed on each
+
+* Tests only checks the interpreter logic and starts any Python process ! Python process is mocked with a class that simply output it input.
+
+* Make sure the code wrote in bootstrap.py and bootstrap_input.py is Python2 and 3 compliant.
+
+* Use PEP8 convention for python code.
+
+### Technical overview
+
+ * When interpreter is starting it launches a python process inside a Java ProcessBuilder. Python is started with -i (interactive mode) and -u (unbuffered stdin, stdout and stderr) options. Thus the interpreter has a "sleeping" python process.
+
+ * Interpreter sends command to python with a Java `outputStreamWiter` and read from an `InputStreamReader`. To know when stop reading stdout, interpreter sends `print "*!?flush reader!?*"`after each command and reads stdout until he receives back the `*!?flush reader!?*`.
+
+ * When interpreter is starting, it sends some Python code (bootstrap.py and bootstrap_input.py) to initialize default behavior and functions (`help(), z.input()...`). bootstrap_input.py is sent only if py4j library is detected inside Python process.
+
+ * [Py4J](https://www.py4j.org/) python and java libraries is used to load Input zeppelin Java class into the python process (make java code with python code !). Therefore the interpreter can directly create Zeppelin input form inside the Python process (and eventually with some python variable already defined). JVM opens a random open port to be accessible from python process.
+
+ * JavaBuilder can't send SIGINT signal to interrupt paragraph execution. Therefore interpreter directly send a `kill SIGINT PID` to python process to interrupt execution. Python process catch SIGINT signal with some code defined in bootstrap.py
+
+ * Matplotlib display feature is made with SVG export (in string) and then displays it with html code.
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/docs/interpreter/screenshots/pythonMatplotlib.png
----------------------------------------------------------------------
diff --git a/docs/interpreter/screenshots/pythonMatplotlib.png b/docs/interpreter/screenshots/pythonMatplotlib.png
new file mode 100644
index 0000000..3d53043
Binary files /dev/null and b/docs/interpreter/screenshots/pythonMatplotlib.png differ
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 899ae80..833f219 100755
--- a/pom.xml
+++ b/pom.xml
@@ -82,6 +82,7 @@
<module>flink</module>
<module>ignite</module>
<module>kylin</module>
+ <module>python</module>
<module>lens</module>
<module>cassandra</module>
<module>elasticsearch</module>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/python/pom.xml
----------------------------------------------------------------------
diff --git a/python/pom.xml b/python/pom.xml
new file mode 100644
index 0000000..5b1b3a1
--- /dev/null
+++ b/python/pom.xml
@@ -0,0 +1,155 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <artifactId>zeppelin</artifactId>
+ <groupId>org.apache.zeppelin</groupId>
+ <version>0.6.0-incubating-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <groupId>org.apache.zeppelin</groupId>
+ <artifactId>zeppelin-python</artifactId>
+ <packaging>jar</packaging>
+ <version>0.6.0-incubating-SNAPSHOT</version>
+ <name>Zeppelin: Python interpreter</name>
+ <url>http://zeppelin.incubator.apache.org</url>
+
+ <properties>
+ <py4j.version>0.9.2</py4j.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>${project.groupId}</groupId>
+ <artifactId>zeppelin-interpreter</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-exec</artifactId>
+ <version>1.3</version>
+ </dependency>
+
+ <dependency>
+ <groupId>net.sf.py4j</groupId>
+ <artifactId>py4j</artifactId>
+ <version>${py4j.version}</version>
+ </dependency>
+
+
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-all</artifactId>
+ <version>1.9.0</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+
+
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-deploy-plugin</artifactId>
+ <version>2.7</version>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
+ </plugin>
+
+ <plugin>
+ <artifactId>maven-enforcer-plugin</artifactId>
+ <version>1.3.1</version>
+ <executions>
+ <execution>
+ <id>enforce</id>
+ <phase>none</phase>
+ </execution>
+ </executions>
+ </plugin>
+
+ <plugin>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <version>2.8</version>
+ <executions>
+ <execution>
+ <id>copy-dependencies</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.build.directory}/../../interpreter/python</outputDirectory>
+ <overWriteReleases>false</overWriteReleases>
+ <overWriteSnapshots>false</overWriteSnapshots>
+ <overWriteIfNewer>true</overWriteIfNewer>
+ <includeScope>runtime</includeScope>
+ </configuration>
+ </execution>
+ <execution>
+ <id>copy-artifact</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.build.directory}/../../interpreter/python</outputDirectory>
+ <overWriteReleases>false</overWriteReleases>
+ <overWriteSnapshots>false</overWriteSnapshots>
+ <overWriteIfNewer>true</overWriteIfNewer>
+ <includeScope>runtime</includeScope>
+ <artifactItems>
+ <artifactItem>
+ <groupId>${project.groupId}</groupId>
+ <artifactId>${project.artifactId}</artifactId>
+ <version>${project.version}</version>
+ <type>${project.packaging}</type>
+ </artifactItem>
+ </artifactItems>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+</project>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
----------------------------------------------------------------------
diff --git a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
new file mode 100644
index 0000000..23aeca2
--- /dev/null
+++ b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
@@ -0,0 +1,262 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.zeppelin.python;
+
+import org.apache.zeppelin.display.GUI;
+import org.apache.zeppelin.interpreter.Interpreter;
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterPropertyBuilder;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.InterpreterResult.Code;
+import org.apache.zeppelin.scheduler.Job;
+import org.apache.zeppelin.scheduler.Scheduler;
+import org.apache.zeppelin.scheduler.SchedulerFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import py4j.GatewayServer;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.net.ServerSocket;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * Python interpreter for Zeppelin.
+ */
+public class PythonInterpreter extends Interpreter {
+ Logger logger = LoggerFactory.getLogger(PythonInterpreter.class);
+
+ public static final String BOOTSTRAP_PY = "/bootstrap.py";
+ public static final String BOOTSTRAP_INPUT_PY = "/bootstrap_input.py";
+ public static final String ZEPPELIN_PYTHON = "zeppelin.python";
+ public static final String DEFAULT_ZEPPELIN_PYTHON = "python";
+
+ private Integer port;
+ private GatewayServer gatewayServer;
+ PythonProcess process = null;
+ private long pythonPid;
+ private Boolean py4J = false;
+ private InterpreterContext context;
+
+ static {
+ Interpreter.register(
+ "python",
+ "python",
+ PythonInterpreter.class.getName(),
+ new InterpreterPropertyBuilder()
+ .add(ZEPPELIN_PYTHON, DEFAULT_ZEPPELIN_PYTHON,
+ "Python directory. Default : python (assume python is in your $PATH)")
+ .build()
+ );
+ }
+
+
+ public PythonInterpreter(Properties property) {
+ super(property);
+ }
+
+ @Override
+ public void open() {
+
+ logger.info("Starting Python interpreter .....");
+
+
+ logger.info("Python path is set to:" + property.getProperty(ZEPPELIN_PYTHON));
+
+ process = getPythonProcess();
+
+ try {
+ process.open();
+ } catch (IOException e) {
+ logger.error("Can't start the python process", e);
+ }
+
+ try {
+ logger.info("python PID : " + process.getPid());
+ } catch (Exception e) {
+ logger.warn("Can't find python pid process", e);
+ }
+
+ try {
+ logger.info("Bootstrap interpreter with " + BOOTSTRAP_PY);
+ bootStrapInterpreter(BOOTSTRAP_PY);
+ } catch (IOException e) {
+ logger.error("Can't execute " + BOOTSTRAP_PY + " to initiate python process", e);
+ }
+
+ if (py4J = isPy4jInstalled()) {
+ port = findRandomOpenPortOnAllLocalInterfaces();
+ logger.info("Py4j gateway port : " + port);
+ try {
+ gatewayServer = new GatewayServer(this, port);
+ gatewayServer.start();
+ logger.info("Bootstrap inputs with " + BOOTSTRAP_INPUT_PY);
+ bootStrapInterpreter(BOOTSTRAP_INPUT_PY);
+ } catch (IOException e) {
+ logger.error("Can't execute " + BOOTSTRAP_INPUT_PY + " to " +
+ "initialize Zeppelin inputs in python process", e);
+ }
+ }
+
+
+ }
+
+ @Override
+ public void close() {
+
+ logger.info("closing Python interpreter .....");
+ try {
+ process.close();
+ gatewayServer.shutdown();
+ } catch (IOException e) {
+ logger.error("Can't close the interpreter", e);
+ }
+
+ }
+
+
+ @Override
+ public InterpreterResult interpret(String cmd, InterpreterContext contextInterpreter) {
+
+ this.context = contextInterpreter;
+
+ String output = sendCommandToPython(cmd);
+ return new InterpreterResult(Code.SUCCESS, output.replaceAll(">>>", "")
+ .replaceAll("\\.\\.\\.", "").trim());
+ }
+
+
+ @Override
+ public void cancel(InterpreterContext context) {
+ try {
+ process.interrupt();
+ } catch (IOException e) {
+ logger.error("Can't interrupt the python interpreter", e);
+ }
+ }
+
+ @Override
+ public FormType getFormType() {
+ return FormType.NATIVE;
+ }
+
+ @Override
+ public int getProgress(InterpreterContext context) {
+ return 0;
+ }
+
+ @Override
+ public Scheduler getScheduler() {
+ return SchedulerFactory.singleton().createOrGetParallelScheduler(
+ PythonInterpreter.class.getName() + this.hashCode(), 10);
+ }
+
+ @Override
+ public List<String> completion(String buf, int cursor) {
+ return null;
+ }
+
+ public PythonProcess getPythonProcess() {
+ if (process == null)
+ return new PythonProcess(getProperty(ZEPPELIN_PYTHON));
+ else
+ return process;
+ }
+
+ private Job getRunningJob(String paragraphId) {
+ Job foundJob = null;
+ Collection<Job> jobsRunning = getScheduler().getJobsRunning();
+ for (Job job : jobsRunning) {
+ if (job.getId().equals(paragraphId)) {
+ foundJob = job;
+ }
+ }
+ return foundJob;
+ }
+
+
+ private String sendCommandToPython(String cmd) {
+
+ String output = "";
+ logger.info("Sending : \n " + cmd);
+ try {
+ output = process.sendAndGetResult(cmd);
+ } catch (IOException e) {
+ logger.error("Error when sending commands to python process", e);
+ }
+
+ return output;
+ }
+
+
+ private void bootStrapInterpreter(String file) throws IOException {
+
+ BufferedReader bootstrapReader = new BufferedReader(
+ new InputStreamReader(
+ PythonInterpreter.class.getResourceAsStream(file)));
+ String line = null;
+ String bootstrapCode = "";
+
+ while ((line = bootstrapReader.readLine()) != null) {
+ bootstrapCode += line + "\n";
+ }
+ if (py4J && port != null && port != -1) {
+ bootstrapCode = bootstrapCode.replaceAll("\\%PORT\\%", port.toString());
+ }
+ logger.info("Bootstrap python interpreter with \n " + bootstrapCode);
+ sendCommandToPython(bootstrapCode);
+ }
+
+
+ public GUI getGui() {
+
+ return context.getGui();
+
+ }
+
+ public Integer getPy4JPort() {
+
+ return port;
+
+ }
+
+ public Boolean isPy4jInstalled() {
+
+ String output = sendCommandToPython("\n\nimport py4j\n");
+ if (output.contains("ImportError"))
+ return false;
+ else return true;
+
+ }
+
+ private int findRandomOpenPortOnAllLocalInterfaces() {
+ Integer port = -1;
+ try (ServerSocket socket = new ServerSocket(0);) {
+ port = socket.getLocalPort();
+ socket.close();
+ } catch (IOException e) {
+ logger.error("Can't find an open port", e);
+ }
+ return port;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/python/src/main/java/org/apache/zeppelin/python/PythonProcess.java
----------------------------------------------------------------------
diff --git a/python/src/main/java/org/apache/zeppelin/python/PythonProcess.java b/python/src/main/java/org/apache/zeppelin/python/PythonProcess.java
new file mode 100644
index 0000000..a671224
--- /dev/null
+++ b/python/src/main/java/org/apache/zeppelin/python/PythonProcess.java
@@ -0,0 +1,133 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.zeppelin.python;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.InputStreamReader;
+import java.lang.reflect.Field;
+
+/**
+ * Object encapsulated interactive
+ * Python process (REPL) used by python interpreter
+ */
+
+public class PythonProcess {
+
+ Logger logger = LoggerFactory.getLogger(PythonProcess.class);
+ InputStream stdout;
+ OutputStream stdin;
+ BufferedWriter writer;
+ BufferedReader reader;
+ Process process;
+ private String binPath;
+ private long pid;
+
+ public PythonProcess(String binPath) {
+ this.binPath = binPath;
+ }
+
+ public void open() throws IOException {
+ ProcessBuilder builder = new ProcessBuilder(binPath, "-iu");
+
+ builder.redirectErrorStream(true);
+ process = builder.start();
+ stdout = process.getInputStream();
+ stdin = process.getOutputStream();
+ writer = new BufferedWriter(new OutputStreamWriter(stdin));
+ reader = new BufferedReader(new InputStreamReader(stdout));
+ try {
+ pid = findPid();
+ } catch (Exception e) {
+ logger.warn("Can't find python pid process", e);
+ pid = -1;
+ }
+
+
+ }
+
+ public void close() throws IOException {
+
+ process.destroy();
+ reader.close();
+ writer.close();
+ stdin.close();
+ stdout.close();
+
+ }
+
+ public void interrupt() throws IOException {
+
+ if (pid > -1) {
+ logger.info("Sending SIGINT signal to PID : " + pid);
+ Runtime.getRuntime().exec("kill -SIGINT " + pid);
+ } else {
+ logger.warn("Non UNIX/Linux system, close the interpreter");
+ close();
+ }
+
+
+ }
+
+ public String sendAndGetResult(String cmd) throws IOException {
+
+ writer.write(cmd + "\n\n");
+ writer.write("print (\"*!?flush reader!?*\")\n\n");
+ writer.flush();
+
+ String output = "";
+ String line;
+ while (!(line = reader.readLine()).contains("*!?flush reader!?*")) {
+ logger.debug("Readed line from python shell : " + line);
+ if (line.equals("...")) {
+ logger.warn("Syntax error ! ");
+ output += "Syntax error ! ";
+ break;
+ }
+
+ output += "\r" + line + "\n";
+ }
+
+ return output;
+
+ }
+
+
+ private long findPid() throws NoSuchFieldException, IllegalAccessException {
+ long pid = -1;
+
+ if (process.getClass().getName().equals("java.lang.UNIXProcess")) {
+ Field f = process.getClass().getDeclaredField("pid");
+ f.setAccessible(true);
+ pid = f.getLong(process);
+ f.setAccessible(false);
+ }
+ return pid;
+ }
+
+ public long getPid() {
+ return pid;
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/python/src/main/resources/bootstrap.py
----------------------------------------------------------------------
diff --git a/python/src/main/resources/bootstrap.py b/python/src/main/resources/bootstrap.py
new file mode 100644
index 0000000..ee199f4
--- /dev/null
+++ b/python/src/main/resources/bootstrap.py
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# PYTHON 2 / 3 comptability :
+# bootstrap.py must be runnable with Python 2 and 3
+
+# Remove interactive mode displayhook
+import sys
+import signal
+try:
+ import StringIO as io
+except ImportError:
+ import io as io
+
+sys.displayhook = lambda x: None
+
+
+def intHandler(signum, frame): # Set the signal handler
+ print ("Paragraph interrupted")
+ raise KeyboardInterrupt()
+
+signal.signal(signal.SIGINT, intHandler)
+
+
+def help():
+ print ('%html')
+ print ('<h2>Python Interpreter help</h2>')
+ print ('<h3>Python 2 & 3 comptability</h3>')
+ print ('<p>The interpreter is compatible with Python 2 & 3.<br/>')
+ print ('To change Python version, ')
+ print ('change in the interpreter configuration the python to the ')
+ print ('desired version (example : python=/usr/bin/python3)</p>')
+ print ('<h3>Python modules</h3>')
+ print ('<p>The interpreter can use all modules already installed ')
+ print ('(with pip, easy_install, etc)</p>')
+ print ('<h3>Forms</h3>')
+ print ('You must install py4j in order to use '
+ 'the form feature (pip install py4j)')
+ print ('<h4>Input form</h4>')
+ print ('<pre>print (z.input("f1","defaultValue"))</pre>')
+ print ('<h4>Selection form</h4>')
+ print ('<pre>print(z.select("f2", [("o1","1"), ("o2","2")],2))</pre>')
+ print ('<h4>Checkbox form</h4>')
+ print ('<pre> print("".join(z.checkbox("f3", [("o1","1"), '
+ '("o2","2")],["1"])))</pre>')
+ print ('<h3>Matplotlib graph</h3>')
+ print ('<div>The interpreter can display matplotlib graph with ')
+ print ('the function zeppelin_show()</div>')
+ print ('<div> You need to already have matplotlib module installed ')
+ print ('to use this functionality !</div><br/>')
+ print ('''<pre>import matplotlib.pyplot as plt
+plt.figure()
+(.. ..)
+zeppelin_show(plt)
+plt.close()
+</pre>''')
+ print ('<div><br/> zeppelin_show function can take optional parameters ')
+ print ('to adapt graph width and height</div>')
+ print ("<div><b>example </b>:")
+ print('''<pre>zeppelin_show(plt,width='50px')
+zeppelin_show(plt,height='150px') </pre></div>''')
+
+
+# Matplotlib show function
+
+def zeppelin_show(p, width="0", height="0"):
+ img = io.StringIO()
+ p.savefig(img, format='svg')
+ img.seek(0)
+ style = ""
+ if(width != "0"):
+ style += 'width:'+width
+ if(height != "0"):
+ if(len(style) != 0):
+ style += ","
+ style += 'height:'+height
+ print("%html <div style='" + style + "'>" + img.read() + "<div>")
+
+
+# If py4j is detected, these class will be override
+# with the implementation in bootstrap_input.py
+
+class PyZeppelinContext():
+ errorMsg = "You must install py4j Python module " \
+ "(pip install py4j) to use Zeppelin dynamic forms features"
+ def __init__(self, zc):
+ self.z = zc
+ def input(self, name, defaultValue=""):
+ print (self.errorMsg)
+ def select(self, name, options, defaultValue=""):
+ print (self.errorMsg)
+ def checkbox(self, name, options, defaultChecked=[]):
+ print (self.errorMsg)
+
+z = PyZeppelinContext("")
+
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/python/src/main/resources/bootstrap_input.py
----------------------------------------------------------------------
diff --git a/python/src/main/resources/bootstrap_input.py b/python/src/main/resources/bootstrap_input.py
new file mode 100644
index 0000000..822434b
--- /dev/null
+++ b/python/src/main/resources/bootstrap_input.py
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from py4j.java_gateway import JavaGateway
+from py4j.java_gateway import java_import, JavaGateway, GatewayClient
+
+client = GatewayClient(port=%PORT%)
+gateway = JavaGateway(client)
+java_import(gateway.jvm, "org.apache.zeppelin.display.Input")
+
+
+class PyZeppelinContext():
+ paramOption = gateway.jvm.org.apache.zeppelin.display.Input.ParamOption
+ javaList = gateway.jvm.java.util.ArrayList
+ def __init__(self, zc):
+ self.z = zc
+ def input(self, name, defaultValue=""):
+ return self.z.getGui().input(name, defaultValue)
+ def select(self, name, options, defaultValue=""):
+ javaOptions = gateway.new_array(self.paramOption, len(options))
+ i = 0
+ for tuple in options:
+ javaOptions[i] = self.paramOption(tuple[0], tuple[1])
+ i += 1
+ return self.z.getGui().select(name, defaultValue, javaOptions)
+ def checkbox(self, name, options, defaultChecked=[]):
+ javaOptions = gateway.new_array(self.paramOption, len(options))
+ i = 0
+ for tuple in options:
+ javaOptions[i] = self.paramOption(tuple[0], tuple[1])
+ i += 1
+ javaDefaultCheck = self.javaList()
+ for check in defaultChecked:
+ javaDefaultCheck.append(check)
+ return self.z.getGui().checkbox(name, javaDefaultCheck, javaOptions)
+
+
+z = PyZeppelinContext(gateway.entry_point)
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterTest.java
----------------------------------------------------------------------
diff --git a/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterTest.java b/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterTest.java
new file mode 100644
index 0000000..2944903
--- /dev/null
+++ b/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterTest.java
@@ -0,0 +1,233 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.zeppelin.python;
+
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.mockito.Mockito.eq;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.anyString;
+
+import org.apache.zeppelin.interpreter.InterpreterResult;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.net.SocketAddress;
+import java.util.Properties;
+
+/**
+ * Python interpreter unit test
+ */
+public class PythonInterpreterTest {
+
+ Logger logger = LoggerFactory.getLogger(PythonProcess.class);
+
+ public static final String ZEPPELIN_PYTHON = "zeppelin.python";
+ public static final String DEFAULT_ZEPPELIN_PYTHON = "python";
+
+ PythonInterpreter pythonInterpreter = null;
+ PythonProcess mockPythonProcess;
+ String cmdHistory;
+
+ @Before
+ public void beforeTest() {
+ cmdHistory = "";
+
+ /*Mock python process*/
+ mockPythonProcess = mock(PythonProcess.class);
+ when(mockPythonProcess.getPid()).thenReturn((long) 1);
+ try {
+ when(mockPythonProcess.sendAndGetResult(anyString())).thenAnswer(
+ new Answer<String>() {
+ @Override
+ public String answer(InvocationOnMock invocationOnMock) throws Throwable {
+ return answerFromPythonMock(invocationOnMock);
+ }
+ });
+ } catch (IOException e) {
+ logger.error("Can't initiate python process", e);
+ }
+
+ Properties properties = new Properties();
+ properties.put(ZEPPELIN_PYTHON, DEFAULT_ZEPPELIN_PYTHON);
+ pythonInterpreter = spy(new PythonInterpreter(properties));
+
+ when(pythonInterpreter.getPythonProcess()).thenReturn(mockPythonProcess);
+
+
+ try {
+ when(mockPythonProcess.sendAndGetResult(eq("\n\nimport py4j\n"))).thenReturn("ImportError");
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+
+
+ }
+
+ @Test
+ public void testOpenInterpreter() {
+ pythonInterpreter.open();
+ assertEquals(pythonInterpreter.getPythonProcess().getPid(), 1);
+
+ }
+
+ @Test
+ public void testPy4jIsNotInstalled() {
+
+ /*
+ If Py4J is not installed, bootstrap_input.py
+ is not sent to Python process and
+ py4j JavaGateway is not running
+ */
+ pythonInterpreter.open();
+ assertNull(pythonInterpreter.getPy4JPort());
+
+ assertTrue(cmdHistory.contains("def help()"));
+ assertTrue(cmdHistory.contains("class PyZeppelinContext():"));
+ assertTrue(cmdHistory.contains("z = PyZeppelinContext"));
+ assertTrue(cmdHistory.contains("def zeppelin_show"));
+ assertFalse(cmdHistory.contains("GatewayClient"));
+
+ }
+
+ @Test
+ public void testPy4JInstalled() {
+
+
+ /*
+ If Py4J installed, bootstrap_input.py
+ is sent to interpreter and JavaGateway is
+ running
+ */
+
+ try {
+ when(mockPythonProcess.sendAndGetResult(eq("\n\nimport py4j\n"))).thenReturn(">>>");
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ pythonInterpreter.open();
+ Integer py4jPort = pythonInterpreter.getPy4JPort();
+ assertNotNull(py4jPort);
+
+ assertTrue(cmdHistory.contains("def help()"));
+ assertTrue(cmdHistory.contains("class PyZeppelinContext():"));
+ assertTrue(cmdHistory.contains("z = PyZeppelinContext"));
+ assertTrue(cmdHistory.contains("def zeppelin_show"));
+ assertTrue(cmdHistory.contains("GatewayClient(port=" + py4jPort + ")"));
+ assertTrue(cmdHistory.contains("org.apache.zeppelin.display.Input"));
+
+
+ assertTrue(checkSocketAdress(py4jPort));
+
+ }
+
+
+ @Test
+ public void testClose() {
+
+ try {
+ when(mockPythonProcess.sendAndGetResult(eq("\n\nimport py4j\n"))).thenReturn(">>>");
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ pythonInterpreter.open();
+ Integer py4jPort = pythonInterpreter.getPy4JPort();
+
+ assertNotNull(py4jPort);
+ pythonInterpreter.close();
+
+ assertFalse(checkSocketAdress(py4jPort));
+ try {
+ verify(mockPythonProcess, times(1)).close();
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+
+ @Test
+ public void testInterpret() {
+
+ pythonInterpreter.open();
+ cmdHistory = "";
+ InterpreterResult result = pythonInterpreter.interpret("print a", null);
+ assertEquals(InterpreterResult.Code.SUCCESS, result.code());
+ assertEquals("%text print a", result.toString());
+
+ }
+
+
+
+ private boolean checkSocketAdress(Integer py4jPort) {
+ Socket s = new Socket();
+ SocketAddress sa = new InetSocketAddress("localhost", py4jPort);
+ Boolean working = null;
+ try {
+ s.connect(sa, 10000);
+ } catch (IOException e) {
+ working = false;
+ }
+
+ if (working == null) {
+ working = s.isConnected();
+ try {
+ s.close();
+ } catch (IOException e) {
+ logger.error("Can't close connection to localhost:" + py4jPort, e);
+ }
+ }
+ return working;
+ }
+
+
+
+ private String answerFromPythonMock(InvocationOnMock invocationOnMock) {
+ Object[] inputs = invocationOnMock.getArguments();
+ String cmdToExecute = (String) inputs[0];
+
+ if (cmdToExecute != null) {
+ cmdHistory += cmdToExecute;
+ String[] lines = cmdToExecute.split("\\n");
+ String output = "";
+
+ for (int i = 0; i < lines.length; i++) {
+ output += ">>>" + lines[i];
+ }
+ return output;
+ } else {
+ return ">>>";
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/zeppelin-distribution/src/bin_license/licenses/LICENSE-py4j-0.9.2
----------------------------------------------------------------------
diff --git a/zeppelin-distribution/src/bin_license/licenses/LICENSE-py4j-0.9.2 b/zeppelin-distribution/src/bin_license/licenses/LICENSE-py4j-0.9.2
new file mode 100644
index 0000000..48a8c28
--- /dev/null
+++ b/zeppelin-distribution/src/bin_license/licenses/LICENSE-py4j-0.9.2
@@ -0,0 +1,26 @@
+Copyright (c) 2009-2016, Barthelemy Dagenais and individual contributors. All
+rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+- Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+- Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+- The name of the author may not be used to endorse or promote products
+ derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/34734b9c/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
index 5f62a53..a07c3fa 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
@@ -486,6 +486,7 @@ public class ZeppelinConfiguration extends XMLConfiguration {
+ "org.apache.zeppelin.postgresql.PostgreSqlInterpreter,"
+ "org.apache.zeppelin.tajo.TajoInterpreter,"
+ "org.apache.zeppelin.flink.FlinkInterpreter,"
+ + "org.apache.zeppelin.python.PythonInterpreter,"
+ "org.apache.zeppelin.ignite.IgniteInterpreter,"
+ "org.apache.zeppelin.ignite.IgniteSqlInterpreter,"
+ "org.apache.zeppelin.lens.LensInterpreter,"