You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by pr...@apache.org on 2016/05/24 05:00:20 UTC
incubator-zeppelin git commit: ZEPPELIN-773 : Livy interperter
Repository: incubator-zeppelin
Updated Branches:
refs/heads/master 40137ab68 -> 6acd0aee9
ZEPPELIN-773 : Livy interperter
### What is this PR for?
As Zeppelin evolves its notebook, for large scale data analysis, multiple zeppelin users are expected to use and connect to the same set of data repositories within an enterprise. Since Zeppelin notebooks could affect data, state and its lineage, it is important to have separation of users, provide them with appropriate sandboxes, in addition to capturing the right audit details. Further, the IT within the organization would prefer to support fewer Zeppelin instances (preferably one) to support its customers. Therefore, the objectives of creating a multi-tenant zeppelin are:
\u25cf Supporting workloads of multiple customers
\u25cf Supporting multiple LOBs (lines of business), on a single data systems
\u25cf Support fine grained audits
As a natural evolution of Zeppelin Authentication and Authorization design, partly user awareness in downstream data systems such as Spark/Hive and others, is essential to achieve the above stated objectives.
### What type of PR is it?
Feature
### Todos
* [x] - Test case
* [x] - Review Comments
* [x] - Documentation
### What is the Jira issue?
ZEPPELIN-773
### How should this be tested?
- Install Livy by following steps on https://github.com/cloudera/livy
- Start the Livy server
- Now by using Zeppelin-Livy interpreter, run any of the spark, pyspark or R commands.
### Screenshots (if appropriate)
<img width="1436" alt="screen shot 2016-04-11 at 12 41 35 pm" src="https://cloud.githubusercontent.com/assets/674497/14419479/b514979c-ffe3-11e5-9dea-df9854d8409c.png">
<img width="1434" alt="screen shot 2016-04-11 at 12 41 59 pm" src="https://cloud.githubusercontent.com/assets/674497/14419478/b514922e-ffe3-11e5-9c98-93c5b99de106.png">
<img width="1440" alt="screen shot 2016-04-11 at 12 48 13 pm" src="https://cloud.githubusercontent.com/assets/674497/14419480/b515d8c8-ffe3-11e5-8c20-4c988c621f51.png">
### Questions:
* Does the licenses files need update? n/a
* Is there breaking changes for older versions? n/a
* Does this needs documentation? yes
Author: Prabhjyot Singh <pr...@gmail.com>
Author: Rohit Choudhary <rc...@gmail.com>
Closes #827 from prabhjyotsingh/livyInterperter and squashes the following commits:
9689da0 [Prabhjyot Singh] check for more session not found error
aeb5a73 [Prabhjyot Singh] update doc with review comment and add FAQ
5c2bf13 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
9833c59 [Prabhjyot Singh] ZEPPELIN-773: log error in all other status code
3deca71 [Prabhjyot Singh] ZEPPELIN-773 update doc for know issue, and more error logging
6f1503f [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
44c5e82 [Prabhjyot Singh] ZEPPELIN-773: fail check if API allows master or conf in parameter
f2ea724 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
53f2804 [Prabhjyot Singh] ZEPPELIN-773: add doc for livy impersonation
8095b3b [Prabhjyot Singh] ZZEPPELIN-773: add doc for spark version
fef1081 [Prabhjyot Singh] ZEPPELIN-773 add documentation for configuring Spark master uri.
23b7811 [Prabhjyot Singh] ZEPPELIN-773 livy to have conf for configuring yarn master
134923d [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
7a12336 [Prabhjyot Singh] missing exception handeling in LivySparkInterpreter
200e715 [Prabhjyot Singh] more exception handeling
1e18465 [Prabhjyot Singh] LOGGER.error
8116b72 [Prabhjyot Singh] remove redundant getResultCode
93708cd [Prabhjyot Singh] setting the right condition
c3e74f2 [Prabhjyot Singh] return error
ad26d0b [Prabhjyot Singh] replace info with error
7f6fa24 [Prabhjyot Singh] check/incorporate recent changes of this code in the Spark interpreter
6c19e35 [Prabhjyot Singh] retry a for a minute and fail, instead of looping forever
45e3d48 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
97d0663 [Prabhjyot Singh] add the doc to docs/_includes/themes/zeppelin/_navigation.html
7ff9744 [Prabhjyot Singh] update default value to 1000
a6e7d0b [Prabhjyot Singh] prefix zeppelin. to property zeppelin.livy.sql.maxResult
9be64e0 [Prabhjyot Singh] doc
5f9be73 [Prabhjyot Singh] adding more mock test
8c4b983 [Prabhjyot Singh] remove class name as it will be avail with the stack trace
a3b0a06 [Prabhjyot Singh] add test for pyspark
f0e3c20 [Prabhjyot Singh] change variable name
bbe2a7c [Prabhjyot Singh] rename lspark to livy
9bfbe47 [Prabhjyot Singh] close livy session on connection close/restart interpreter
84bd755 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
cb65c86 [Prabhjyot Singh] rename r to sparkr for consistent naming
eb8706f [Prabhjyot Singh] fix paragraph abort
1b79c07 [Prabhjyot Singh] more loging and exception handeling
9a84e11 [Prabhjyot Singh] revert merge conflict
ff3c4ed [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
40ce7cd [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
3863682 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
7cec9ba [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
a28d674 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
557d1e1 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
e43385e [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
dc0a3dc [Prabhjyot Singh] fix append "%html "
756558e [Prabhjyot Singh] check for html tags and append "%html "
c58eae7 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
6c6b164 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
01ec474 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
32fbc1a [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
18468a0 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
ca06e91 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
5bb5775 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
5eb4eff [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
78eca1e [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
ee1c9f4 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
ea05fe9 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
0fbb74b [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
dadc257 [Prabhjyot Singh] reverting zeppelinConfiguration
b8e1779 [Prabhjyot Singh] adding back LivySparkRInterpreter.java , pyspark
9d89b0d [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
948615a [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
4f513a5 [Prabhjyot Singh] site.xml
426bbe8 [Rohit Choudhary] Merge branch 'livyInterperter' of https://github.com/prabhjyotsingh/incubator-zeppelin into livyInterperter
68f438d [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into livyInterperter
ee2dceb [Prabhjyot Singh] working spark sql
1f9a111 [Rohit Choudhary] remove references to LivyInterpreters
b53fd8b [Rohit Choudhary] Don't need so many interpreters
de2fd3c [Prabhjyot Singh] have spark streaming
9cb0819 [Prabhjyot Singh] Fix for 1st request failing
8f4ec47 [Prabhjyot Singh] removing unrequired logs
07f0846 [Prabhjyot Singh] with lspark sql
10311d3 [Prabhjyot Singh] This works in all cases
d0519d5 [Prabhjyot Singh] Working livy interpreter with spark, pyspark, R
ace28a8 [Prabhjyot Singh] working livy
4053497 [Prabhjyot Singh] test livy
0709b9c [Prabhjyot Singh] moving AuthenticationInfo to org.apache.zeppelin.display.AuthenticationInfo
95e7c13 [Prabhjyot Singh] test for selenium
a5a991d [Prabhjyot Singh] check for selenium
34dcc32 [Prabhjyot Singh] instead of null pass "new AuthenticationInfo()"
ba91da4 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into UserInInterpreterContext
57ca577 [Prabhjyot Singh] review change create such class AuthenticationInfo, and pass it into InterpreterContext
320790c [Prabhjyot Singh] fix for CI, missing change signature
d928203 [Prabhjyot Singh] revert shiri.ini if fromMessage.principal.equals("anonymous") then set user as null
fadc6d9 [Prabhjyot Singh] userName to be present in InterpreterContext/RemoteInterpreterContext
Project: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/commit/6acd0aee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/tree/6acd0aee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/diff/6acd0aee
Branch: refs/heads/master
Commit: 6acd0aee94b9e8552c907d9d8f4e9b84bcb81be0
Parents: 40137ab
Author: Prabhjyot Singh <pr...@gmail.com>
Authored: Thu May 19 13:37:09 2016 +0530
Committer: Prabhjyot Singh <pr...@gmail.com>
Committed: Tue May 24 10:30:12 2016 +0530
----------------------------------------------------------------------
conf/zeppelin-site.xml.template | 2 +-
docs/_includes/themes/zeppelin/_navigation.html | 1 +
docs/interpreter/livy.md | 107 +++++
livy/pom.xml | 170 ++++++++
.../org/apache/zeppelin/livy/LivyHelper.java | 409 +++++++++++++++++++
.../apache/zeppelin/livy/LivyOutputStream.java | 75 ++++
.../zeppelin/livy/LivyPySparkInterpreter.java | 122 ++++++
.../zeppelin/livy/LivySparkInterpreter.java | 135 ++++++
.../zeppelin/livy/LivySparkRInterpreter.java | 122 ++++++
.../zeppelin/livy/LivySparkSQLInterpreter.java | 165 ++++++++
.../apache/zeppelin/livy/LivyHelperTest.java | 114 ++++++
.../zeppelin/livy/LivyInterpreterTest.java | 86 ++++
pom.xml | 1 +
.../zeppelin/conf/ZeppelinConfiguration.java | 4 +
14 files changed, 1512 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/conf/zeppelin-site.xml.template
----------------------------------------------------------------------
diff --git a/conf/zeppelin-site.xml.template b/conf/zeppelin-site.xml.template
index f756dca..caf0900 100755
--- a/conf/zeppelin-site.xml.template
+++ b/conf/zeppelin-site.xml.template
@@ -144,7 +144,7 @@
<property>
<name>zeppelin.interpreters</name>
- <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.rinterpreter.RRepl,org.apache.zeppelin.rinterpreter.KnitR,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.file.HDFSFileInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.phoenix.PhoenixInterpreter,org.apache.zeppelin.k
ylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter,org.apache.zeppelin.alluxio.AlluxioInterpreter,org.apache.zeppelin.hbase.HbaseInterpreter</value>
+ <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.rinterpreter.RRepl,org.apache.zeppelin.rinterpreter.KnitR,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.file.HDFSFileInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.phoenix.PhoenixInterpreter,org.apache.zeppelin.k
ylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter,org.apache.zeppelin.alluxio.AlluxioInterpreter,org.apache.zeppelin.hbase.HbaseInterpreter,org.apache.zeppelin.livy.LivySparkInterpreter,org.apache.zeppelin.livy.LivyPySparkInterpreter,org.apache.zeppelin.livy.LivySparkRInterpreter,org.apache.zeppelin.livy.LivySparkSQLInterpreter</value>
<description>Comma separated interpreter configurations. First interpreter become a default</description>
</property>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/docs/_includes/themes/zeppelin/_navigation.html
----------------------------------------------------------------------
diff --git a/docs/_includes/themes/zeppelin/_navigation.html b/docs/_includes/themes/zeppelin/_navigation.html
index 828efba..1884a23 100644
--- a/docs/_includes/themes/zeppelin/_navigation.html
+++ b/docs/_includes/themes/zeppelin/_navigation.html
@@ -54,6 +54,7 @@
<li><a href="{{BASE_PATH}}/interpreter/ignite.html">Ignite</a></li>
<li><a href="{{BASE_PATH}}/interpreter/jdbc.html">JDBC</a></li>
<li><a href="{{BASE_PATH}}/interpreter/lens.html">Lens</a></li>
+ <li><a href="{{BASE_PATH}}/interpreter/livy.html">Livy</a></li>
<li><a href="{{BASE_PATH}}/interpreter/markdown.html">Markdown</a></li>
<li><a href="{{BASE_PATH}}/interpreter/postgresql.html">Postgresql, hawq</a></li>
<li><a href="{{BASE_PATH}}/interpreter/r.html">R</a></li>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/docs/interpreter/livy.md
----------------------------------------------------------------------
diff --git a/docs/interpreter/livy.md b/docs/interpreter/livy.md
new file mode 100644
index 0000000..295a508
--- /dev/null
+++ b/docs/interpreter/livy.md
@@ -0,0 +1,107 @@
+---
+layout: page
+title: "Livy Interpreter"
+description: ""
+group: manual
+---
+{% include JB/setup %}
+
+## Livy Interpreter for Apache Zeppelin
+Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN.
+
+* Interactive Scala, Python and R shells
+* Batch submissions in Scala, Java, Python
+* Multi users can share the same server (impersonation support)
+* Can be used for submitting jobs from anywhere with REST
+* Does not require any code change to your programs
+
+### Requirements
+
+Additional requirements for the Livy interpreter are:
+
+ * Spark 1.3 or above.
+ * Livy server.
+
+### Configuration
+<table class="table-configuration">
+ <tr>
+ <th>Property</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ <tr>
+ <td>zeppelin.livy.master</td>
+ <td>local[*]</td>
+ <td>Spark master uri. ex) spark://masterhost:7077</td>
+ </tr>
+ <tr>
+ <td>zeppelin.livy.url</td>
+ <td>http://localhost:8998</td>
+ <td>URL where livy server is running</td>
+ </tr>
+ <tr>
+ <td>zeppelin.livy.spark.maxResult</td>
+ <td>1000</td>
+ <td>Max number of SparkSQL result to display.</td>
+ </tr>
+</table>
+
+
+
+## How to use
+Basically, you can use
+
+**spark**
+
+```
+%livy.spark
+sc.version
+```
+
+
+**pyspark**
+
+```
+%livy.pyspark
+print "1"
+```
+
+**sparkR**
+
+```
+%livy.sparkr
+hello <- function( name ) {
+ sprintf( "Hello, %s", name );
+}
+
+hello("livy")
+```
+
+## Impersonation
+When Zeppelin server is running with authentication enabled, then this interpreter utilizes Livy\u2019s user impersonation feature i.e. sends extra parameter for creating and running a session ("proxyUser": "${loggedInUser}"). This is particularly useful when multi users are sharing a Notebook server.
+
+
+### Apply Zeppelin Dynamic Forms
+You can leverage [Zeppelin Dynamic Form]({{BASE_PATH}}/manual/dynamicform.html). You can use both the `text input` and `select form` parameterization features.
+
+```
+%livy.pyspark
+print "${group_by=product_id,product_id|product_name|customer_id|store_id}"
+```
+
+## FAQ
+
+Livy debugging: If you see any of these in error console
+
+> Connect to livyhost:8998 [livyhost/127.0.0.1, livyhost/0:0:0:0:0:0:0:1] failed: Connection refused
+
+Looks like the livy server is not up yet or the config is wrong
+
+> Exception: Session not found, Livy server would have restarted, or lost session.
+
+The session would have timed out, you may need to restart the interpreter.
+
+
+> Blacklisted configuration values in session config: spark.master
+
+edit `conf/spark-blacklist.conf` file in livy server and comment out `#spark.master` line.
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/livy/pom.xml
----------------------------------------------------------------------
diff --git a/livy/pom.xml b/livy/pom.xml
new file mode 100644
index 0000000..6ad9622
--- /dev/null
+++ b/livy/pom.xml
@@ -0,0 +1,170 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <artifactId>zeppelin</artifactId>
+ <groupId>org.apache.zeppelin</groupId>
+ <version>0.6.0-incubating-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <groupId>org.apache.zeppelin</groupId>
+ <artifactId>zeppelin-livy</artifactId>
+ <packaging>jar</packaging>
+ <version>0.6.0-incubating-SNAPSHOT</version>
+ <name>Zeppelin: Livy interpreter</name>
+ <url>http://zeppelin.incubator.apache.org</url>
+
+ <properties>
+ <!--TEST-->
+ <junit.version>4.12</junit.version>
+ <achilles.version>3.2.4-Zeppelin</achilles.version>
+ <assertj.version>1.7.0</assertj.version>
+ <mockito.version>1.9.5</mockito.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>${project.groupId}</groupId>
+ <artifactId>zeppelin-interpreter</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-exec</artifactId>
+ <version>1.3</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>4.3.4</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.google.code.gson</groupId>
+ <artifactId>gson</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.assertj</groupId>
+ <artifactId>assertj-core</artifactId>
+ <version>${assertj.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-all</artifactId>
+ <version>${mockito.version}</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-deploy-plugin</artifactId>
+ <version>2.7</version>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
+ </plugin>
+
+ <plugin>
+ <artifactId>maven-enforcer-plugin</artifactId>
+ <version>1.3.1</version>
+ <executions>
+ <execution>
+ <id>enforce</id>
+ <phase>none</phase>
+ </execution>
+ </executions>
+ </plugin>
+
+ <plugin>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <version>2.8</version>
+ <executions>
+ <execution>
+ <id>copy-dependencies</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.build.directory}/../../interpreter/livy
+ </outputDirectory>
+ <overWriteReleases>false</overWriteReleases>
+ <overWriteSnapshots>false</overWriteSnapshots>
+ <overWriteIfNewer>true</overWriteIfNewer>
+ <includeScope>runtime</includeScope>
+ </configuration>
+ </execution>
+ <execution>
+ <id>copy-artifact</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.build.directory}/../../interpreter/livy
+ </outputDirectory>
+ <overWriteReleases>false</overWriteReleases>
+ <overWriteSnapshots>false</overWriteSnapshots>
+ <overWriteIfNewer>true</overWriteIfNewer>
+ <includeScope>runtime</includeScope>
+ <artifactItems>
+ <artifactItem>
+ <groupId>${project.groupId}</groupId>
+ <artifactId>${project.artifactId}</artifactId>
+ <version>${project.version}</version>
+ <type>${project.packaging}</type>
+ </artifactItem>
+ </artifactItems>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+</project>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java
----------------------------------------------------------------------
diff --git a/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java b/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java
new file mode 100644
index 0000000..27fc422
--- /dev/null
+++ b/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.livy;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.google.gson.reflect.TypeToken;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.InterpreterResult.Code;
+import org.apache.zeppelin.interpreter.InterpreterUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.InputStreamReader;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+
+/***
+ * Livy helper class
+ */
+public class LivyHelper {
+ Logger LOGGER = LoggerFactory.getLogger(LivyHelper.class);
+ Gson gson = new GsonBuilder().setPrettyPrinting().create();
+ HashMap<String, Object> paragraphHttpMap = new HashMap<>();
+ Properties property;
+ Integer MAX_NOS_RETRY = 60;
+
+ LivyHelper(Properties property) {
+ this.property = property;
+ }
+
+ public Integer createSession(InterpreterContext context, String kind) throws Exception {
+ try {
+ String json = executeHTTP(property.getProperty("zeppelin.livy.url") + "/sessions",
+ "POST",
+ "{" +
+ "\"kind\": \"" + kind + "\", " +
+ "\"master\": \"" + property.getProperty("zeppelin.livy.master") + "\", " +
+ "\"proxyUser\": \"" + context.getAuthenticationInfo().getUser() + "\"" +
+ "}",
+ context.getParagraphId()
+ );
+ if (json.contains("CreateInteractiveRequest[\\\"master\\\"]")) {
+ json = executeHTTP(property.getProperty("zeppelin.livy.url") + "/sessions",
+ "POST",
+ "{" +
+ "\"kind\": \"" + kind + "\", " +
+ "\"conf\":{\"spark.master\": \""
+ + property.getProperty("zeppelin.livy.master") + "\"}," +
+ "\"proxyUser\": \"" + context.getAuthenticationInfo().getUser() + "\"" +
+ "}",
+ context.getParagraphId()
+ );
+ }
+ Map jsonMap = (Map<Object, Object>) gson.fromJson(json,
+ new TypeToken<Map<Object, Object>>() {
+ }.getType());
+ Integer sessionId = ((Double) jsonMap.get("id")).intValue();
+ if (!jsonMap.get("state").equals("idle")) {
+ Integer nosRetry = MAX_NOS_RETRY;
+
+ while (nosRetry >= 0) {
+ LOGGER.error(String.format("sessionId:%s state is %s",
+ jsonMap.get("id"), jsonMap.get("state")));
+ Thread.sleep(1000);
+ json = executeHTTP(property.getProperty("zeppelin.livy.url") + "/sessions/" + sessionId,
+ "GET", null,
+ context.getParagraphId());
+ jsonMap = (Map<Object, Object>) gson.fromJson(json,
+ new TypeToken<Map<Object, Object>>() {
+ }.getType());
+ if (jsonMap.get("state").equals("idle")) {
+ break;
+ } else if (jsonMap.get("state").equals("error")) {
+ json = executeHTTP(property.getProperty("zeppelin.livy.url") + "/sessions/" +
+ sessionId + "/log",
+ "GET", null,
+ context.getParagraphId());
+ jsonMap = (Map<Object, Object>) gson.fromJson(json,
+ new TypeToken<Map<Object, Object>>() {
+ }.getType());
+ String logs = StringUtils.join((ArrayList<String>) jsonMap.get("log"), '\n');
+ LOGGER.error(String.format("Cannot start %s.\n%s", kind, logs));
+ throw new Exception(String.format("Cannot start %s.\n%s", kind, logs));
+ }
+ nosRetry--;
+ }
+ if (nosRetry <= 0) {
+ LOGGER.error("Error getting session for user within 60Sec.");
+ throw new Exception(String.format("Cannot start %s.", kind));
+ }
+ }
+ return sessionId;
+ } catch (Exception e) {
+ LOGGER.error("Error getting session for user", e);
+ throw e;
+ }
+ }
+
+ protected void initializeSpark(final InterpreterContext context,
+ final Map<String, Integer> userSessionMap) throws Exception {
+ interpret("val sqlContext= new org.apache.spark.sql.SQLContext(sc)\n" +
+ "import sqlContext.implicits._", context, userSessionMap);
+ }
+
+ public InterpreterResult interpretInput(String stringLines,
+ final InterpreterContext context,
+ final Map<String, Integer> userSessionMap,
+ LivyOutputStream out) {
+ try {
+ String[] lines = stringLines.split("\n");
+ String[] linesToRun = new String[lines.length + 1];
+ for (int i = 0; i < lines.length; i++) {
+ linesToRun[i] = lines[i];
+ }
+ linesToRun[lines.length] = "print(\"\")";
+
+ out.setInterpreterOutput(context.out);
+ context.out.clear();
+ Code r = null;
+ String incomplete = "";
+ boolean inComment = false;
+
+ for (int l = 0; l < linesToRun.length; l++) {
+ String s = linesToRun[l];
+ // check if next line starts with "." (but not ".." or "./") it is treated as an invocation
+ //for spark
+ if (l + 1 < linesToRun.length) {
+ String nextLine = linesToRun[l + 1].trim();
+ boolean continuation = false;
+ if (nextLine.isEmpty()
+ || nextLine.startsWith("//") // skip empty line or comment
+ || nextLine.startsWith("}")
+ || nextLine.startsWith("object")) { // include "} object" for Scala companion object
+ continuation = true;
+ } else if (!inComment && nextLine.startsWith("/*")) {
+ inComment = true;
+ continuation = true;
+ } else if (inComment && nextLine.lastIndexOf("*/") >= 0) {
+ inComment = false;
+ continuation = true;
+ } else if (nextLine.length() > 1
+ && nextLine.charAt(0) == '.'
+ && nextLine.charAt(1) != '.' // ".."
+ && nextLine.charAt(1) != '/') { // "./"
+ continuation = true;
+ } else if (inComment) {
+ continuation = true;
+ }
+ if (continuation) {
+ incomplete += s + "\n";
+ continue;
+ }
+ }
+
+ InterpreterResult res;
+ try {
+ res = interpret(incomplete + s, context, userSessionMap);
+ } catch (Exception e) {
+ LOGGER.error("Interpreter exception", e);
+ return new InterpreterResult(Code.ERROR, InterpreterUtils.getMostRelevantMessage(e));
+ }
+
+ r = res.code();
+
+ if (r == Code.ERROR) {
+ out.setInterpreterOutput(null);
+ return res;
+ } else if (r == Code.INCOMPLETE) {
+ incomplete += s + "\n";
+ } else {
+ out.write((res.message() + "\n").getBytes(Charset.forName("UTF-8")));
+ incomplete = "";
+ }
+ }
+
+ if (r == Code.INCOMPLETE) {
+ out.setInterpreterOutput(null);
+ return new InterpreterResult(r, "Incomplete expression");
+ } else {
+ out.setInterpreterOutput(null);
+ return new InterpreterResult(Code.SUCCESS);
+ }
+
+ } catch (Exception e) {
+ LOGGER.error("error in interpretInput", e);
+ return new InterpreterResult(Code.ERROR, e.getMessage());
+ }
+ }
+
+ public InterpreterResult interpret(String stringLines,
+ final InterpreterContext context,
+ final Map<String, Integer> userSessionMap)
+ throws Exception {
+ stringLines = stringLines
+ //for "\n" present in string
+ .replaceAll("\\\\n", "\\\\\\\\n")
+ //for new line present in string
+ .replaceAll("\\n", "\\\\n")
+ // for \" present in string
+ .replaceAll("\\\\\"", "\\\\\\\\\"")
+ // for " present in string
+ .replaceAll("\"", "\\\\\"");
+
+ if (stringLines.trim().equals("")) {
+ return new InterpreterResult(Code.SUCCESS, "");
+ }
+ Map jsonMap = executeCommand(stringLines, context, userSessionMap);
+ Integer id = ((Double) jsonMap.get("id")).intValue();
+ InterpreterResult res = getResultFromMap(jsonMap);
+ if (res != null) {
+ return res;
+ }
+
+ while (true) {
+ Thread.sleep(1000);
+ if (paragraphHttpMap.get(context.getParagraphId()) == null) {
+ return new InterpreterResult(Code.INCOMPLETE, "");
+ }
+ jsonMap = getStatusById(context, userSessionMap, id);
+ InterpreterResult interpreterResult = getResultFromMap(jsonMap);
+ if (interpreterResult != null) {
+ return interpreterResult;
+ }
+ }
+ }
+
+ private InterpreterResult getResultFromMap(Map jsonMap) {
+ if (jsonMap.get("state").equals("available")) {
+ if (((Map) jsonMap.get("output")).get("status").equals("error")) {
+ StringBuilder errorMessage = new StringBuilder((String) ((Map) jsonMap
+ .get("output")).get("evalue"));
+ if (errorMessage.toString().equals("incomplete statement")
+ || errorMessage.toString().contains("EOF")) {
+ return new InterpreterResult(Code.INCOMPLETE, "");
+ }
+ String traceback = gson.toJson(((Map) jsonMap.get("output")).get("traceback"));
+ if (!traceback.equals("[]")) {
+ errorMessage
+ .append("\n")
+ .append("traceback: \n")
+ .append(traceback);
+ }
+
+ return new InterpreterResult(Code.ERROR, errorMessage.toString());
+ }
+ if (((Map) jsonMap.get("output")).get("status").equals("ok")) {
+ String result = (String) ((Map) ((Map) jsonMap.get("output"))
+ .get("data")).get("text/plain");
+ if (result != null) {
+ result = result.trim();
+ if (result.startsWith("<link")
+ || result.startsWith("<script")
+ || result.startsWith("<style")
+ || result.startsWith("<div")) {
+ result = "%html " + result;
+ }
+ }
+ return new InterpreterResult(Code.SUCCESS, result);
+ }
+ }
+ return null;
+ }
+
+ private Map executeCommand(String lines, InterpreterContext context,
+ Map<String, Integer> userSessionMap) throws Exception {
+ String json = executeHTTP(property.get("zeppelin.livy.url") + "/sessions/"
+ + userSessionMap.get(context.getAuthenticationInfo().getUser())
+ + "/statements",
+ "POST",
+ "{\"code\": \"" + lines + "\" }",
+ context.getParagraphId());
+ if (json.matches("^(\")?Session (\'[0-9]\' )?not found(.?\"?)$")) {
+ throw new Exception("Exception: Session not found, Livy server would have restarted, " +
+ "or lost session.");
+ }
+ try {
+ Map jsonMap = gson.fromJson(json,
+ new TypeToken<Map>() {
+ }.getType());
+ return jsonMap;
+ } catch (Exception e) {
+ LOGGER.error("Error executeCommand", e);
+ throw e;
+ }
+ }
+
+ private Map getStatusById(InterpreterContext context,
+ Map<String, Integer> userSessionMap, Integer id) throws Exception {
+ String json = executeHTTP(property.getProperty("zeppelin.livy.url") + "/sessions/"
+ + userSessionMap.get(context.getAuthenticationInfo().getUser())
+ + "/statements/" + id,
+ "GET", null, context.getParagraphId());
+ try {
+ Map jsonMap = gson.fromJson(json,
+ new TypeToken<Map>() {
+ }.getType());
+ return jsonMap;
+ } catch (Exception e) {
+ LOGGER.error("Error getStatusById", e);
+ throw e;
+ }
+ }
+
+ protected String executeHTTP(String targetURL, String method, String jsonData, String paragraphId)
+ throws Exception {
+ HttpClient client = HttpClientBuilder.create().build();
+ HttpResponse response = null;
+ if (method.equals("POST")) {
+ HttpPost request = new HttpPost(targetURL);
+ request.addHeader("Content-Type", "application/json");
+ StringEntity se = new StringEntity(jsonData);
+ request.setEntity(se);
+ response = client.execute(request);
+ paragraphHttpMap.put(paragraphId, request);
+ } else if (method.equals("GET")) {
+ HttpGet request = new HttpGet(targetURL);
+ request.addHeader("Content-Type", "application/json");
+ response = client.execute(request);
+ paragraphHttpMap.put(paragraphId, request);
+ } else if (method.equals("DELETE")) {
+ HttpDelete request = new HttpDelete(targetURL);
+ request.addHeader("Content-Type", "application/json");
+ response = client.execute(request);
+ }
+
+ if (response == null) {
+ return null;
+ }
+
+ if (response.getStatusLine().getStatusCode() == 200
+ || response.getStatusLine().getStatusCode() == 201
+ || response.getStatusLine().getStatusCode() == 404) {
+ return getResponse(response);
+ } else {
+ String responseString = getResponse(response);
+ if (responseString.contains("CreateInteractiveRequest[\\\"master\\\"]")) {
+ return responseString;
+ }
+ LOGGER.error(String.format("Error with %s StatusCode: %s",
+ response.getStatusLine().getStatusCode(), responseString));
+ throw new Exception(String.format("Error with %s StatusCode: %s",
+ response.getStatusLine().getStatusCode(), responseString));
+ }
+ }
+
+ private String getResponse(HttpResponse response) throws Exception {
+ BufferedReader rd = new BufferedReader(
+ new InputStreamReader(response.getEntity().getContent()));
+
+ StringBuffer result = new StringBuffer();
+ String line = "";
+ while ((line = rd.readLine()) != null) {
+ result.append(line);
+ }
+ return result.toString();
+ }
+
+ public void cancelHTTP(String paragraphId) {
+ if (paragraphHttpMap.get(paragraphId).getClass().getName().contains("HttpPost")) {
+ ((HttpPost) paragraphHttpMap.get(paragraphId)).abort();
+ } else {
+ ((HttpGet) paragraphHttpMap.get(paragraphId)).abort();
+ }
+ paragraphHttpMap.put(paragraphId, null);
+ }
+
+ public void closeSession(Map<String, Integer> userSessionMap) {
+ for (Map.Entry<String, Integer> entry : userSessionMap.entrySet()) {
+ try {
+ executeHTTP(property.getProperty("zeppelin.livy.url") + "/sessions/"
+ + entry.getValue(),
+ "DELETE", null, null);
+ } catch (Exception e) {
+ LOGGER.error(String.format("Error closing session for user with session ID: %s",
+ entry.getValue()), e);
+ }
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/livy/src/main/java/org/apache/zeppelin/livy/LivyOutputStream.java
----------------------------------------------------------------------
diff --git a/livy/src/main/java/org/apache/zeppelin/livy/LivyOutputStream.java b/livy/src/main/java/org/apache/zeppelin/livy/LivyOutputStream.java
new file mode 100644
index 0000000..6a525eb
--- /dev/null
+++ b/livy/src/main/java/org/apache/zeppelin/livy/LivyOutputStream.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.livy;
+
+import org.apache.zeppelin.interpreter.InterpreterOutput;
+
+import java.io.IOException;
+import java.io.OutputStream;
+
+/**
+ * InterpreterOutput can be attached / detached.
+ */
+public class LivyOutputStream extends OutputStream {
+ InterpreterOutput interpreterOutput;
+
+ public LivyOutputStream() {
+ }
+
+ public InterpreterOutput getInterpreterOutput() {
+ return interpreterOutput;
+ }
+
+ public void setInterpreterOutput(InterpreterOutput interpreterOutput) {
+ this.interpreterOutput = interpreterOutput;
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (interpreterOutput != null) {
+ interpreterOutput.write(b);
+ }
+ }
+
+ @Override
+ public void write(byte[] b) throws IOException {
+ if (interpreterOutput != null) {
+ interpreterOutput.write(b);
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int offset, int len) throws IOException {
+ if (interpreterOutput != null) {
+ interpreterOutput.write(b, offset, len);
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (interpreterOutput != null) {
+ interpreterOutput.close();
+ }
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (interpreterOutput != null) {
+ interpreterOutput.flush();
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/livy/src/main/java/org/apache/zeppelin/livy/LivyPySparkInterpreter.java
----------------------------------------------------------------------
diff --git a/livy/src/main/java/org/apache/zeppelin/livy/LivyPySparkInterpreter.java b/livy/src/main/java/org/apache/zeppelin/livy/LivyPySparkInterpreter.java
new file mode 100644
index 0000000..a24e1ae
--- /dev/null
+++ b/livy/src/main/java/org/apache/zeppelin/livy/LivyPySparkInterpreter.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.livy;
+
+import org.apache.zeppelin.interpreter.*;
+import org.apache.zeppelin.scheduler.Scheduler;
+import org.apache.zeppelin.scheduler.SchedulerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+
+/**
+ * Livy PySpark interpreter for Zeppelin.
+ */
+public class LivyPySparkInterpreter extends Interpreter {
+
+ Logger LOGGER = LoggerFactory.getLogger(LivyPySparkInterpreter.class);
+
+ static {
+ Interpreter.register(
+ "pyspark",
+ "livy",
+ LivyPySparkInterpreter.class.getName(),
+ new InterpreterPropertyBuilder()
+ .build()
+ );
+ }
+
+ protected Map<String, Integer> userSessionMap;
+ protected LivyHelper livyHelper;
+
+ public LivyPySparkInterpreter(Properties property) {
+ super(property);
+ userSessionMap = new HashMap<>();
+ livyHelper = new LivyHelper(property);
+ }
+
+ @Override
+ public void open() {
+ }
+
+ @Override
+ public void close() {
+ livyHelper.closeSession(userSessionMap);
+ }
+
+ @Override
+ public InterpreterResult interpret(String line, InterpreterContext interpreterContext) {
+ try {
+ if (userSessionMap.get(interpreterContext.getAuthenticationInfo().getUser()) == null) {
+ try {
+ userSessionMap.put(
+ interpreterContext.getAuthenticationInfo().getUser(),
+ livyHelper.createSession(
+ interpreterContext,
+ "pyspark")
+ );
+ } catch (Exception e) {
+ LOGGER.error("Exception in LivyPySparkInterpreter while interpret ", e);
+ return new InterpreterResult(InterpreterResult.Code.ERROR, e.getMessage());
+ }
+ }
+
+ if (line == null || line.trim().length() == 0) {
+ return new InterpreterResult(InterpreterResult.Code.SUCCESS, "");
+ }
+
+ return livyHelper.interpret(line, interpreterContext, userSessionMap);
+ } catch (Exception e) {
+ LOGGER.error("Exception in LivyPySparkInterpreter while interpret ", e);
+ return new InterpreterResult(InterpreterResult.Code.ERROR,
+ InterpreterUtils.getMostRelevantMessage(e));
+ }
+ }
+
+ @Override
+ public void cancel(InterpreterContext context) {
+ livyHelper.cancelHTTP(context.getParagraphId());
+ }
+
+ @Override
+ public FormType getFormType() {
+ return FormType.SIMPLE;
+ }
+
+ @Override
+ public int getProgress(InterpreterContext context) {
+ return 0;
+ }
+
+ @Override
+ public Scheduler getScheduler() {
+ return SchedulerFactory.singleton().createOrGetFIFOScheduler(
+ LivyPySparkInterpreter.class.getName() + this.hashCode());
+ }
+
+ @Override
+ public List<String> completion(String buf, int cursor) {
+ return null;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/livy/src/main/java/org/apache/zeppelin/livy/LivySparkInterpreter.java
----------------------------------------------------------------------
diff --git a/livy/src/main/java/org/apache/zeppelin/livy/LivySparkInterpreter.java b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkInterpreter.java
new file mode 100644
index 0000000..23a6379
--- /dev/null
+++ b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkInterpreter.java
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.livy;
+
+import org.apache.zeppelin.interpreter.*;
+import org.apache.zeppelin.scheduler.Scheduler;
+import org.apache.zeppelin.scheduler.SchedulerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+/**
+ * Livy Spark interpreter for Zeppelin.
+ */
+public class LivySparkInterpreter extends Interpreter {
+
+ static String DEFAULT_URL = "http://localhost:8998";
+ static String LOCAL = "local[*]";
+ Logger LOGGER = LoggerFactory.getLogger(LivySparkInterpreter.class);
+ private LivyOutputStream out;
+
+ static {
+ Interpreter.register(
+ "spark",
+ "livy",
+ LivySparkInterpreter.class.getName(),
+ new InterpreterPropertyBuilder()
+ .add("zeppelin.livy.url", DEFAULT_URL, "The URL for Livy Server.")
+ .add("zeppelin.livy.master", LOCAL, "Spark master uri. ex) spark://masterhost:7077")
+ .build()
+ );
+ }
+
+ protected static Map<String, Integer> userSessionMap;
+ private LivyHelper livyHelper;
+
+ public LivySparkInterpreter(Properties property) {
+ super(property);
+ userSessionMap = new HashMap<>();
+ livyHelper = new LivyHelper(property);
+ out = new LivyOutputStream();
+ }
+
+ protected static Map<String, Integer> getUserSessionMap() {
+ return userSessionMap;
+ }
+
+ public void setUserSessionMap(Map<String, Integer> userSessionMap) {
+ this.userSessionMap = userSessionMap;
+ }
+
+ @Override
+ public void open() {
+ }
+
+ @Override
+ public void close() {
+ livyHelper.closeSession(userSessionMap);
+ }
+
+ @Override
+ public InterpreterResult interpret(String line, InterpreterContext interpreterContext) {
+ try {
+ if (userSessionMap.get(interpreterContext.getAuthenticationInfo().getUser()) == null) {
+ try {
+ userSessionMap.put(
+ interpreterContext.getAuthenticationInfo().getUser(),
+ livyHelper.createSession(
+ interpreterContext,
+ "spark")
+ );
+ livyHelper.initializeSpark(interpreterContext, userSessionMap);
+ } catch (Exception e) {
+ LOGGER.error("Exception in LivySparkInterpreter while interpret ", e);
+ return new InterpreterResult(InterpreterResult.Code.ERROR, e.getMessage());
+ }
+ }
+ if (line == null || line.trim().length() == 0) {
+ return new InterpreterResult(InterpreterResult.Code.SUCCESS, "");
+ }
+
+ return livyHelper.interpretInput(line, interpreterContext, userSessionMap, out);
+ } catch (Exception e) {
+ LOGGER.error("Exception in LivySparkInterpreter while interpret ", e);
+ return new InterpreterResult(InterpreterResult.Code.ERROR,
+ InterpreterUtils.getMostRelevantMessage(e));
+ }
+ }
+
+ @Override
+ public void cancel(InterpreterContext context) {
+ livyHelper.cancelHTTP(context.getParagraphId());
+ }
+
+ @Override
+ public FormType getFormType() {
+ return FormType.SIMPLE;
+ }
+
+ @Override
+ public int getProgress(InterpreterContext context) {
+ return 0;
+ }
+
+ @Override
+ public Scheduler getScheduler() {
+ return SchedulerFactory.singleton().createOrGetFIFOScheduler(
+ LivySparkInterpreter.class.getName() + this.hashCode());
+ }
+
+ @Override
+ public List<String> completion(String buf, int cursor) {
+ return null;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/livy/src/main/java/org/apache/zeppelin/livy/LivySparkRInterpreter.java
----------------------------------------------------------------------
diff --git a/livy/src/main/java/org/apache/zeppelin/livy/LivySparkRInterpreter.java b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkRInterpreter.java
new file mode 100644
index 0000000..a41f4a7
--- /dev/null
+++ b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkRInterpreter.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.livy;
+
+import org.apache.zeppelin.interpreter.*;
+import org.apache.zeppelin.scheduler.Scheduler;
+import org.apache.zeppelin.scheduler.SchedulerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+
+/**
+ * Livy PySpark interpreter for Zeppelin.
+ */
+public class LivySparkRInterpreter extends Interpreter {
+
+ Logger LOGGER = LoggerFactory.getLogger(LivySparkRInterpreter.class);
+
+ static {
+ Interpreter.register(
+ "sparkr",
+ "livy",
+ LivySparkRInterpreter.class.getName(),
+ new InterpreterPropertyBuilder()
+ .build()
+ );
+ }
+
+ protected Map<String, Integer> userSessionMap;
+ private LivyHelper livyHelper;
+
+ public LivySparkRInterpreter(Properties property) {
+ super(property);
+ userSessionMap = new HashMap<>();
+ livyHelper = new LivyHelper(property);
+ }
+
+ @Override
+ public void open() {
+ }
+
+ @Override
+ public void close() {
+ livyHelper.closeSession(userSessionMap);
+ }
+
+ @Override
+ public InterpreterResult interpret(String line, InterpreterContext interpreterContext) {
+ try {
+ if (userSessionMap.get(interpreterContext.getAuthenticationInfo().getUser()) == null) {
+ try {
+ userSessionMap.put(
+ interpreterContext.getAuthenticationInfo().getUser(),
+ livyHelper.createSession(
+ interpreterContext,
+ "sparkr")
+ );
+ } catch (Exception e) {
+ LOGGER.error("Exception in LivySparkRInterpreter while interpret ", e);
+ return new InterpreterResult(InterpreterResult.Code.ERROR, e.getMessage());
+ }
+ }
+
+ if (line == null || line.trim().length() == 0) {
+ return new InterpreterResult(InterpreterResult.Code.SUCCESS, "");
+ }
+
+ return livyHelper.interpret(line, interpreterContext, userSessionMap);
+ } catch (Exception e) {
+ LOGGER.error("Exception in LivySparkRInterpreter while interpret ", e);
+ return new InterpreterResult(InterpreterResult.Code.ERROR,
+ InterpreterUtils.getMostRelevantMessage(e));
+ }
+ }
+
+ @Override
+ public void cancel(InterpreterContext context) {
+ livyHelper.cancelHTTP(context.getParagraphId());
+ }
+
+ @Override
+ public FormType getFormType() {
+ return FormType.SIMPLE;
+ }
+
+ @Override
+ public int getProgress(InterpreterContext context) {
+ return 0;
+ }
+
+ @Override
+ public Scheduler getScheduler() {
+ return SchedulerFactory.singleton().createOrGetFIFOScheduler(
+ LivySparkRInterpreter.class.getName() + this.hashCode());
+ }
+
+ @Override
+ public List<String> completion(String buf, int cursor) {
+ return null;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java
----------------------------------------------------------------------
diff --git a/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java
new file mode 100644
index 0000000..3637c64
--- /dev/null
+++ b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.livy;
+
+import org.apache.zeppelin.interpreter.*;
+import org.apache.zeppelin.scheduler.Scheduler;
+import org.apache.zeppelin.scheduler.SchedulerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+
+/**
+ * Livy PySpark interpreter for Zeppelin.
+ */
+public class LivySparkSQLInterpreter extends Interpreter {
+
+ Logger LOGGER = LoggerFactory.getLogger(LivySparkSQLInterpreter.class);
+ static String DEFAULT_MAX_RESULT = "1000";
+
+ static {
+ Interpreter.register(
+ "sql",
+ "livy",
+ LivySparkSQLInterpreter.class.getName(),
+ new InterpreterPropertyBuilder()
+ .add("zeppelin.livy.spark.maxResult",
+ DEFAULT_MAX_RESULT,
+ "Max number of SparkSQL result to display.")
+ .build()
+ );
+ }
+
+ protected Map<String, Integer> userSessionMap;
+ private LivyHelper livyHelper;
+
+ public LivySparkSQLInterpreter(Properties property) {
+ super(property);
+ livyHelper = new LivyHelper(property);
+ userSessionMap = LivySparkInterpreter.getUserSessionMap();
+ }
+
+ @Override
+ public void open() {
+ }
+
+ @Override
+ public void close() {
+ livyHelper.closeSession(userSessionMap);
+ }
+
+ @Override
+ public InterpreterResult interpret(String line, InterpreterContext interpreterContext) {
+ try {
+ if (userSessionMap.get(interpreterContext.getAuthenticationInfo().getUser()) == null) {
+ try {
+ userSessionMap.put(
+ interpreterContext.getAuthenticationInfo().getUser(),
+ livyHelper.createSession(
+ interpreterContext,
+ "spark")
+ );
+ livyHelper.initializeSpark(interpreterContext, userSessionMap);
+ } catch (Exception e) {
+ LOGGER.error("Exception in LivySparkSQLInterpreter while interpret ", e);
+ return new InterpreterResult(InterpreterResult.Code.ERROR, e.getMessage());
+ }
+ }
+
+ if (line == null || line.trim().length() == 0) {
+ return new InterpreterResult(InterpreterResult.Code.SUCCESS, "");
+ }
+
+ InterpreterResult res = livyHelper.interpret("sqlContext.sql(\"" +
+ line.replaceAll("\"", "\\\\\"")
+ .replaceAll("\\n", " ")
+ + "\").show(" +
+ property.get("zeppelin.livy.spark.maxResult") + ")",
+ interpreterContext, userSessionMap);
+
+ if (res.code() == InterpreterResult.Code.SUCCESS) {
+ StringBuilder resMsg = new StringBuilder();
+ resMsg.append("%table ");
+ String[] rows = res.message().split("\n");
+
+ String[] headers = rows[1].split("\\|");
+ for (int head = 1; head < headers.length; head++) {
+ resMsg.append(headers[head].trim()).append("\t");
+ }
+ resMsg.append("\n");
+ if (rows[3].indexOf("+") == 0) {
+
+ } else {
+ for (int cols = 3; cols < rows.length - 1; cols++) {
+ String[] col = rows[cols].split("\\|");
+ for (int data = 1; data < col.length; data++) {
+ resMsg.append(col[data].trim()).append("\t");
+ }
+ resMsg.append("\n");
+ }
+ }
+ if (rows[rows.length - 1].indexOf("only") == 0) {
+ resMsg.append("<font color=red>" + rows[rows.length - 1] + ".</font>");
+ }
+
+ return new InterpreterResult(InterpreterResult.Code.SUCCESS,
+ resMsg.toString()
+ );
+ } else {
+ return res;
+ }
+
+
+ } catch (Exception e) {
+ LOGGER.error("Exception in LivySparkSQLInterpreter while interpret ", e);
+ return new InterpreterResult(InterpreterResult.Code.ERROR,
+ InterpreterUtils.getMostRelevantMessage(e));
+ }
+ }
+
+ @Override
+ public void cancel(InterpreterContext context) {
+ livyHelper.cancelHTTP(context.getParagraphId());
+ }
+
+ @Override
+ public FormType getFormType() {
+ return FormType.SIMPLE;
+ }
+
+ @Override
+ public int getProgress(InterpreterContext context) {
+ return 0;
+ }
+
+ @Override
+ public Scheduler getScheduler() {
+ return SchedulerFactory.singleton().createOrGetFIFOScheduler(
+ LivySparkInterpreter.class.getName() + this.hashCode());
+ }
+
+ @Override
+ public List<String> completion(String buf, int cursor) {
+ return null;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/livy/src/main/test/org/apache/zeppelin/livy/LivyHelperTest.java
----------------------------------------------------------------------
diff --git a/livy/src/main/test/org/apache/zeppelin/livy/LivyHelperTest.java b/livy/src/main/test/org/apache/zeppelin/livy/LivyHelperTest.java
new file mode 100644
index 0000000..4928901
--- /dev/null
+++ b/livy/src/main/test/org/apache/zeppelin/livy/LivyHelperTest.java
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.livy;
+
+import com.google.gson.GsonBuilder;
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.hamcrest.CoreMatchers;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ErrorCollector;
+import org.junit.runner.RunWith;
+import org.mockito.Answers;
+import org.mockito.Mock;
+import org.mockito.runners.MockitoJUnitRunner;
+
+import java.util.HashMap;
+import java.util.Properties;
+
+import static org.mockito.Mockito.doReturn;
+
+/**
+ * Created for org.apache.zeppelin.livy on 22/04/16.
+ */
+
+@RunWith(MockitoJUnitRunner.class)
+public class LivyHelperTest {
+
+ @Rule
+ public ErrorCollector collector = new ErrorCollector();
+
+ @Mock(answer = Answers.RETURNS_DEEP_STUBS)
+ private static LivyPySparkInterpreter interpreter;
+
+ @Mock(answer = Answers.RETURNS_DEEP_STUBS)
+ private InterpreterContext interpreterContext;
+
+ @Mock(answer = Answers.CALLS_REAL_METHODS)
+ private LivyHelper livyHelper;
+
+ @Before
+ public void prepareContext() throws Exception {
+ interpreter.userSessionMap = new HashMap<>();
+ interpreter.userSessionMap.put(null, 1);
+
+ Properties properties = new Properties();
+ properties.setProperty("zeppelin.livy.url", "http://localhost:8998");
+ livyHelper.property = properties;
+ livyHelper.paragraphHttpMap = new HashMap<>();
+ livyHelper.gson = new GsonBuilder().setPrettyPrinting().create();
+
+
+ doReturn("{\"id\":1,\"state\":\"idle\",\"kind\":\"spark\",\"proxyUser\":\"null\",\"log\":[]}")
+ .when(livyHelper)
+ .executeHTTP(
+ livyHelper.property.getProperty("zeppelin.livy.url") + "/sessions",
+ "POST",
+ "{\"kind\": \"spark\", \"proxyUser\": \"null\"}",
+ null
+ );
+
+ doReturn("{\"id\":1,\"state\":\"available\",\"output\":{\"status\":\"ok\"," +
+ "\"execution_count\":1,\"data\":{\"text/plain\":\"1\"}}}")
+ .when(livyHelper)
+ .executeHTTP(
+ livyHelper.property.getProperty("zeppelin.livy.url") + "/sessions/1/statements",
+ "POST",
+ "{\"code\": \"print(1)\" }",
+ null
+ );
+
+ }
+
+
+ @Test
+ public void checkCreateSession() {
+ try {
+ Integer sessionId = livyHelper.createSession(interpreterContext, "spark");
+
+ collector.checkThat("check sessionId", 1, CoreMatchers.equalTo(sessionId));
+
+ } catch (Exception e) {
+ collector.addError(e);
+ }
+ }
+
+ @Test
+ public void checkInterpret() {
+ try {
+ InterpreterResult result = livyHelper.interpret("print(1)", interpreterContext, interpreter.userSessionMap);
+
+ collector.checkThat("check sessionId", InterpreterResult.Code.SUCCESS, CoreMatchers.equalTo(result.code()));
+
+ } catch (Exception e) {
+ collector.addError(e);
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/livy/src/main/test/org/apache/zeppelin/livy/LivyInterpreterTest.java
----------------------------------------------------------------------
diff --git a/livy/src/main/test/org/apache/zeppelin/livy/LivyInterpreterTest.java b/livy/src/main/test/org/apache/zeppelin/livy/LivyInterpreterTest.java
new file mode 100644
index 0000000..b21afa4
--- /dev/null
+++ b/livy/src/main/test/org/apache/zeppelin/livy/LivyInterpreterTest.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.livy;
+
+
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.hamcrest.CoreMatchers;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ErrorCollector;
+import org.junit.runner.RunWith;
+import org.mockito.Answers;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.runners.MockitoJUnitRunner;
+
+import java.util.HashMap;
+import java.util.Properties;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.doReturn;
+
+@RunWith(MockitoJUnitRunner.class)
+public class LivyInterpreterTest {
+
+ @Rule
+ public ErrorCollector collector = new ErrorCollector();
+
+ private static LivyPySparkInterpreter interpreter;
+
+ @Mock(answer = Answers.RETURNS_DEEP_STUBS)
+ private InterpreterContext interpreterContext;
+
+ @AfterClass
+ public static void tearDown() {
+ interpreter.close();
+ }
+
+ @Before
+ public void prepareContext() throws Exception {
+ interpreter = new LivyPySparkInterpreter(new Properties());
+ interpreter.userSessionMap = new HashMap<>();
+ interpreter.userSessionMap.put(null, 0);
+ interpreter.livyHelper = Mockito.mock(LivyHelper.class);
+ interpreter.open();
+
+ doReturn(new InterpreterResult(InterpreterResult.Code.SUCCESS)).when(interpreter.livyHelper)
+ .interpret("print \"x is 1.\"", interpreterContext, interpreter.userSessionMap);
+ }
+
+ @Test
+ public void checkInitVariables() throws Exception {
+ collector.checkThat("Check that, if userSessionMap is made: ",
+ interpreter.userSessionMap, CoreMatchers.notNullValue());
+ }
+
+ @Test
+ public void checkBasicInterpreter() throws Exception {
+
+ String paragraphString = "print \"x is 1.\"";
+
+ final InterpreterResult actual = interpreter.interpret(paragraphString, interpreterContext);
+
+ collector.checkThat("Check that, result is computed: ",
+ actual.code(), CoreMatchers.equalTo(InterpreterResult.Code.SUCCESS));
+ assertThat(actual).isNotNull();
+ }
+
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index c7d9312..77f3667 100755
--- a/pom.xml
+++ b/pom.xml
@@ -92,6 +92,7 @@
<module>markdown</module>
<module>angular</module>
<module>shell</module>
+ <module>livy</module>
<module>hive</module>
<module>hbase</module>
<module>phoenix</module>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/6acd0aee/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
index 541aae1..9eceeed 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
@@ -463,6 +463,10 @@ public class ZeppelinConfiguration extends XMLConfiguration {
+ "org.apache.zeppelin.markdown.Markdown,"
+ "org.apache.zeppelin.angular.AngularInterpreter,"
+ "org.apache.zeppelin.shell.ShellInterpreter,"
+ + "org.apache.zeppelin.livy.LivySparkInterpreter,"
+ + "org.apache.zeppelin.livy.LivySparkSQLInterpreter,"
+ + "org.apache.zeppelin.livy.LivyPySparkInterpreter,"
+ + "org.apache.zeppelin.livy.LivySparkRInterpreter,"
+ "org.apache.zeppelin.hive.HiveInterpreter,"
+ "org.apache.zeppelin.alluxio.AlluxioInterpreter,"
+ "org.apache.zeppelin.file.HDFSFileInterpreter,"