You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by fe...@apache.org on 2016/01/29 00:00:14 UTC
incubator-zeppelin git commit: Add tachyon interpreter
Repository: incubator-zeppelin
Updated Branches:
refs/heads/master 4a23139fd -> 1b3784d23
Add tachyon interpreter
### What is this PR for?
Add an interpreter to work with a tachyon file system.
Properties required for the interpreter are:
* tachyon master hostname
* tachyon master port
### What type of PR is it?
feature
### Is there a relevant Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-604
### How should this be tested?
* [Install and configure Tachyon](http://tachyon-project.org/downloads/) on a local machine.
* run Tachyon in local mode ```$ ./bin/tachyon-start.sh local```
* check that interpreter params are setted to default values (hostname: localhost, port: 19998)
* use the [Tachyon CLI commands](http://tachyon-project.org/documentation/Command-Line-Interface.html) to interact with your Tachyon file system
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
/cc jsimsa for the support he give us to develop this feature
Author: maocorte <ma...@radicalbit.io>
Closes #632 from maocorte/tachyon-interpreter and squashes the following commits:
6f01654 [maocorte] added new line on tachyon doc to fix visualization problem
700ff48 [maocorte] added a simple test example to tachyon doc
e7341af [maocorte] small fixes on tachyon doc
3f1c455 [maocorte] formatted tachyon doc with new guidelines + added link to tachyon doc into navigation page
1018409 [maocorte] added tachyon dependency licences
cc622ce [maocorte] added Tachyon interpreter tests
9c51dca [maocorte] changed tachyon module order into main pom
5eaafbb [maocorte] added tachyon interpreter md doc
12fbf03 [maocorte] added help command to list all available commands
e5465a5 [maocorte] resolved conflict
eeb37d6 [maocorte] resolved comflict
f0ed930 [maocorte] removed unused import
8d7ae93 [maocorte] changed completion implementation
0ded511 [maocorte] added tachyon shell license
8a33ea5 [maocorte] added logs on opening and closing interpreter
53011dc [maocorte] Merge branch 'master' into tachyon-interpreter
ca64582 [maocorte] code cleanup
7c09709 [maocorte] updated code to master branch
b8add66 [maocorte] using system properties to set tachyon client configuration
2895a59 [maocorte] updated to code to lastest master version
b5a4b54 [maocorte] added tachyon interpreter to available interpreters list
8a84514 [maocorte] added tachyon interpreter module to main project pom
88f2bcf [maocorte] added tachyon interpreter module
Project: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/commit/1b3784d2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/tree/1b3784d2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/diff/1b3784d2
Branch: refs/heads/master
Commit: 1b3784d2314195319b6c92182a03e457c2341bf0
Parents: 4a23139
Author: maocorte <ma...@radicalbit.io>
Authored: Wed Jan 27 15:15:01 2016 +0100
Committer: Felix Cheung <fe...@apache.org>
Committed: Thu Jan 28 15:00:09 2016 -0800
----------------------------------------------------------------------
conf/zeppelin-site.xml.template | 2 +-
docs/_includes/themes/zeppelin/_navigation.html | 1 +
.../zeppelin/img/docs-img/tachyon-example.png | Bin 0 -> 44267 bytes
docs/interpreter/tachyon.md | 216 +++++++++
notebook/2A94M5J1Z/note.json | 10 +-
pom.xml | 1 +
tachyon/pom.xml | 164 +++++++
.../zeppelin/tachyon/TachyonInterpreter.java | 251 ++++++++++
.../tachyon/TachyonInterpreterTest.java | 484 +++++++++++++++++++
zeppelin-distribution/src/bin_license/LICENSE | 4 +
.../zeppelin/conf/ZeppelinConfiguration.java | 1 +
11 files changed, 1131 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/conf/zeppelin-site.xml.template
----------------------------------------------------------------------
diff --git a/conf/zeppelin-site.xml.template b/conf/zeppelin-site.xml.template
index 4ed7a2a..9ca740d 100755
--- a/conf/zeppelin-site.xml.template
+++ b/conf/zeppelin-site.xml.template
@@ -105,7 +105,7 @@
<property>
<name>zeppelin.interpreters</name>
- <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.phoenix.PhoenixInterpreter,org.apache.zeppelin.kylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter</value>
+ <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.phoenix.PhoenixInterpreter,org.apache.zeppelin.kylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter,org.apache.zeppelin.tachyon.TachyonIn
terpreter</value>
<description>Comma separated interpreter configurations. First interpreter become a default</description>
</property>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/docs/_includes/themes/zeppelin/_navigation.html
----------------------------------------------------------------------
diff --git a/docs/_includes/themes/zeppelin/_navigation.html b/docs/_includes/themes/zeppelin/_navigation.html
index 7481e32..d0581b1 100644
--- a/docs/_includes/themes/zeppelin/_navigation.html
+++ b/docs/_includes/themes/zeppelin/_navigation.html
@@ -51,6 +51,7 @@
<li><a href="{{BASE_PATH}}/interpreter/scalding.html">Scalding</a></li>
<li><a href="{{BASE_PATH}}/pleasecontribute.html">Shell</a></li>
<li><a href="{{BASE_PATH}}/interpreter/spark.html">Spark</a></li>
+ <li><a href="{{BASE_PATH}}/interpreter/tachyon.html">Tachyon</a></li>
<li><a href="{{BASE_PATH}}/pleasecontribute.html">Tajo</a></li>
</ul>
</li>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/docs/assets/themes/zeppelin/img/docs-img/tachyon-example.png
----------------------------------------------------------------------
diff --git a/docs/assets/themes/zeppelin/img/docs-img/tachyon-example.png b/docs/assets/themes/zeppelin/img/docs-img/tachyon-example.png
new file mode 100644
index 0000000..1a57ffa
Binary files /dev/null and b/docs/assets/themes/zeppelin/img/docs-img/tachyon-example.png differ
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/docs/interpreter/tachyon.md
----------------------------------------------------------------------
diff --git a/docs/interpreter/tachyon.md b/docs/interpreter/tachyon.md
new file mode 100644
index 0000000..6c741e2
--- /dev/null
+++ b/docs/interpreter/tachyon.md
@@ -0,0 +1,216 @@
+---
+layout: page
+title: "Tachyon Interpreter"
+description: "Tachyon Interpreter"
+group: manual
+---
+{% include JB/setup %}
+
+## Tachyon Interpreter for Apache Zeppelin
+[Tachyon](http://tachyon-project.org/) is a memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks.
+
+## Configuration
+<table class="table-configuration">
+ <tr>
+ <th>Name</th>
+ <th>Class</th>
+ <th>Description</th>
+ </tr>
+ <tr>
+ <td>tachyon.master.hostname</td>
+ <td>localhost</td>
+ <td>Tachyon master hostname</td>
+ </tr>
+ <tr>
+ <td>tachyon.master.port</td>
+ <td>19998</td>
+ <td>Tachyon master port</td>
+ </tr>
+</table>
+
+## Enabling Tachyon Interpreter
+In a notebook, to enable the **Tachyon** interpreter, click on the **Gear** icon and select **Tachyon**.
+
+## Using the Tachyon Interpreter
+In a paragraph, use `%tachyon` to select the **Tachyon** interpreter and then input all commands.
+
+```bash
+%tachyon
+help
+```
+
+> **Tip :** Use ( Ctrl + . ) for autocompletion.
+
+## Interpreter Commands
+The **Tachyon** interpreter accepts the following commands.
+
+<center>
+ <table class="table-configuration">
+ <tr>
+ <th>Operation</th>
+ <th>Syntax</th>
+ <th>Description</th>
+ </tr>
+ <tr>
+ <td>copyFromLocal</td>
+ <td>copyFromLocal "source path" "remote path"</td>
+ <td>Copy the specified file specified by "source path" to the path specified by "remote path".
+ This command will fail if "remote path" already exists.</td>
+ </tr>
+ <tr>
+ <td>copyToLocal</td>
+ <td>copyToLocal "remote path" "local path"</td>
+ <td>Copy the specified file from the path specified by "remote source" to a local
+ destination.</td>
+ </tr>
+ <tr>
+ <td>count</td>
+ <td>count "path"</td>
+ <td>Display the number of folders and files matching the specified prefix in "path".</td>
+ </tr>
+ <tr>
+ <td>du</td>
+ <td>du "path"</td>
+ <td>Display the size of a file or a directory specified by the input path.</td>
+ </tr>
+ <tr>
+ <td>fileinfo</td>
+ <td>fileinfo "path"</td>
+ <td>Print the information of the blocks of a specified file.</td>
+ </tr>
+ <tr>
+ <td>free</td>
+ <td>free "path"</td>
+ <td>Free a file or all files under a directory from Tachyon. If the file/directory is also
+ in under storage, it will still be available there.</td>
+ </tr>
+ <tr>
+ <td>getCapacityBytes</td>
+ <td>getCapacityBytes</td>
+ <td>Get the capacity of the TachyonFS.</td>
+ </tr>
+ <tr>
+ <td>getUsedBytes</td>
+ <td>getUsedBytes</td>
+ <td>Get number of bytes used in the TachyonFS.</td>
+ </tr>
+ <tr>
+ <td>load</td>
+ <td>load "path"</td>
+ <td>Load the data of a file or a directory from under storage into Tachyon.</td>
+ </tr>
+ <tr>
+ <td>loadMetadata</td>
+ <td>loadMetadata "path"</td>
+ <td>Load the metadata of a file or a directory from under storage into Tachyon.</td>
+ </tr>
+ <tr>
+ <td>location</td>
+ <td>location "path"</td>
+ <td>Display a list of hosts that have the file data.</td>
+ </tr>
+ <tr>
+ <td>ls</td>
+ <td>ls "path"</td>
+ <td>List all the files and directories directly under the given path with information such as
+ size.</td>
+ </tr>
+ <tr>
+ <td>lsr</td>
+ <td>lsr "path"</td>
+ <td>Recursively list all the files and directories under the given path with information such
+ as size.</td>
+ </tr>
+ <tr>
+ <td>mkdir</td>
+ <td>mkdir "path"</td>
+ <td>Create a directory under the given path, along with any necessary parent directories. This
+ command will fail if the given path already exists.</td>
+ </tr>
+ <tr>
+ <td>mount</td>
+ <td>mount "path" "uri"</td>
+ <td>Mount the underlying file system path "uri" into the Tachyon namespace as "path". The "path"
+ is assumed not to exist and is created by the operation. No data or metadata is loaded from under
+ storage into Tachyon. After a path is mounted, operations on objects under the mounted path are
+ mirror to the mounted under storage.</td>
+ </tr>
+ <tr>
+ <td>mv</td>
+ <td>mv "source" "destination"</td>
+ <td>Move a file or directory specified by "source" to a new location "destination". This command
+ will fail if "destination" already exists.</td>
+ </tr>
+ <tr>
+ <td>pin</td>
+ <td>pin "path"</td>
+ <td>Pin the given file to avoid evicting it from memory. If the given path is a directory, it
+ recursively pins all the files contained and any new files created within this directory.</td>
+ </tr>
+ <tr>
+ <td>report</td>
+ <td>report "path"</td>
+ <td>Report to the master that a file is lost.</td>
+ </tr>
+ <tr>
+ <td>request</td>
+ <td>request "path" "dependency ID"</td>
+ <td>Request the file for a given dependency ID.</td>
+ </tr>
+ <tr>
+ <td>rm</td>
+ <td>rm "path"</td>
+ <td>Remove a file. This command will fail if the given path is a directory rather than a
+ file.</td>
+ </tr>
+ <tr>
+ <td>rmr</td>
+ <td>rmr "path"</td>
+ <td>Remove a file, or a directory with all the files and sub-directories that this directory
+ contains.</td>
+ </tr>
+ <tr>
+ <td>tail</td>
+ <td>tail "path"</td>
+ <td>Print the last 1KB of the specified file to the console.</td>
+ </tr>
+ <tr>
+ <td>touch</td>
+ <td>touch "path"</td>
+ <td>Create a 0-byte file at the specified location.</td>
+ </tr>
+ <tr>
+ <td>unmount</td>
+ <td>unmount "path"</td>
+ <td>Unmount the underlying file system path mounted in the Tachyon namespace as "path". Tachyon
+ objects under "path" are removed from Tachyon, but they still exist in the previously mounted
+ under storage.</td>
+ </tr>
+ <tr>
+ <td>unpin</td>
+ <td>unpin "path"</td>
+ <td>Unpin the given file to allow Tachyon to evict this file again. If the given path is a
+ directory, it recursively unpins all files contained and any new files created within this
+ directory.</td>
+ </tr>
+ </table>
+</center>
+
+## How to test it's working
+Be sure to have configured correctly the Tachyon interpreter, then open a new paragraph and type one of the above commands.
+
+Below a simple example to show how to interact with Tachyon interpreter.
+Following steps are performed:
+
+* using sh interpreter a new text file is created on local machine
+* using Tachyon interpreter:
+ * is listed the content of the tfs (Tachyon File System) root
+ * the file previously created is copied to tfs
+ * is listed again the content of the tfs root to check the existence of the new copied file
+ * is showed the content of the copied file (using the tail command)
+ * the file previously copied to tfs is copied to local machine
+* using sh interpreter it's checked the existence of the new file copied from Tachyon and its content is showed
+
+<center>
+ 
+</center>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/notebook/2A94M5J1Z/note.json
----------------------------------------------------------------------
diff --git a/notebook/2A94M5J1Z/note.json b/notebook/2A94M5J1Z/note.json
index 785ccea..5687785 100644
--- a/notebook/2A94M5J1Z/note.json
+++ b/notebook/2A94M5J1Z/note.json
@@ -35,6 +35,7 @@
{
"title": "Load data into table",
"text": "import org.apache.commons.io.IOUtils\nimport java.net.URL\nimport java.nio.charset.Charset\n\n// Zeppelin creates and injects sc (SparkContext) and sqlContext (HiveContext or SqlContext)\n// So you don\u0027t need create them manually\n\n// load bank data\nval bankText \u003d sc.parallelize(\n IOUtils.toString(\n new URL(\"https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv\"),\n Charset.forName(\"utf8\")).split(\"\\n\"))\n\ncase class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer)\n\nval bank \u003d bankText.map(s \u003d\u003e s.split(\";\")).filter(s \u003d\u003e s(0) !\u003d \"\\\"age\\\"\").map(\n s \u003d\u003e Bank(s(0).toInt, \n s(1).replaceAll(\"\\\"\", \"\"),\n s(2).replaceAll(\"\\\"\", \"\"),\n s(3).replaceAll(\"\\\"\", \"\"),\n s(5).replaceAll(\"\\\"\", \"\").toInt\n )\n).toDF()\nbank.registerTempTable(\"bank\")",
+ "dateUpdated": "Jan 14, 2016 7:58:56 PM",
"config": {
"colWidth": 12.0,
"graph": {
@@ -46,7 +47,9 @@
"groups": [],
"scatter": {}
},
- "title": true
+ "title": true,
+ "enabled": true,
+ "editorMode": "ace/mode/scala"
},
"settings": {
"params": {},
@@ -333,7 +336,10 @@
],
"name": "Zeppelin Tutorial",
"id": "2A94M5J1Z",
- "angularObjects": {},
+ "angularObjects": {
+ "2B6FF8NNU": [],
+ "2B67PH63Z": []
+ },
"config": {
"looknfeel": "default"
},
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 2e75d2a..d73caf3 100755
--- a/pom.xml
+++ b/pom.xml
@@ -102,6 +102,7 @@
<module>lens</module>
<module>cassandra</module>
<module>elasticsearch</module>
+ <module>tachyon</module>
<module>zeppelin-web</module>
<module>zeppelin-server</module>
<module>zeppelin-distribution</module>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/tachyon/pom.xml
----------------------------------------------------------------------
diff --git a/tachyon/pom.xml b/tachyon/pom.xml
new file mode 100644
index 0000000..adcc47d
--- /dev/null
+++ b/tachyon/pom.xml
@@ -0,0 +1,164 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <artifactId>zeppelin</artifactId>
+ <groupId>org.apache.zeppelin</groupId>
+ <version>0.6.0-incubating-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <groupId>org.apache.zeppelin</groupId>
+ <artifactId>zeppelin-tachyon</artifactId>
+ <packaging>jar</packaging>
+ <version>0.6.0-incubating-SNAPSHOT</version>
+ <name>Zeppelin: Tachyon interpreter</name>
+ <url>http://www.apache.org</url>
+
+ <properties>
+ <tachyon.version>0.8.2</tachyon.version>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.zeppelin</groupId>
+ <artifactId>zeppelin-interpreter</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.tachyonproject</groupId>
+ <artifactId>tachyon-shell</artifactId>
+ <version>${tachyon.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </dependency>
+
+ <!-- TEST -->
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.tachyonproject</groupId>
+ <artifactId>tachyon-servers</artifactId>
+ <version>${tachyon.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.tachyonproject</groupId>
+ <artifactId>tachyon-minicluster</artifactId>
+ <version>${tachyon.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.tachyonproject</groupId>
+ <artifactId>tachyon-underfs-local</artifactId>
+ <version>${tachyon.version}</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-deploy-plugin</artifactId>
+ <version>2.7</version>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
+ </plugin>
+
+ <plugin>
+ <artifactId>maven-enforcer-plugin</artifactId>
+ <version>1.3.1</version>
+ <executions>
+ <execution>
+ <id>enforce</id>
+ <phase>none</phase>
+ </execution>
+ </executions>
+ </plugin>
+
+ <plugin>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <version>2.8</version>
+ <executions>
+ <execution>
+ <id>copy-dependencies</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.build.directory}/../../interpreter/tachyon</outputDirectory>
+ <overWriteReleases>false</overWriteReleases>
+ <overWriteSnapshots>false</overWriteSnapshots>
+ <overWriteIfNewer>true</overWriteIfNewer>
+ <includeScope>runtime</includeScope>
+ </configuration>
+ </execution>
+ <execution>
+ <id>copy-artifact</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy</goal>
+ </goals>
+ <configuration>
+ <outputDirectory>${project.build.directory}/../../interpreter/tachyon</outputDirectory>
+ <overWriteReleases>false</overWriteReleases>
+ <overWriteSnapshots>false</overWriteSnapshots>
+ <overWriteIfNewer>true</overWriteIfNewer>
+ <includeScope>runtime</includeScope>
+ <artifactItems>
+ <artifactItem>
+ <groupId>${project.groupId}</groupId>
+ <artifactId>${project.artifactId}</artifactId>
+ <version>${project.version}</version>
+ <type>${project.packaging}</type>
+ </artifactItem>
+ </artifactItems>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+</project>
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/tachyon/src/main/java/org/apache/zeppelin/tachyon/TachyonInterpreter.java
----------------------------------------------------------------------
diff --git a/tachyon/src/main/java/org/apache/zeppelin/tachyon/TachyonInterpreter.java b/tachyon/src/main/java/org/apache/zeppelin/tachyon/TachyonInterpreter.java
new file mode 100644
index 0000000..242bac3
--- /dev/null
+++ b/tachyon/src/main/java/org/apache/zeppelin/tachyon/TachyonInterpreter.java
@@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.tachyon;
+
+import java.io.IOException;
+import java.io.PrintStream;
+import java.io.ByteArrayOutputStream;
+import java.util.*;
+
+import org.apache.zeppelin.interpreter.Interpreter;
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterPropertyBuilder;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.InterpreterResult.Code;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import tachyon.conf.TachyonConf;
+import tachyon.shell.TfsShell;
+
+/**
+ * Tachyon interpreter for Zeppelin.
+ */
+public class TachyonInterpreter extends Interpreter {
+
+ Logger logger = LoggerFactory.getLogger(TachyonInterpreter.class);
+
+ protected static final String TACHYON_MASTER_HOSTNAME = "tachyon.master.hostname";
+ protected static final String TACHYON_MASTER_PORT = "tachyon.master.port";
+
+ private TfsShell tfs;
+
+ private int totalCommands = 0;
+ private int completedCommands = 0;
+
+ private final String tachyonMasterHostname;
+ private final String tachyonMasterPort;
+
+ protected final List<String> keywords = Arrays.asList("cat", "copyFromLocal",
+ "copyToLocal", "count", "du", "fileinfo", "free", "getUsedBytes",
+ "getCapacityBytes", "load", "loadMetadata", "location", "ls", "lsr",
+ "mkdir", "mount", "mv", "pin", "report", "request", "rm", "rmr",
+ "setTTL", "unsetTTL", "tail", "touch", "unmount", "unpin");
+
+ public TachyonInterpreter(Properties property) {
+ super(property);
+
+ tachyonMasterHostname = property.getProperty(TACHYON_MASTER_HOSTNAME);
+ tachyonMasterPort = property.getProperty(TACHYON_MASTER_PORT);
+ }
+
+ static {
+ Interpreter.register("tachyon", "tachyon",
+ TachyonInterpreter.class.getName(),
+ new InterpreterPropertyBuilder()
+ .add(TACHYON_MASTER_HOSTNAME, "localhost", "Tachyon master hostname")
+ .add(TACHYON_MASTER_PORT, "19998", "Tachyon master port")
+ .build());
+ }
+
+ @Override
+ public void open() {
+ logger.info("Starting Tachyon shell to connect to " + tachyonMasterHostname +
+ " on port " + tachyonMasterPort);
+
+ System.setProperty(TACHYON_MASTER_HOSTNAME, tachyonMasterHostname);
+ System.setProperty(TACHYON_MASTER_PORT, tachyonMasterPort);
+ tfs = new TfsShell(new TachyonConf());
+ }
+
+ @Override
+ public void close() {
+ logger.info("Closing Tachyon shell");
+
+ try {
+ tfs.close();
+ } catch (IOException e) {
+ logger.error("Cannot close connection", e);
+ }
+ }
+
+ @Override
+ public InterpreterResult interpret(String st, InterpreterContext context) {
+ String[] lines = splitAndRemoveEmpty(st, "\n");
+ return interpret(lines, context);
+ }
+
+ private InterpreterResult interpret(String[] commands, InterpreterContext context) {
+ boolean isSuccess = true;
+ totalCommands = commands.length;
+ completedCommands = 0;
+
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ PrintStream ps = new PrintStream(baos);
+ PrintStream old = System.out;
+
+ System.setOut(ps);
+
+ for (String command : commands) {
+ int commandResuld = 1;
+ String[] args = splitAndRemoveEmpty(command, " ");
+ if (args.length > 0 && args[0].equals("help")) {
+ System.out.println(getCommandList());
+ } else {
+ commandResuld = tfs.run(args);
+ }
+ if (commandResuld != 0) {
+ isSuccess = false;
+ break;
+ } else {
+ completedCommands += 1;
+ }
+ System.out.println();
+ }
+
+ System.out.flush();
+ System.setOut(old);
+
+ if (isSuccess) {
+ return new InterpreterResult(Code.SUCCESS, baos.toString());
+ } else {
+ return new InterpreterResult(Code.ERROR, baos.toString());
+ }
+ }
+
+ private String[] splitAndRemoveEmpty(String st, String splitSeparator) {
+ String[] voices = st.split(splitSeparator);
+ ArrayList<String> result = new ArrayList<String>();
+ for (String voice : voices) {
+ if (!voice.trim().isEmpty()) {
+ result.add(voice);
+ }
+ }
+ return result.toArray(new String[result.size()]);
+ }
+
+ private String[] splitAndRemoveEmpty(String[] sts, String splitSeparator) {
+ ArrayList<String> result = new ArrayList<String>();
+ for (String st : sts) {
+ result.addAll(Arrays.asList(splitAndRemoveEmpty(st, splitSeparator)));
+ }
+ return result.toArray(new String[result.size()]);
+ }
+
+ @Override
+ public void cancel(InterpreterContext context) { }
+
+ @Override
+ public FormType getFormType() {
+ return FormType.NATIVE;
+ }
+
+ @Override
+ public int getProgress(InterpreterContext context) {
+ return completedCommands * 100 / totalCommands;
+ }
+
+ @Override
+ public List<String> completion(String buf, int cursor) {
+ String[] words = splitAndRemoveEmpty(splitAndRemoveEmpty(buf, "\n"), " ");
+ String lastWord = "";
+ if (words.length > 0) {
+ lastWord = words[ words.length - 1 ];
+ }
+ ArrayList<String> voices = new ArrayList<String>();
+ for (String command : keywords) {
+ if (command.startsWith(lastWord)) {
+ voices.add(command);
+ }
+ }
+ return voices;
+ }
+
+ private String getCommandList() {
+ StringBuilder sb = new StringBuilder();
+ sb.append("Commands list:");
+ sb.append("\n\t[help] - List all available commands.");
+ sb.append("\n\t[cat <path>] - Print the content of the file to the console.");
+ sb.append("\n\t[copyFromLocal <src> <remoteDst>] - Copy the specified file specified " +
+ "by \"source path\" to the path specified by \"remote path\". " +
+ "This command will fail if \"remote path\" already exists.");
+ sb.append("\n\t[copyToLocal <src> <localDst>] - Copy the specified file from the path " +
+ "specified by \"remote source\" to a local destination.");
+ sb.append("\n\t[count <path>] - Display the number of folders and files matching " +
+ "the specified prefix in \"path\".");
+ sb.append("\n\t[du <path>] - Display the size of a file or a directory specified " +
+ "by the input path.");
+ sb.append("\n\t[fileinfo <path>] - Print the information of the blocks of a specified file.");
+ sb.append("\n\t[free <file path|folder path>] - Free a file or all files under a " +
+ "directory from Tachyon. If the file/directory is also in under storage, " +
+ "it will still be available there.");
+ sb.append("\n\t[getUsedBytes] - Get number of bytes used in the TachyonFS.");
+ sb.append("\n\t[getCapacityBytes] - Get the capacity of the TachyonFS.");
+ sb.append("\n\t[load <path>] - Load the data of a file or a directory from under " +
+ "storage into Tachyon.");
+ sb.append("\n\t[loadMetadata <path>] - Load the metadata of a file or a directory " +
+ "from under storage into Tachyon.");
+ sb.append("\n\t[location <path>] - Display a list of hosts that have the file data.");
+ sb.append("\n\t[ls <path>] - List all the files and directories directly under the " +
+ "given path with information such as size.");
+ sb.append("\n\t[lsr <path>] - Recursively list all the files and directories under " +
+ "the given path with information such as size.");
+ sb.append("\n\t[mkdir <path>] - Create a directory under the given path, along with " +
+ "any necessary parent directories. This command will fail if the given " +
+ "path already exists.");
+ sb.append("\n\t[mount <tachyonPath> <ufsURI>] - Mount the underlying file system " +
+ "path \"uri\" into the Tachyon namespace as \"path\". The \"path\" is assumed " +
+ "not to exist and is created by the operation. No data or metadata is loaded " +
+ "from under storage into Tachyon. After a path is mounted, operations on objects " +
+ "under the mounted path are mirror to the mounted under storage.");
+ sb.append("\n\t[mv <src> <dst>] - Move a file or directory specified by \"source\" " +
+ "to a new location \"destination\". This command will fail if " +
+ "\"destination\" already exists.");
+ sb.append("\n\t[pin <path>] - Pin the given file to avoid evicting it from memory. " +
+ "If the given path is a directory, it recursively pins all the files contained " +
+ "and any new files created within this directory.");
+ sb.append("\n\t[report <path>] - Report to the master that a file is lost.");
+ sb.append("\n\t[request <tachyonaddress> <dependencyId>] - Request the file for " +
+ "a given dependency ID.");
+ sb.append("\n\t[rm <path>] - Remove a file. This command will fail if the given " +
+ "path is a directory rather than a file.");
+ sb.append("\n\t[rmr <path>] - Remove a file, or a directory with all the files and " +
+ "sub-directories that this directory contains.");
+ sb.append("\n\t[tail <path>] - Print the last 1KB of the specified file to the console.");
+ sb.append("\n\t[touch <path>] - Create a 0-byte file at the specified location.");
+ sb.append("\n\t[unmount <tachyonPath>] - Unmount the underlying file system path " +
+ "mounted in the Tachyon namespace as \"path\". Tachyon objects under \"path\" " +
+ "are removed from Tachyon, but they still exist in the previously " +
+ "mounted under storage.");
+ sb.append("\n\t[unpin <path>] - Unpin the given file to allow Tachyon to evict " +
+ "this file again. If the given path is a directory, it recursively unpins " +
+ "all files contained and any new files created within this directory.");
+ return sb.toString();
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/tachyon/src/test/java/org/apache/zeppelin/tachyon/TachyonInterpreterTest.java
----------------------------------------------------------------------
diff --git a/tachyon/src/test/java/org/apache/zeppelin/tachyon/TachyonInterpreterTest.java b/tachyon/src/test/java/org/apache/zeppelin/tachyon/TachyonInterpreterTest.java
new file mode 100644
index 0000000..3154176
--- /dev/null
+++ b/tachyon/src/test/java/org/apache/zeppelin/tachyon/TachyonInterpreterTest.java
@@ -0,0 +1,484 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.tachyon;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.InterpreterResult.Code;
+import org.junit.*;
+
+import tachyon.Constants;
+import tachyon.TachyonURI;
+import tachyon.client.TachyonFSTestUtils;
+import tachyon.client.TachyonStorageType;
+import tachyon.client.UnderStorageType;
+import tachyon.client.file.FileInStream;
+import tachyon.client.file.TachyonFile;
+import tachyon.client.file.TachyonFileSystem;
+import tachyon.client.file.options.InStreamOptions;
+import tachyon.conf.TachyonConf;
+import tachyon.exception.ExceptionMessage;
+import tachyon.exception.TachyonException;
+import tachyon.master.LocalTachyonCluster;
+import tachyon.shell.TfsShell;
+import tachyon.thrift.FileInfo;
+import tachyon.util.FormatUtils;
+import tachyon.util.io.BufferUtils;
+import tachyon.util.io.PathUtils;
+
+
+public class TachyonInterpreterTest {
+ private TachyonInterpreter tachyonInterpreter;
+ private static final int SIZE_BYTES = Constants.MB * 10;
+ private LocalTachyonCluster mLocalTachyonCluster = null;
+ private TachyonFileSystem mTfs = null;
+
+ @After
+ public final void after() throws Exception {
+ if (tachyonInterpreter != null) {
+ tachyonInterpreter.close();
+ }
+ mLocalTachyonCluster.stop();
+ }
+
+ @Before
+ public final void before() throws Exception {
+ mLocalTachyonCluster = new LocalTachyonCluster(SIZE_BYTES, 1000, Constants.GB);
+ mLocalTachyonCluster.start();
+ mTfs = mLocalTachyonCluster.getClient();
+
+ final Properties props = new Properties();
+ props.put(TachyonInterpreter.TACHYON_MASTER_HOSTNAME, mLocalTachyonCluster.getMasterHostname());
+ props.put(TachyonInterpreter.TACHYON_MASTER_PORT, mLocalTachyonCluster.getMasterPort() + "");
+ tachyonInterpreter = new TachyonInterpreter(props);
+ tachyonInterpreter.open();
+ }
+
+ @Test
+ public void testCompletion() {
+ List<String> expectedResultOne = Arrays.asList("cat", "copyFromLocal",
+ "copyToLocal", "count");
+ List<String> expectedResultTwo = Arrays.asList("copyFromLocal",
+ "copyToLocal", "count");
+ List<String> expectedResultThree = Arrays.asList("copyFromLocal", "copyToLocal");
+ List<String> expectedResultNone = new ArrayList<String>();
+
+ List<String> resultOne = tachyonInterpreter.completion("c", 0);
+ List<String> resultTwo = tachyonInterpreter.completion("co", 0);
+ List<String> resultThree = tachyonInterpreter.completion("copy", 0);
+ List<String> resultNotMatch = tachyonInterpreter.completion("notMatch", 0);
+ List<String> resultAll = tachyonInterpreter.completion("", 0);
+
+ Assert.assertEquals(expectedResultOne, resultOne);
+ Assert.assertEquals(expectedResultTwo, resultTwo);
+ Assert.assertEquals(expectedResultThree, resultThree);
+ Assert.assertEquals(expectedResultNone, resultNotMatch);
+ Assert.assertEquals(tachyonInterpreter.keywords, resultAll);
+ }
+
+ @Test
+ public void catDirectoryTest() throws IOException {
+ String expected = "Successfully created directory /testDir\n\n" +
+ "/testDir is not a file.\n";
+
+ InterpreterResult output = tachyonInterpreter.interpret("mkdir /testDir" +
+ "\ncat /testDir", null);
+
+ Assert.assertEquals(Code.ERROR, output.code());
+ Assert.assertEquals(expected, output.message());
+ }
+
+ @Test
+ public void catNotExistTest() throws IOException {
+ InterpreterResult output = tachyonInterpreter.interpret("cat /testFile", null);
+ Assert.assertEquals(Code.ERROR, output.code());
+ }
+
+ @Test
+ public void catTest() throws IOException {
+ TachyonFSTestUtils.createByteFile(mTfs, "/testFile", TachyonStorageType.STORE,
+ UnderStorageType.NO_PERSIST, 10);
+ InterpreterResult output = tachyonInterpreter.interpret("cat /testFile", null);
+
+ byte[] expected = BufferUtils.getIncreasingByteArray(10);
+
+ Assert.assertEquals(Code.SUCCESS, output.code());
+ Assert.assertArrayEquals(expected,
+ output.message().substring(0, output.message().length() - 1).getBytes());
+ }
+
+ @Test
+ public void copyFromLocalLargeTest() throws IOException, TachyonException {
+ File testFile = new File(mLocalTachyonCluster.getTachyonHome() + "/testFile");
+ testFile.createNewFile();
+ FileOutputStream fos = new FileOutputStream(testFile);
+ byte[] toWrite = BufferUtils.getIncreasingByteArray(SIZE_BYTES);
+ fos.write(toWrite);
+ fos.close();
+
+ InterpreterResult output = tachyonInterpreter.interpret("copyFromLocal " +
+ testFile.getAbsolutePath() + " /testFile", null);
+ Assert.assertEquals(
+ "Copied " + testFile.getAbsolutePath() + " to /testFile\n\n",
+ output.message());
+
+ TachyonFile tFile = mTfs.open(new TachyonURI("/testFile"));
+ FileInfo fileInfo = mTfs.getInfo(tFile);
+ Assert.assertNotNull(fileInfo);
+ Assert.assertEquals(SIZE_BYTES, fileInfo.length);
+
+ InStreamOptions options =
+ new InStreamOptions.Builder(new TachyonConf()).setTachyonStorageType(
+ TachyonStorageType.NO_STORE).build();
+ FileInStream tfis = mTfs.getInStream(tFile, options);
+ byte[] read = new byte[SIZE_BYTES];
+ tfis.read(read);
+ Assert.assertTrue(BufferUtils.equalIncreasingByteArray(SIZE_BYTES, read));
+ }
+
+ @Test
+ public void loadFileTest() throws IOException, TachyonException {
+ TachyonFile file =
+ TachyonFSTestUtils.createByteFile(mTfs, "/testFile", TachyonStorageType.NO_STORE,
+ UnderStorageType.SYNC_PERSIST, 10);
+ FileInfo fileInfo = mTfs.getInfo(file);
+ Assert.assertFalse(fileInfo.getInMemoryPercentage() == 100);
+
+ tachyonInterpreter.interpret("load /testFile", null);
+
+ fileInfo = mTfs.getInfo(file);
+ Assert.assertTrue(fileInfo.getInMemoryPercentage() == 100);
+ }
+
+ @Test
+ public void loadDirTest() throws IOException, TachyonException {
+ TachyonFile fileA = TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testFileA",
+ TachyonStorageType.NO_STORE, UnderStorageType.SYNC_PERSIST, 10);
+ TachyonFile fileB = TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testFileB",
+ TachyonStorageType.STORE, UnderStorageType.NO_PERSIST, 10);
+ FileInfo fileInfoA = mTfs.getInfo(fileA);
+ FileInfo fileInfoB = mTfs.getInfo(fileB);
+ Assert.assertFalse(fileInfoA.getInMemoryPercentage() == 100);
+ Assert.assertTrue(fileInfoB.getInMemoryPercentage() == 100);
+
+ tachyonInterpreter.interpret("load /testRoot", null);
+
+ fileInfoA = mTfs.getInfo(fileA);
+ fileInfoB = mTfs.getInfo(fileB);
+ Assert.assertTrue(fileInfoA.getInMemoryPercentage() == 100);
+ Assert.assertTrue(fileInfoB.getInMemoryPercentage() == 100);
+ }
+
+ @Test
+ public void copyFromLocalTest() throws IOException, TachyonException {
+ File testDir = new File(mLocalTachyonCluster.getTachyonHome() + "/testDir");
+ testDir.mkdir();
+ File testDirInner = new File(mLocalTachyonCluster.getTachyonHome() + "/testDir/testDirInner");
+ testDirInner.mkdir();
+ File testFile =
+ generateFileContent("/testDir/testFile", BufferUtils.getIncreasingByteArray(10));
+
+ generateFileContent("/testDir/testDirInner/testFile2",
+ BufferUtils.getIncreasingByteArray(10, 20));
+
+ InterpreterResult output = tachyonInterpreter.interpret("copyFromLocal " +
+ testFile.getParent() + " /testDir", null);
+ Assert.assertEquals(
+ "Copied " + testFile.getParent() + " to /testDir\n\n",
+ output.message());
+
+ TachyonFile file1 = mTfs.open(new TachyonURI("/testDir/testFile"));
+ TachyonFile file2 = mTfs.open(new TachyonURI("/testDir/testDirInner/testFile2"));
+ FileInfo fileInfo1 = mTfs.getInfo(file1);
+ FileInfo fileInfo2 = mTfs.getInfo(file2);
+ Assert.assertNotNull(fileInfo1);
+ Assert.assertNotNull(fileInfo2);
+ Assert.assertEquals(10, fileInfo1.length);
+ Assert.assertEquals(20, fileInfo2.length);
+
+ byte[] read = readContent(file1, 10);
+ Assert.assertTrue(BufferUtils.equalIncreasingByteArray(10, read));
+ read = readContent(file2, 20);
+ Assert.assertTrue(BufferUtils.equalIncreasingByteArray(10, 20, read));
+ }
+
+ @Test
+ public void copyFromLocalTestWithFullURI() throws IOException, TachyonException {
+ File testFile = generateFileContent("/srcFileURI", BufferUtils.getIncreasingByteArray(10));
+ String tachyonURI = "tachyon://" + mLocalTachyonCluster.getMasterHostname() + ":"
+ + mLocalTachyonCluster.getMasterPort() + "/destFileURI";
+
+ InterpreterResult output = tachyonInterpreter.interpret("copyFromLocal " +
+ testFile.getPath() + " " + tachyonURI, null);
+ Assert.assertEquals(
+ "Copied " + testFile.getPath() + " to " + tachyonURI + "\n\n",
+ output.message());
+
+ TachyonFile file = mTfs.open(new TachyonURI("/destFileURI"));
+ FileInfo fileInfo = mTfs.getInfo(file);
+ Assert.assertEquals(10L, fileInfo.length);
+ byte[] read = readContent(file, 10);
+ Assert.assertTrue(BufferUtils.equalIncreasingByteArray(10, read));
+ }
+
+ @Test
+ public void copyFromLocalFileToDstPathTest() throws IOException, TachyonException {
+ String dataString = "copyFromLocalFileToDstPathTest";
+ byte[] data = dataString.getBytes();
+ File localDir = new File(mLocalTachyonCluster.getTachyonHome() + "/localDir");
+ localDir.mkdir();
+ File localFile = generateFileContent("/localDir/testFile", data);
+
+ tachyonInterpreter.interpret("mkdir /dstDir", null);
+ tachyonInterpreter.interpret("copyFromLocal " + localFile.getPath() + " /dstDir", null);
+
+ TachyonFile file = mTfs.open(new TachyonURI("/dstDir/testFile"));
+ FileInfo fileInfo = mTfs.getInfo(file);
+ Assert.assertNotNull(fileInfo);
+ byte[] read = readContent(file, data.length);
+ Assert.assertEquals(new String(read), dataString);
+ }
+
+ @Test
+ public void copyToLocalLargeTest() throws IOException {
+ copyToLocalWithBytes(SIZE_BYTES);
+ }
+
+ @Test
+ public void copyToLocalTest() throws IOException {
+ copyToLocalWithBytes(10);
+ }
+
+ private void copyToLocalWithBytes(int bytes) throws IOException {
+ TachyonFSTestUtils.createByteFile(mTfs, "/testFile", TachyonStorageType.STORE,
+ UnderStorageType.NO_PERSIST, bytes);
+
+ InterpreterResult output = tachyonInterpreter.interpret("copyToLocal /testFile " +
+ mLocalTachyonCluster.getTachyonHome() + "/testFile", null);
+
+ Assert.assertEquals(
+ "Copied /testFile to " + mLocalTachyonCluster.getTachyonHome() + "/testFile\n\n",
+ output.message());
+ fileReadTest("/testFile", 10);
+ }
+
+ @Test
+ public void countNotExistTest() throws IOException {
+ InterpreterResult output = tachyonInterpreter.interpret("count /NotExistFile", null);
+ Assert.assertEquals(Code.ERROR, output.code());
+ Assert.assertEquals(ExceptionMessage.PATH_DOES_NOT_EXIST.getMessage("/NotExistFile") + "\n",
+ output.message());
+ }
+
+ @Test
+ public void countTest() throws IOException {
+ TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testFileA", TachyonStorageType.STORE,
+ UnderStorageType.NO_PERSIST, 10);
+ TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testDir/testFileB", TachyonStorageType.STORE,
+ UnderStorageType.NO_PERSIST, 20);
+ TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testFileB", TachyonStorageType.STORE,
+ UnderStorageType.NO_PERSIST, 30);
+
+ InterpreterResult output = tachyonInterpreter.interpret("count /testRoot", null);
+
+ String expected = "";
+ String format = "%-25s%-25s%-15s\n";
+ expected += String.format(format, "File Count", "Folder Count", "Total Bytes");
+ expected += String.format(format, 3, 2, 60);
+ expected += "\n";
+ Assert.assertEquals(expected, output.message());
+ }
+
+ @Test
+ public void fileinfoNotExistTest() throws IOException {
+ InterpreterResult output = tachyonInterpreter.interpret("fileinfo /NotExistFile", null);
+ Assert.assertEquals(ExceptionMessage.PATH_DOES_NOT_EXIST.getMessage("/NotExistFile") + "\n",
+ output.message());
+ Assert.assertEquals(Code.ERROR, output.code());
+ }
+
+ @Test
+ public void locationNotExistTest() throws IOException {
+ InterpreterResult output = tachyonInterpreter.interpret("location /NotExistFile", null);
+ Assert.assertEquals(ExceptionMessage.PATH_DOES_NOT_EXIST.getMessage("/NotExistFile") + "\n",
+ output.message());
+ Assert.assertEquals(Code.ERROR, output.code());
+ }
+
+ @Test
+ public void lsTest() throws IOException, TachyonException {
+ FileInfo[] files = new FileInfo[3];
+
+ TachyonFile fileA = TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testFileA",
+ TachyonStorageType.STORE, UnderStorageType.NO_PERSIST, 10);
+ files[0] = mTfs.getInfo(fileA);
+ TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testDir/testFileB", TachyonStorageType.STORE,
+ UnderStorageType.NO_PERSIST, 20);
+ files[1] = mTfs.getInfo(mTfs.open(new TachyonURI("/testRoot/testDir")));
+ TachyonFile fileC = TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testFileC",
+ TachyonStorageType.NO_STORE, UnderStorageType.SYNC_PERSIST, 30);
+ files[2] = mTfs.getInfo(fileC);
+
+ InterpreterResult output = tachyonInterpreter.interpret("ls /testRoot", null);
+
+ String expected = "";
+ String format = "%-10s%-25s%-15s%-5s\n";
+ expected += String.format(format, FormatUtils.getSizeFromBytes(10),
+ TfsShell.convertMsToDate(files[0].getCreationTimeMs()), "In Memory", "/testRoot/testFileA");
+ expected += String.format(format, FormatUtils.getSizeFromBytes(0),
+ TfsShell.convertMsToDate(files[1].getCreationTimeMs()), "", "/testRoot/testDir");
+ expected += String.format(format, FormatUtils.getSizeFromBytes(30),
+ TfsShell.convertMsToDate(files[2].getCreationTimeMs()), "Not In Memory",
+ "/testRoot/testFileC");
+ expected += "\n";
+
+ Assert.assertEquals(Code.SUCCESS, output.code());
+ Assert.assertEquals(expected, output.message());
+ }
+
+ @Test
+ public void lsrTest() throws IOException, TachyonException {
+ FileInfo[] files = new FileInfo[4];
+ TachyonFile fileA = TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testFileA",
+ TachyonStorageType.STORE, UnderStorageType.NO_PERSIST, 10);
+ files[0] = mTfs.getInfo(fileA);
+ TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testDir/testFileB", TachyonStorageType.STORE,
+ UnderStorageType.NO_PERSIST, 20);
+ files[1] = mTfs.getInfo(mTfs.open(new TachyonURI("/testRoot/testDir")));
+ files[2] = mTfs.getInfo(mTfs.open(new TachyonURI("/testRoot/testDir/testFileB")));
+ TachyonFile fileC = TachyonFSTestUtils.createByteFile(mTfs, "/testRoot/testFileC",
+ TachyonStorageType.NO_STORE, UnderStorageType.SYNC_PERSIST, 30);
+ files[3] = mTfs.getInfo(fileC);
+
+ InterpreterResult output = tachyonInterpreter.interpret("lsr /testRoot", null);
+
+ String expected = "";
+ String format = "%-10s%-25s%-15s%-5s\n";
+ expected +=
+ String.format(format, FormatUtils.getSizeFromBytes(10),
+ TfsShell.convertMsToDate(files[0].getCreationTimeMs()), "In Memory",
+ "/testRoot/testFileA");
+ expected +=
+ String.format(format, FormatUtils.getSizeFromBytes(0),
+ TfsShell.convertMsToDate(files[1].getCreationTimeMs()), "", "/testRoot/testDir");
+ expected +=
+ String.format(format, FormatUtils.getSizeFromBytes(20),
+ TfsShell.convertMsToDate(files[2].getCreationTimeMs()), "In Memory",
+ "/testRoot/testDir/testFileB");
+ expected +=
+ String.format(format, FormatUtils.getSizeFromBytes(30),
+ TfsShell.convertMsToDate(files[3].getCreationTimeMs()), "Not In Memory",
+ "/testRoot/testFileC");
+ expected += "\n";
+ Assert.assertEquals(expected, output.message());
+ }
+
+ @Test
+ public void mkdirComplexPathTest() throws IOException, TachyonException {
+ InterpreterResult output = tachyonInterpreter.interpret(
+ "mkdir /Complex!@#$%^&*()-_=+[]{};\"'<>,.?/File", null);
+
+ TachyonFile tFile = mTfs.open(new TachyonURI("/Complex!@#$%^&*()-_=+[]{};\"'<>,.?/File"));
+ FileInfo fileInfo = mTfs.getInfo(tFile);
+ Assert.assertNotNull(fileInfo);
+ Assert.assertEquals(
+ "Successfully created directory /Complex!@#$%^&*()-_=+[]{};\"'<>,.?/File\n\n",
+ output.message());
+ Assert.assertTrue(fileInfo.isIsFolder());
+ }
+
+ @Test
+ public void mkdirExistingTest() throws IOException {
+ String command = "mkdir /festFile1";
+ Assert.assertEquals(Code.SUCCESS, tachyonInterpreter.interpret(command, null).code());
+ Assert.assertEquals(Code.SUCCESS, tachyonInterpreter.interpret(command, null).code());
+ }
+
+ @Test
+ public void mkdirInvalidPathTest() throws IOException {
+ Assert.assertEquals(
+ Code.ERROR,
+ tachyonInterpreter.interpret("mkdir /test File Invalid Path", null).code());
+ }
+
+ @Test
+ public void mkdirShortPathTest() throws IOException, TachyonException {
+ InterpreterResult output = tachyonInterpreter.interpret("mkdir /root/testFile1", null);
+ TachyonFile tFile = mTfs.open(new TachyonURI("/root/testFile1"));
+ FileInfo fileInfo = mTfs.getInfo(tFile);
+ Assert.assertNotNull(fileInfo);
+ Assert.assertEquals(
+ "Successfully created directory /root/testFile1\n\n",
+ output.message());
+ Assert.assertTrue(fileInfo.isIsFolder());
+ }
+
+ @Test
+ public void mkdirTest() throws IOException, TachyonException {
+ String qualifiedPath =
+ "tachyon://" + mLocalTachyonCluster.getMasterHostname() + ":"
+ + mLocalTachyonCluster.getMasterPort() + "/root/testFile1";
+ InterpreterResult output = tachyonInterpreter.interpret("mkdir " + qualifiedPath, null);
+ TachyonFile tFile = mTfs.open(new TachyonURI("/root/testFile1"));
+ FileInfo fileInfo = mTfs.getInfo(tFile);
+ Assert.assertNotNull(fileInfo);
+ Assert.assertEquals(
+ "Successfully created directory " + qualifiedPath + "\n\n",
+ output.message());
+ Assert.assertTrue(fileInfo.isIsFolder());
+ }
+
+ private File generateFileContent(String path, byte[] toWrite)
+ throws IOException, FileNotFoundException {
+ File testFile = new File(mLocalTachyonCluster.getTachyonHome() + path);
+ testFile.createNewFile();
+ FileOutputStream fos = new FileOutputStream(testFile);
+ fos.write(toWrite);
+ fos.close();
+ return testFile;
+ }
+
+ private byte[] readContent(TachyonFile tFile, int length) throws IOException, TachyonException {
+ InStreamOptions options =
+ new InStreamOptions.Builder(new TachyonConf()).setTachyonStorageType(
+ TachyonStorageType.NO_STORE).build();
+ FileInStream tfis = mTfs.getInStream(tFile, options);
+ byte[] read = new byte[length];
+ tfis.read(read);
+ return read;
+ }
+
+ private void fileReadTest(String fileName, int size) throws IOException {
+ File testFile = new File(PathUtils.concatPath(mLocalTachyonCluster.getTachyonHome(), fileName));
+ FileInputStream fis = new FileInputStream(testFile);
+ byte[] read = new byte[size];
+ fis.read(read);
+ fis.close();
+ Assert.assertTrue(BufferUtils.equalIncreasingByteArray(size, read));
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/zeppelin-distribution/src/bin_license/LICENSE
----------------------------------------------------------------------
diff --git a/zeppelin-distribution/src/bin_license/LICENSE b/zeppelin-distribution/src/bin_license/LICENSE
index a5324d0..36e9836 100644
--- a/zeppelin-distribution/src/bin_license/LICENSE
+++ b/zeppelin-distribution/src/bin_license/LICENSE
@@ -94,6 +94,10 @@ The following components are provided under Apache License.
(Apache 2.0) Shiro Core (org.apache.shiro:shiro-core:1.2.3 - https://shiro.apache.org)
(Apache 2.0) Shiro Web (org.apache.shiro:shiro-web:1.2.3 - https://shiro.apache.org)
(Apache 2.0) SnakeYAML (org.yaml:snakeyaml:1.15 - http://www.snakeyaml.org)
+ (Apache 2.0) Tachyon Shell (org.tachyonproject:tachyon-shell:0.8.2 - http://tachyon-project.org)
+ (Apache 2.0) Tachyon Servers (org.tachyonproject:tachyon-servers:0.8.2 - http://tachyon-project.org)
+ (Apache 2.0) Tachyon Minicluster (org.tachyonproject:tachyon-minicluster:0.8.2 - http://tachyon-project.org)
+ (Apache 2.0) Tachyon Underfs Local (org.tachyonproject:tachyon-underfs-local:0.8.2 - http://tachyon-project.org)
http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/1b3784d2/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
index 231c5c6..6d726f5 100755
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
@@ -441,6 +441,7 @@ public class ZeppelinConfiguration extends XMLConfiguration {
+ "org.apache.zeppelin.angular.AngularInterpreter,"
+ "org.apache.zeppelin.shell.ShellInterpreter,"
+ "org.apache.zeppelin.hive.HiveInterpreter,"
+ + "org.apache.zeppelin.tachyon.TachyonInterpreter,"
+ "org.apache.zeppelin.phoenix.PhoenixInterpreter,"
+ "org.apache.zeppelin.postgresql.PostgreSqlInterpreter,"
+ "org.apache.zeppelin.tajo.TajoInterpreter,"