You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by mo...@apache.org on 2017/01/28 22:45:27 UTC
zeppelin git commit: [ZEPPELIN-2004] List all helium packages in
Zeppelin GUI
Repository: zeppelin
Updated Branches:
refs/heads/master fe2fffe08 -> 940a8b7d3
[ZEPPELIN-2004] List all helium packages in Zeppelin GUI
### What is this PR for?
ZEPPELIN-1973 will provides catalogue for all available helium (visualization) packages in npm registry. And https://github.com/apache/zeppelin/pull/1935 shows available packages in Zeppelin website.
This PR make Zeppelin reads package information and display in Zeppelin's helium gui menu.
To do that, this PR changes configuration environment variable (java property) from
```
ZEPPELIN_HELIUM_LOCALREGISTRY_DEFAULT (zeppelin.helium.localregistry.default)
```
to
```
ZEPPELIN_HELIUM_REGISTRY (zeppelin.helium.registry)
```
and allow multiple comma separated items.
Registry is either filesystem directory (e.g. `/helium`) or http location.
default value is `helium,https://s3.amazonaws.com/helium-package/helium.json`
### What type of PR is it?
Feature
### Todos
* [x] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2004
### How should this be tested?
Go to helium menu and check if you can see packages available.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? yes
`ZEPPELIN_HELIUM_LOCALREGISTRY_DEFAULT` changed to `ZEPPELIN_HELIUM_REGISTRY`
* Does this needs documentation? no
Author: Lee moon soo <mo...@apache.org>
Closes #1936 from Leemoonsoo/ZEPPELIN-2004 and squashes the following commits:
80bbebd [Lee moon soo] update doc
53f0caa [Lee moon soo] update test
65c6092 [Lee moon soo] Cache online registry under local-repo for offline support
b2985e9 [Lee moon soo] Add user agent
f562b12 [Lee moon soo] Use only local helium registry to run test
9fabeae [Lee moon soo] implement online registry
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/940a8b7d
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/940a8b7d
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/940a8b7d
Branch: refs/heads/master
Commit: 940a8b7d369015faf0f923ec5b58dc664ea66362
Parents: fe2fffe
Author: Lee moon soo <mo...@apache.org>
Authored: Fri Jan 27 05:38:32 2017 +0900
Committer: Lee moon soo <mo...@apache.org>
Committed: Sun Jan 29 07:45:28 2017 +0900
----------------------------------------------------------------------
.travis.yml | 1 +
.../development/writingzeppelinvisualization.md | 32 ++---
.../apache/zeppelin/server/ZeppelinServer.java | 4 +-
.../zeppelin/conf/ZeppelinConfiguration.java | 9 +-
.../java/org/apache/zeppelin/helium/Helium.java | 30 +++--
.../org/apache/zeppelin/helium/HeliumConf.java | 11 --
.../zeppelin/helium/HeliumOnlineRegistry.java | 133 +++++++++++++++++++
.../org/apache/zeppelin/helium/HeliumTest.java | 13 +-
8 files changed, 181 insertions(+), 52 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/940a8b7d/.travis.yml
----------------------------------------------------------------------
diff --git a/.travis.yml b/.travis.yml
index c2a47e5..0e0ad69 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -98,6 +98,7 @@ before_script:
- if [[ -n $LIVY_VER ]]; then export LIVY_HOME=`pwd`/livy-server-$LIVY_VER; fi
- if [[ -n $LIVY_VER ]]; then export SPARK_HOME=`pwd`/spark-$SPARK_VER-bin-hadoop$HADOOP_VER; fi
- echo "export SPARK_HOME=`pwd`/spark-$SPARK_VER-bin-hadoop$HADOOP_VER" > conf/zeppelin-env.sh
+ - echo "export ZEPPELIN_HELIUM_REGISTRY=helium" >> conf/zeppelin-env.sh
- tail conf/zeppelin-env.sh
script:
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/940a8b7d/docs/development/writingzeppelinvisualization.md
----------------------------------------------------------------------
diff --git a/docs/development/writingzeppelinvisualization.md b/docs/development/writingzeppelinvisualization.md
index ab46958..f7e1408 100644
--- a/docs/development/writingzeppelinvisualization.md
+++ b/docs/development/writingzeppelinvisualization.md
@@ -32,27 +32,12 @@ Apache Zeppelin Visualization is a pluggable package that can be loaded/unloaded
#### 1. Load Helium package files from registry
-Zeppelin needs to know what Visualization packages are available. Zeppelin searches _Helium package file_ from local registry (by default helium/ directory) by default.
-_Helium package file_ provides informations like name, artifact, and so on. It's similar to _package.json_ in npm package.
-
-Here's an example `helium/zeppelin-example-horizontalbar.json`
-
-```
-{
- "type" : "VISUALIZATION",
- "name" : "zeppelin_horizontalbar",
- "description" : "Horizontal Bar chart (example)",
- "artifact" : "./zeppelin-examples/zeppelin-example-horizontalbar",
- "license" : "Apache-2.0",
- "icon" : "<i class='fa fa-bar-chart rotate90flipX'></i>"
-}
-```
-
-Check [Create helium package file](#3-create-helium-package-file) section to learn about it.
+Zeppelin needs to know what Visualization packages are available. Zeppelin will read information of packages from both online and local registry.
+Registries are configurable through `ZEPPELIN_HELIUM_LOCALREGISTRY_DEFAULT` env variable or `zeppelin.helium.localregistry.default` property.
#### 2. Enable packages
-Once Zeppelin loads _Helium package files_ from local registry, available packages are displayed in Helium menu.
+Once Zeppelin loads _Helium package files_ from registries, available packages are displayed in Helium menu.
Click 'enable' button.
@@ -129,7 +114,7 @@ You can check complete visualization package example [here](https://github.com/a
Zeppelin's built-in visualization uses the same API, so you can check [built-in visualizations](https://github.com/apache/zeppelin/tree/master/zeppelin-web/src/app/visualization/builtins) as additional examples.
-#### 3. Create __Helium package file__
+#### 3. Create __Helium package file__ and locally deploy
__Helium Package file__ is a json file that provides information about the application.
Json file contains the following information
@@ -145,6 +130,9 @@ Json file contains the following information
}
```
+Place file in your local registry directory (default `./helium`).
+
+
##### type
When you're creating a visualization, 'type' should be 'VISUALIZATION'.
@@ -210,3 +198,9 @@ yarn run visdev
```
You can browse localhost:9000. Everytime refresh your browser, Zeppelin will rebuild your visualization and reload changes.
+
+
+#### 5. Publish your visualization
+
+Once it's done, publish your visualization package using `npm publish`.
+That's it. With in an hour, your visualization will be available in Zeppelin's helium menu.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/940a8b7d/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java
----------------------------------------------------------------------
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java b/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java
index 2cc2fec..6c4fcd8 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java
@@ -123,7 +123,9 @@ public class ZeppelinServer extends Application {
this.helium = new Helium(
conf.getHeliumConfPath(),
- conf.getHeliumDefaultLocalRegistryPath(),
+ conf.getHeliumRegistry(),
+ new File(
+ conf.getRelativeDir(ConfVars.ZEPPELIN_DEP_LOCALREPO), "helium_registry_cache"),
heliumVisualizationFactory,
heliumApplicationFactory);
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/940a8b7d/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
index 388f432..c0cae07 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
@@ -41,6 +41,9 @@ public class ZeppelinConfiguration extends XMLConfiguration {
private static final String ZEPPELIN_SITE_XML = "zeppelin-site.xml";
private static final long serialVersionUID = 4749305895693848035L;
private static final Logger LOG = LoggerFactory.getLogger(ZeppelinConfiguration.class);
+
+ private static final String HELIUM_PACKAGE_DEFAULT_URL =
+ "https://s3.amazonaws.com/helium-package/helium.json";
private static ZeppelinConfiguration conf;
public ZeppelinConfiguration(URL url) throws ConfigurationException {
@@ -397,8 +400,8 @@ public class ZeppelinConfiguration extends XMLConfiguration {
return getRelativeDir(String.format("%s/helium.json", getConfDir()));
}
- public String getHeliumDefaultLocalRegistryPath() {
- return getRelativeDir(ConfVars.ZEPPELIN_HELIUM_LOCALREGISTRY_DEFAULT);
+ public String getHeliumRegistry() {
+ return getRelativeDir(ConfVars.ZEPPELIN_HELIUM_REGISTRY);
}
public String getNotebookAuthorizationPath() {
@@ -599,7 +602,7 @@ public class ZeppelinConfiguration extends XMLConfiguration {
ZEPPELIN_NOTEBOOK_AUTO_INTERPRETER_BINDING("zeppelin.notebook.autoInterpreterBinding", true),
ZEPPELIN_CONF_DIR("zeppelin.conf.dir", "conf"),
ZEPPELIN_DEP_LOCALREPO("zeppelin.dep.localrepo", "local-repo"),
- ZEPPELIN_HELIUM_LOCALREGISTRY_DEFAULT("zeppelin.helium.localregistry.default", "helium"),
+ ZEPPELIN_HELIUM_REGISTRY("zeppelin.helium.registry", "helium," + HELIUM_PACKAGE_DEFAULT_URL),
// Allows a way to specify a ',' separated list of allowed origins for rest and websockets
// i.e. http://localhost:8080
ZEPPELIN_ALLOWED_ORIGINS("zeppelin.server.allowed.origins", "*"),
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/940a8b7d/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
index b8584ef..0ef3237 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
@@ -41,19 +41,23 @@ public class Helium {
private final HeliumConf heliumConf;
private final String heliumConfPath;
- private final String defaultLocalRegistryPath;
+ private final String registryPaths;
+ private final File registryCacheDir;
+
private final Gson gson;
private final HeliumVisualizationFactory visualizationFactory;
private final HeliumApplicationFactory applicationFactory;
public Helium(
String heliumConfPath,
- String defaultLocalRegistryPath,
+ String registryPaths,
+ File registryCacheDir,
HeliumVisualizationFactory visualizationFactory,
HeliumApplicationFactory applicationFactory)
throws IOException {
this.heliumConfPath = heliumConfPath;
- this.defaultLocalRegistryPath = defaultLocalRegistryPath;
+ this.registryPaths = registryPaths;
+ this.registryCacheDir = registryCacheDir;
this.visualizationFactory = visualizationFactory;
this.applicationFactory = applicationFactory;
@@ -96,19 +100,28 @@ public class Helium {
}
private synchronized HeliumConf loadConf(String path) throws IOException {
+ // add registry
+ if (registryPaths != null && !registryPaths.isEmpty()) {
+ String[] paths = registryPaths.split(",");
+ for (String uri : paths) {
+ if (uri.startsWith("http://") || uri.startsWith("https://")) {
+ logger.info("Add helium online registry {}", uri);
+ registry.add(new HeliumOnlineRegistry(uri, uri, registryCacheDir));
+ } else {
+ logger.info("Add helium local registry {}", uri);
+ registry.add(new HeliumLocalRegistry(uri, uri));
+ }
+ }
+ }
+
File heliumConfFile = new File(path);
if (!heliumConfFile.isFile()) {
logger.warn("{} does not exists", path);
HeliumConf conf = new HeliumConf();
- LinkedList<HeliumRegistry> defaultRegistry = new LinkedList<>();
- defaultRegistry.add(new HeliumLocalRegistry("local", defaultLocalRegistryPath));
- conf.setRegistry(defaultRegistry);
- this.registry = conf.getRegistry();
return conf;
} else {
String jsonString = FileUtils.readFileToString(heliumConfFile);
HeliumConf conf = gson.fromJson(jsonString, HeliumConf.class);
- this.registry = conf.getRegistry();
return conf;
}
}
@@ -117,7 +130,6 @@ public class Helium {
String jsonString;
synchronized (registry) {
clearNotExistsPackages();
- heliumConf.setRegistry(registry);
jsonString = gson.toJson(heliumConf);
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/940a8b7d/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/HeliumConf.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/HeliumConf.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/HeliumConf.java
index d3b0fb4..5094934 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/HeliumConf.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/HeliumConf.java
@@ -22,23 +22,12 @@ import java.util.*;
* Helium config. This object will be persisted to conf/heliumc.conf
*/
public class HeliumConf {
- List<HeliumRegistry> registry = new LinkedList<>();
-
// enabled packages {name, version}
Map<String, String> enabled = Collections.synchronizedMap(new HashMap<String, String>());
// enabled visualization package display order
List<String> visualizationDisplayOrder = new LinkedList<>();
-
- public List<HeliumRegistry> getRegistry() {
- return registry;
- }
-
- public void setRegistry(List<HeliumRegistry> registry) {
- this.registry = registry;
- }
-
public Map<String, String> getEnabledPackages() {
return new HashMap<>(enabled);
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/940a8b7d/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/HeliumOnlineRegistry.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/HeliumOnlineRegistry.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/HeliumOnlineRegistry.java
new file mode 100644
index 0000000..d6bf499
--- /dev/null
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/HeliumOnlineRegistry.java
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.helium;
+
+import com.google.gson.Gson;
+import com.google.gson.reflect.TypeToken;
+import org.apache.commons.io.FileUtils;
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.zeppelin.util.Util;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.*;
+import java.net.URL;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * This registry reads helium package json data
+ * from specified url.
+ *
+ * File should be look like
+ * [
+ * "packageName": {
+ * "0.0.1": json serialized HeliumPackage class,
+ * "0.0.2": json serialized HeliumPackage class,
+ * ...
+ * },
+ * ...
+ * ]
+ */
+public class HeliumOnlineRegistry extends HeliumRegistry {
+ Logger logger = LoggerFactory.getLogger(HeliumOnlineRegistry.class);
+ private final Gson gson;
+ private final File registryCacheFile;
+
+ public HeliumOnlineRegistry(String name, String uri, File registryCacheDir) {
+ super(name, uri);
+ registryCacheDir.mkdirs();
+ this.registryCacheFile = new File(registryCacheDir, name);
+ gson = new Gson();
+ }
+
+ @Override
+ public synchronized List<HeliumPackage> getAll() throws IOException {
+ HttpClient client = HttpClientBuilder.create()
+ .setUserAgent("ApacheZeppelin/" + Util.getVersion())
+ .build();
+ HttpGet get = new HttpGet(uri());
+ HttpResponse response;
+
+ try {
+ response = client.execute(get);
+ } catch (Exception e) {
+ logger.error(e.getMessage());
+ return readFromCache();
+ }
+
+
+ if (response.getStatusLine().getStatusCode() != 200) {
+ // try read from cache
+ logger.error(uri() + " returned " + response.getStatusLine().toString());
+ return readFromCache();
+ } else {
+ List<HeliumPackage> packageList = new LinkedList<>();
+
+ BufferedReader reader;
+ reader = new BufferedReader(
+ new InputStreamReader(response.getEntity().getContent()));
+
+ List<Map<String, Map<String, HeliumPackage>>> packages = gson.fromJson(
+ reader,
+ new TypeToken<List<Map<String, Map<String, HeliumPackage>>>>() {
+ }.getType());
+ reader.close();
+
+ for (Map<String, Map<String, HeliumPackage>> pkg : packages) {
+ for (Map<String, HeliumPackage> versions : pkg.values()) {
+ packageList.addAll(versions.values());
+ }
+ }
+
+ writeToCache(packageList);
+ return packageList;
+ }
+ }
+
+ private List<HeliumPackage> readFromCache() {
+ synchronized (registryCacheFile) {
+ if (registryCacheFile.isFile()) {
+ try {
+ return gson.fromJson(
+ new FileReader(registryCacheFile),
+ new TypeToken<List<HeliumPackage>>() {
+ }.getType());
+ } catch (FileNotFoundException e) {
+ logger.error(e.getMessage(), e);
+ return new LinkedList<>();
+ }
+ } else {
+ return new LinkedList<>();
+ }
+ }
+ }
+
+ private void writeToCache(List<HeliumPackage> pkg) throws IOException {
+ synchronized (registryCacheFile) {
+ if (registryCacheFile.exists()) {
+ registryCacheFile.delete();
+ }
+ String jsonToCache = gson.toJson(pkg);
+ FileUtils.writeStringToFile(registryCacheFile, jsonToCache);
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/940a8b7d/zeppelin-zengine/src/test/java/org/apache/zeppelin/helium/HeliumTest.java
----------------------------------------------------------------------
diff --git a/zeppelin-zengine/src/test/java/org/apache/zeppelin/helium/HeliumTest.java b/zeppelin-zengine/src/test/java/org/apache/zeppelin/helium/HeliumTest.java
index 7c168d0..9db9477 100644
--- a/zeppelin-zengine/src/test/java/org/apache/zeppelin/helium/HeliumTest.java
+++ b/zeppelin-zengine/src/test/java/org/apache/zeppelin/helium/HeliumTest.java
@@ -53,12 +53,8 @@ public class HeliumTest {
// given
File heliumConf = new File(tmpDir, "helium.conf");
Helium helium = new Helium(heliumConf.getAbsolutePath(), localRegistryPath.getAbsolutePath(),
- null, null);
+ null, null, null);
assertFalse(heliumConf.exists());
- HeliumTestRegistry registry1 = new HeliumTestRegistry("r1", "r1");
- helium.addRegistry(registry1);
- assertEquals(2, helium.getAllRegistry().size());
- assertEquals(0, helium.getAllPackageInfo().size());
// when
helium.save();
@@ -66,17 +62,16 @@ public class HeliumTest {
// then
assertTrue(heliumConf.exists());
- // then
+ // then load without exception
Helium heliumRestored = new Helium(
- heliumConf.getAbsolutePath(), localRegistryPath.getAbsolutePath(), null, null);
- assertEquals(2, heliumRestored.getAllRegistry().size());
+ heliumConf.getAbsolutePath(), localRegistryPath.getAbsolutePath(), null, null, null);
}
@Test
public void testRestoreRegistryInstances() throws IOException, URISyntaxException, TaskRunnerException {
File heliumConf = new File(tmpDir, "helium.conf");
Helium helium = new Helium(
- heliumConf.getAbsolutePath(), localRegistryPath.getAbsolutePath(), null, null);
+ heliumConf.getAbsolutePath(), localRegistryPath.getAbsolutePath(), null, null, null);
HeliumTestRegistry registry1 = new HeliumTestRegistry("r1", "r1");
HeliumTestRegistry registry2 = new HeliumTestRegistry("r2", "r2");
helium.addRegistry(registry1);