svn commit: r1818166 [4/6] - in /zeppelin/site/docs/0.8.0-SNAPSHOT: ./ assets/themes/zeppelin/img/ assets/themes/zeppelin/img/docs-img/ assets/themes/zeppelin/img/screenshots/ development/ development/contribution/ development/helium/ interpreter/ quic...

[mo...@apache.org]

Modified: zeppelin/site/docs/0.8.0-SNAPSHOT/search_data.json
URL: http://svn.apache.org/viewvc/zeppelin/site/docs/0.8.0-SNAPSHOT/search_data.json?rev=1818166&r1=1818165&r2=1818166&view=diff
==============================================================================
--- zeppelin/site/docs/0.8.0-SNAPSHOT/search_data.json (original)
+++ zeppelin/site/docs/0.8.0-SNAPSHOT/search_data.json Thu Dec 14 16:46:23 2017
@@ -1,169 +1,144 @@
 {
   
-  
-  
-
-    "/development/contribution/how_to_contribute_code.html": {
-      "title": "Contributing to Apache Zeppelin (Code)",
-      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Contributing to Apache Zeppelin ( Code )NOTE : Apache Zeppelin is an Apache2 License Software.Any contributions to Zeppelin (Source code, Documents, Image, Website) means you agree with license all your contributions as Apache2 License.Setting upHere are some tools you will need to build and test Zeppelin.Software Configuration Management ( SCM )Since Zeppelin uses Git for it's SCM system, you need git client installed in y
 our development machine.Integrated Development Environment ( IDE )You are free to use whatever IDE you prefer, or your favorite command line editor.Build ToolsTo build the code, installOracle Java 7Apache MavenGetting the source codeFirst of all, you need Zeppelin source code. The official location of Zeppelin is http://git.apache.org/zeppelin.git.git accessGet the source code on your development machine using git.git clone git://git.apache.org/zeppelin.git zeppelinYou may also want to develop against a specific branch. For example, for branch-0.5.6git clone -b branch-0.5.6 git://git.apache.org/zeppelin.git zeppelinApache Zeppelin follows Fork & Pull as a source control workflow.If you want to not only build Zeppelin but also make any changes, then you need to fork Zeppelin github mirror repository and make a pull request.Before making a pull request, please take a look Contribution Guidelines.Buildmvn installTo skip testmvn install -DskipTestsTo build with specific spark / 
 hadoop versionmvn install -Dspark.version=x.x.x -Dhadoop.version=x.x.xFor the further Run Zeppelin server in development modecd zeppelin-serverHADOOP_HOME=YOUR_HADOOP_HOME JAVA_HOME=YOUR_JAVA_HOME mvn exec:java -Dexec.mainClass="org.apache.zeppelin.server.ZeppelinServer" -Dexec.args=""Note: Make sure you first run mvn clean install -DskipTests on your zeppelin root directory, otherwise your server build will fail to find the required dependencies in the local repro.or use daemon scriptbin/zeppelin-daemon startServer will be run on http://localhost:8080.Generating Thrift CodeSome portions of the Zeppelin code are generated by Thrift. For most Zeppelin changes, you don't need to worry about this. But if you modify any of the Thrift IDL files (e.g. zeppelin-interpreter/src/main/thrift/*.thrift), then you also need to regenerate these files and submit their updated version as part of your patch.To regenerate the code, install thrift-0.9.2 and 
 then run the following command to generate thrift code.cd <zeppelin_home>/zeppelin-interpreter/src/main/thrift./genthrift.shRun Selenium testZeppelin has set of integration tests using Selenium. To run these test, first build and run Zeppelin and make sure Zeppelin is running on port 8080. Then you can run test using following commandTEST_SELENIUM=true mvn test -Dtest=[TEST_NAME] -DfailIfNoTests=false -pl 'zeppelin-interpreter,zeppelin-zengine,zeppelin-server'For example, to run ParagraphActionIT,TEST_SELENIUM=true mvn test -Dtest=ParagraphActionsIT -DfailIfNoTests=false -pl 'zeppelin-interpreter,zeppelin-zengine,zeppelin-server'You'll need Firefox web browser installed in your development environment. While CI server uses Firefox 31.0 to run selenium test, it is good idea to install the same version (disable auto update to keep the version).Where to StartYou can find issues for beginner & newbieStay involvedContributors 
 should join the Zeppelin mailing lists.dev@zeppelin.apache.org is for people who want to contribute code to Zeppelin. subscribe, unsubscribe, archivesIf you have any issues, create a ticket in JIRA.",
-      "url": " /development/contribution/how_to_contribute_code.html",
-      "group": "development/contribution",
-      "excerpt": "How can you contribute to Apache Zeppelin project? This document covers from setting up your develop environment to making a pull request on Github."
-    }
-    ,
-    
-  
-
-    "/development/contribution/how_to_contribute_website.html": {
-      "title": "Contributing to Apache Zeppelin (Website)",
-      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Contributing to Apache Zeppelin ( Website )This page will give you an overview of how to build and contribute to the documentation of Apache Zeppelin.The online documentation at zeppelin.apache.org is also generated from the files found here.NOTE : Apache Zeppelin is an Apache2 License Software.Any contributions to Zeppelin (Source code, Documents, Image, Website) means you agree with license all your contributions as Apache2 Licen
 se.Getting the source codeFirst of all, you need Zeppelin source code. The official location of Zeppelin is http://git.apache.org/zeppelin.git.Documentation website is hosted in 'master' branch under /docs/ dir.git accessFirst of all, you need the website source code. The official location of mirror for Zeppelin is http://git.apache.org/zeppelin.git.Get the source code on your development machine using git.git clone git://git.apache.org/zeppelin.gitcd docsApache Zeppelin follows Fork & Pull as a source control workflow.If you want to not only build Zeppelin but also make any changes, then you need to fork Zeppelin github mirror repository and make a pull request.BuildYou'll need to install some prerequisites to build the code. Please check Build documentation section in docs/README.md.Run website in development modeWhile you're modifying website, you might want to see preview of it. Please check Run website section in docs/README.md.Then you&a
 mp;#39;ll be able to access it on http://localhost:4000 with your web browser.Making a Pull RequestWhen you are ready, just make a pull-request.Alternative wayYou can directly edit .md files in /docs/ directory at the web interface of github and make pull-request immediately.Stay involvedContributors should join the Zeppelin mailing lists.dev@zeppelin.apache.org is for people who want to contribute code to Zeppelin. subscribe, unsubscribe, archivesIf you have any issues, create a ticket in JIRA.",
-      "url": " /development/contribution/how_to_contribute_website.html",
-      "group": "development/contribution",
-      "excerpt": "How can you contribute to Apache Zeppelin project website? This document covers from building Zeppelin documentation site to making a pull request on Github."
-    }
-    ,
-    
-  
 
-    "/development/contribution/useful_developer_tools.html": {
-      "title": "Useful Developer Tools",
-      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Useful Developer ToolsDeveloping zeppelin-webCheck zeppelin-web: Local Development.ToolsSVM: Scala Version Managersvm would be useful when changing scala version frequently.JDK change script: OSXthis script would be helpful when changing JDK version frequently.function setjdk() {  if [ $# -ne 0 ]; then  # written based on OSX.   # use diffrent base path for other OS  removeFromPath '/System/Library/Frameworks/JavaVM.framewo
 rk/Home/bin'  if [ -n "${JAVA_HOME+x}" ]; then    removeFromPath $JAVA_HOME  fi  export JAVA_HOME=`/usr/libexec/java_home -v $@`  export PATH=$JAVA_HOME/bin:$PATH  fi}function removeFromPath() {  export PATH=$(echo $PATH | sed -E -e "s;:$1;;" -e "s;$1:?;;")}you can use this function like setjdk 1.8 / setjdk 1.7Building Submodules Selectively# build `zeppelin-web` onlymvn clean -pl 'zeppelin-web' package -DskipTests;# build `zeppelin-server` and its dependencies onlymvn clean package -pl 'spark,spark-dependencies,python,markdown,zeppelin-server' --am -DskipTests# build spark related modules with default profiles: scala 2.10 mvn clean package -pl 'spark,spark-dependencies,zeppelin-server' --am -DskipTests# build spark related modules with profiles: scala 2.11, spark 2.1 hadoop 2.7 ./dev/change_scala_version.sh 2.11mvn clean package -Pspark-2.1 -Phadoop-2.7 -Pscala-2.11 -pl &am
 p;#39;spark,spark-dependencies,zeppelin-server' --am -DskipTests# build `zeppelin-server` and `markdown` with dependenciesmvn clean package -pl 'markdown,zeppelin-server' --am -DskipTestsRunning Individual Tests# run the `HeliumBundleFactoryTest` test classmvn test -pl 'zeppelin-server' --am -DfailIfNoTests=false -Dtest=HeliumBundleFactoryTestRunning Selenium TestsMake sure that Zeppelin instance is started to execute integration tests (= selenium tests).# run the `SparkParagraphIT` test classTEST_SELENIUM="true" mvn test -pl 'zeppelin-server' --am -DfailIfNoTests=false -Dtest=SparkParagraphIT# run the `testSqlSpark` test function only in the `SparkParagraphIT` class# but note that, some test might be dependent on the previous testsTEST_SELENIUM="true" mvn test -pl 'zeppelin-server' --am -DfailIfNoTests=false -Dtest=SparkParagraphIT#testSqlSpark",
-      "url": " /development/contribution/useful_developer_tools.html",
-      "group": "development/contribution",
-      "excerpt": ""
+    "/interpreter/livy.html": {
+      "title": "Livy Interpreter for Apache Zeppelin",
+      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Livy Interpreter for Apache ZeppelinOverviewLivy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN.Interactive Scala, Python and R shellsBatch submissions in Scala, Java, PythonMulti users can share the same server (impersonation support)Can be used for submitting jobs from anywhere with RESTDoes not require a
 ny code change to your programsRequirementsAdditional requirements for the Livy interpreter are:Spark 1.3 or above.Livy server.ConfigurationWe added some common configurations for spark, and you can set any configuration you want.You can find all Spark configurations in here.And instead of starting property with spark. it should be replaced with livy.spark..Example: spark.driver.memory to livy.spark.driver.memory      Property    Default    Description        zeppelin.livy.url    http://localhost:8998    URL where livy server is running        zeppelin.livy.spark.sql.maxResult    1000    Max number of Spark SQL result to display.        zeppelin.livy.spark.sql.field.truncate    true    Whether to truncate field values longer than 20 characters or not        zeppelin.livy.session.create_timeout    120    Timeout in seconds for session creation        zeppelin.livy.displayAppInfo    true    Whether to display app info        zeppelin.livy.pull_status.interval.millis    1000    The int
 erval for checking paragraph execution status        livy.spark.driver.cores        Driver cores. ex) 1, 2.          livy.spark.driver.memory        Driver memory. ex) 512m, 32g.          livy.spark.executor.instances        Executor instances. ex) 1, 4.          livy.spark.executor.cores        Num cores per executor. ex) 1, 4.        livy.spark.executor.memory        Executor memory per worker instance. ex) 512m, 32g.        livy.spark.dynamicAllocation.enabled        Use dynamic resource allocation. ex) True, False.        livy.spark.dynamicAllocation.cachedExecutorIdleTimeout        Remove an executor which has cached data blocks.        livy.spark.dynamicAllocation.minExecutors        Lower bound for the number of executors.        livy.spark.dynamicAllocation.initialExecutors        Initial number of executors to run.        livy.spark.dynamicAllocation.maxExecutors        Upper bound for the number of executors.            livy.spark.jars.packages            Adding extra libr
 aries to livy interpreter          zeppelin.livy.ssl.trustStore        client trustStore file. Used when livy ssl is enabled        zeppelin.livy.ssl.trustStorePassword        password for trustStore file. Used when livy ssl is enabled        zeppelin.livy.http.headers    key_1: value_1; key_2: value_2    custom http headers when calling livy rest api. Each http header is separated by `;`, and each header is one key value pair where key value is separated by `:`  We remove livy.spark.master in zeppelin-0.7. Because we sugguest user to use livy 0.3 in zeppelin-0.7. And livy 0.3 don't allow to specify livy.spark.master, it enfornce yarn-cluster mode.Adding External librariesYou can load dynamic library to livy interpreter by set livy.spark.jars.packages property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version.Example      Property    Example    Description
           livy.spark.jars.packages      io.spray:spray-json_2.10:1.3.1      Adding extra libraries to livy interpreter      How to useBasically, you can usespark%livy.sparksc.versionpyspark%livy.pysparkprint "1"sparkR%livy.sparkrhello <- function( name ) {    sprintf( "Hello, %s", name );}hello("livy")ImpersonationWhen Zeppelin server is running with authentication enabled,then this interpreter utilizes Livy’s user impersonation featurei.e. sends extra parameter for creating and running a session ("proxyUser": "${loggedInUser}").This is particularly useful when multi users are sharing a Notebook server.Apply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form. Form templates is only avalible for livy sql interpreter.%livy.sqlselect * from products where ${product_id=1}And creating dynamic formst programmatically is not feasible in livy interpreter, because ZeppelinContext i
 s not available in livy interpreter.FAQLivy debugging: If you see any of these in error consoleConnect to livyhost:8998 [livyhost/127.0.0.1, livyhost/0:0:0:0:0:0:0:1] failed: Connection refusedLooks like the livy server is not up yet or the config is wrongException: Session not found, Livy server would have restarted, or lost session.The session would have timed out, you may need to restart the interpreter.Blacklisted configuration values in session config: spark.masterEdit conf/spark-blacklist.conf file in livy server and comment out #spark.master line.If you choose to work on livy in apps/spark/java directory in https://github.com/cloudera/hue,copy spark-user-configurable-options.template to spark-user-configurable-options.conf file in livy server and comment out #spark.master.",
+      "url": " /interpreter/livy.html",
+      "group": "interpreter",
+      "excerpt": "Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN."
     }
     ,
     
   
 
-    "/development/helium/overview.html": {
-      "title": "Helium",
-      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Helium OverviewWhat is Helium?Helium is a plugin system that can extend Zeppelin a lot. For example, you can write custom display system or install already published one in Heliun Online Registry. Currently, Helium supports 4 types of package.Helium Visualization: Adding a new chart typeHelium Spell: Adding new interpreter, display system running on browserHelium Application Helium Interpreter: Adding a new custom interpreter",
-      "url": " /development/helium/overview.html",
-      "group": "development/helium",
-      "excerpt": ""
+    "/interpreter/pig.html": {
+      "title": "Pig Interpreter for Apache Zeppelin",
+      "content"  : "Pig Interpreter for Apache ZeppelinOverviewApache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets.Supported interpreter type%pig.script (default Pig interpreter, so you can use %pig)%pig.script is like the Pig grunt shell. Anything you can run in Pig grunt shell can be run in %pig.script interpreter, it is used for running Pig script where you don’t need to visualize the data, it is suitable for data munging. %pig.query%pig.query is a little different compared with %pig.script. It is used for exploratory data analysis via Pig latin where you can leverage Zeppelin’s visualization ability. There're 2 minor differences in the last statement between %pig
 .script and %pig.queryNo pig alias in the last statement in %pig.query (read the examples below).The last statement must be in single line in %pig.queryHow to useHow to setup Pig execution modes.Local ModeSet zeppelin.pig.execType as local.MapReduce ModeSet zeppelin.pig.execType as mapreduce. HADOOP_CONF_DIR needs to be specified in ZEPPELIN_HOME/conf/zeppelin-env.sh.Tez Local ModeOnly Tez 0.7 is supported. Set zeppelin.pig.execType as tez_local.Tez ModeOnly Tez 0.7 is supported. Set zeppelin.pig.execType as tez. HADOOP_CONF_DIR and TEZ_CONF_DIR needs to be specified in ZEPPELIN_HOME/conf/zeppelin-env.sh.Spark Local ModeOnly Spark 1.6.x is supported, by default it is Spark 1.6.3. Set zeppelin.pig.execType as spark_local.Spark ModeOnly Spark 1.6.x is supported, by default it is Spark 1.6.3. Set zeppelin.pig.execType as spark. For now, only yarn-client mode is supported. To enable it, you need to set property SPARK_MASTER to yarn-client and set SPARK_JAR to the spark assembly jar.How 
 to choose custom Spark VersionBy default, Pig Interpreter would use Spark 1.6.3 built with scala 2.10, if you want to use another spark version or scala version, you need to rebuild Zeppelin by specifying the custom Spark version via -Dpig.spark.version= and scala version via -Dpig.scala.version= in the maven build command.How to configure interpreterAt the Interpreters menu, you have to create a new Pig interpreter. Pig interpreter has below properties by default.And you can set any Pig properties here which will be passed to Pig engine. (like tez.queue.name & mapred.job.queue.name).Besides, we use paragraph title as job name if it exists, else use the last line of Pig script. So you can use that to find app running in YARN RM UI.            Property        Default        Description                zeppelin.pig.execType        mapreduce        Execution mode for pig runtime. local | mapreduce | tez_local | tez | spark_local | spark                 zeppelin.pig.includeJobSta
 ts        false        whether display jobStats info in %pig.script                zeppelin.pig.maxResult        1000        max row number displayed in %pig.query                tez.queue.name        default        queue name for tez engine                mapred.job.queue.name        default        queue name for mapreduce engine                SPARK_MASTER        local        local | yarn-client                SPARK_JAR                The spark assembly jar, both jar in local or hdfs is supported. Put it on hdfs could have        performance benefit      Examplepig%pigbankText = load 'bank.csv' using PigStorage(';');bank = foreach bankText generate $0 as age, $1 as job, $2 as marital, $3 as education, $5 as balance; bank = filter bank by age != '"age"';bank = foreach bank generate (int)age, REPLACE(job,'"','') as job, REPLACE(marital, '"', '&a
 mp;#39;) as marital, (int)(REPLACE(balance, '"', '')) as balance;store bank into 'clean_bank.csv' using PigStorage(';'); -- this statement is optional, it just show you that most of time %pig.script is used for data munging before querying the data. pig.queryGet the number of each age where age is less than 30%pig.querybank_data = filter bank by age < 30;b = group bank_data by age;foreach b generate group, COUNT($1);The same as above, but use dynamic text form so that use can specify the variable maxAge in textbox. (See screenshot below). Dynamic form is a very cool feature of Zeppelin, you can refer this link) for details.%pig.querybank_data = filter bank by age < ${maxAge=40};b = group bank_data by age;foreach b generate group, COUNT($1) as count;Get the number of each age for specific marital type, also use dynamic form here. User can choose the marital type in the dropdown list (see screenshot
  below).%pig.querybank_data = filter bank by marital=='${marital=single,single|divorced|married}';b = group bank_data by age;foreach b generate group, COUNT($1) as count;The above examples are in the Pig tutorial note in Zeppelin, you can check that for details. Here's the screenshot.Data is shared between %pig and %pig.query, so that you can do some common work in %pig, and do different kinds of query based on the data of %pig. Besides, we recommend you to specify alias explicitly so that the visualization can display the column name correctly. In the above example 2 and 3 of %pig.query, we name COUNT($1) as count. If you don't do this, then we will name it using position. E.g. in the above first example of %pig.query, we will use col_1 in chart to represent COUNT($1).",
+      "url": " /interpreter/pig.html",
+      "group": "manual",
+      "excerpt": "Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs."
     }
     ,
     
   
 
-    "/development/helium/writing_application.html": {
-      "title": "Writing a new Helium Application",
-      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Writing a new ApplicationWhat is Apache Zeppelin ApplicationApache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. While application runs on Interpreter process, it's able to access resources provided by Interpreter through ResourcePool. Output is always rendered by AngularDisplaySystem. Therefore application provides all the possiblities of making 
 interactive graphical application that uses data and processing power of any Interpreter.Make your own ApplicationWriting Application means extending org.apache.zeppelin.helium.Application. You can use your favorite IDE and language while Java class files are packaged into jar. Application class looks like/** * Constructor. Invoked when application is loaded */public Application(ApplicationContext context);/** * Invoked when there're (possible) updates in required resource set. * i.e. invoked after application load and after paragraph finishes. */public abstract void run(ResourceSet args);/** * Invoked before application unload. * Application is automatically unloaded with paragraph/notebook removal */public abstract void unload();You can check example applications under ./zeppelin-examples directory.Development modeIn the development mode, you can run your Application in your IDE as a normal java application and see the result inside of Zeppelin notebook.org.apache.zeppelin
 .helium.ZeppelinApplicationDevServer can run Zeppelin Application in development mode.// entry point for development modepublic static void main(String[] args) throws Exception {  // add resources for development mode  LocalResourcePool pool = new LocalResourcePool("dev");  pool.put("date", new Date());  // run application in devlopment mode with given resource  // in this case, Clock.class.getName() will be the application class name    org.apache.zeppelin.helium.ZeppelinApplicationDevServer devServer = new org.apache.zeppelin.helium.ZeppelinApplicationDevServer(    Clock.class.getName(), pool.getAll());  // start development mode  devServer.start();  devServer.join();}In the Zeppelin notebook, run %dev run will connect to application running in development mode.Package filePackage file is a json file that provides information about the application.Json file contains the following information{  "name" : "[organization].
 [name]",  "description" : "Description",  "artifact" : "groupId:artifactId:version",  "className" : "your.package.name.YourApplicationClass",  "resources" : [    ["resource.name", ":resource.class.name"],    ["alternative.resource.name", ":alternative.class.name"]  ],  "icon" : "<i class='icon'></i>"}nameName is a string in [group].[name] format.[group] and [name] allow only [A-Za-z0-9_].Group is normally the name of an organization who creates this application.descriptionA short description about the applicationartifactLocation of the jar artifact."groupId:artifactId:version" will load artifact from maven repository.If jar exists in the local filesystem, absolute/relative can be use
 d.e.g.When artifact exists in Maven repositoryartifact: "org.apache.zeppelin:zeppelin-examples:0.6.0"When artifact exists in the local filesystemartifact: "zeppelin-example/target/zeppelin-example-0.6.0.jar"classNameEntry point. Class that extends org.apache.zeppelin.helium.ApplicationresourcesTwo dimensional array that defines required resources by name or by className. Helium Application launcher will compare resources in the ResourcePool with the information in this field and suggest application only when all required resources are available in the ResourcePool.Resouce name is a string which will be compared with the name of objects in the ResourcePool. className is a string with ":" prepended, which will be compared with className of the objects in the ResourcePool.Application may require two or more resources. Required resources can be listed inside of the json array. For example, if the application requires object &quot
 ;name1", "name2" and "className1" type of object to run, resources field can beresources: [  [ "name1", "name2", ":className1", ...]]If Application can handle alternative combination of required resources, alternative set can be listed as below.resources: [  [ "name", ":className"],  [ "altName", ":altClassName1"],  ...]Easier way to understand this scheme isresources: [   [ 'resource' AND 'resource' AND ... ] OR   [ 'resource' AND 'resource' AND ... ] OR   ...]iconIcon to be used on the application button. String in this field will be rendered as a HTML tag.e.g.icon: "<i class='fa fa-clock-o'></i>"",
-      "url": " /development/helium/writing_application.html",
-      "group": "development/helium",
-      "excerpt": "Apache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. Make your own Application in Apache Zeppelin is quite easy."
+    "/interpreter/markdown.html": {
+      "title": "Markdown Interpreter for Apache Zeppelin",
+      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Markdown Interpreter for Apache ZeppelinOverviewMarkdown is a plain text formatting syntax designed so that it can be converted to HTML.Apache Zeppelin uses pegdown and markdown4j as markdown parsers.In Zeppelin notebook, you can use %md in the beginning of a paragraph to invoke the Markdown interpreter and generate static html from Markdown plain text.In Zeppelin, Markdown interpreter is enabled by default and uses the pegdown par
 ser.ExampleThe following example demonstrates the basic usage of Markdown in a Zeppelin notebook.Mathematical expressionMarkdown interpreter leverages %html display system internally. That means you can mix mathematical expressions with markdown syntax. For more information, please see Mathematical Expression section.Configuration      Name    Default Value    Description        markdown.parser.type    pegdown    Markdown Parser Type.  Available values: pegdown, markdown4j.  Pegdown Parserpegdown parser provides github flavored markdown.pegdown parser provides YUML and Websequence plugins also. Markdown4j ParserSince pegdown parser is more accurate and provides much more markdown syntaxmarkdown4j option might be removed later. But keep this parser for the backward compatibility.",
+      "url": " /interpreter/markdown.html",
+      "group": "interpreter",
+      "excerpt": "Markdown is a plain text formatting syntax designed so that it can be converted to HTML. Apache Zeppelin uses markdown4j."
     }
     ,
     
   
 
-    "/development/helium/writing_spell.html": {
-      "title": "Writing a new Helium Spell",
-      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Writing a new SpellWhat is Apache Zeppelin SpellSpell is a kind of interpreter that runs on browser not on backend. So, technically it's the frontend interpreter.It can provide many benefits.Spell is pluggable frontend interpreter. So it can be installed and removed easily using helium registry.Every spell is written in javascript. It means you can use existing javascript libraries whatever you want.Spell runs on browser li
 ke display system (%html, %table). In other words, every spell can be used as display system as well.How it worksHelium Spell works like Helium Visualization.Every helium packages are loaded from central (online) registry or local registryYou can see loaded packages in /helium page.When you enable a spell, it's built from server and sent to clientFinally it will be loaded into browser.How to use spell1. EnablingFind a spell what you want to use in /helium package and click Enable button.2. UsingSpell works like an interpreter. Use the MAGIC value to execute spell in a note. (you might need to refresh after enabling)For example, Use %echo for the Echo Spell.Write a new SpellMaking a new spell is similar to Helium Visualization#write-new-visualization.Add framework dependency called zeppelin-spell into package.jsonWrite code using frameworkPublish your spell to npm1. Create a npm packageCreate a package.json in new directory for spell.You have to add a framework called zeppeli
 n-spell as a dependency to create spell (zeppelin-spell)Also, you can add any dependencies you want to utilise.Here's an example{  "name": "zeppelin-echo-spell",  "description": "Zeppelin Echo Spell (example)",  "version": "1.0.0",  "main": "index",  "author": "",  "license": "Apache-2.0",  "dependencies": {    "zeppelin-spell": "*"  },  "helium": {    "icon" : "<i class='fa fa-repeat'></i>",    "spell": {      "magic": "%echo",      "usage": "%echo <TEXT>"    }  }}2. Write spell using frameworkHere are some exa
 mples you can referEcho SpellMarkdown Spell: Using libraryFlowchart Spell: Using DOMGoogle Translation API Spell: Using API (returning promise)Now, you need to write code to create spell which processing text.import {    SpellBase,    SpellResult,    DefaultDisplayType,} from 'zeppelin-spell';export default class EchoSpell extends SpellBase {    constructor() {        /** pass magic to super class's constructor parameter */        super("%echo");    }    interpret(paragraphText) {        const processed = paragraphText + '!';        /**         * should return `SpellResult` which including `data` and `type`         * default type is `TEXT` if you don't specify.           */        return new SpellResult(processed);    }}Here is another example. Let's say we want to create markdown spell. First of all, we should add a dependency for markdown in package.json// package.json "dependencies": {    
 "markdown": "0.5.0",    "zeppelin-spell": "*"  },And here is spell code.import {    SpellBase,    SpellResult,    DefaultDisplayType,} from 'zeppelin-spell';import md from 'markdown';const markdown = md.markdown;export default class MarkdownSpell extends SpellBase {    constructor() {        super("%markdown");    }    interpret(paragraphText) {        const parsed = markdown.toHTML(paragraphText);        /**         * specify `DefaultDisplayType.HTML` since `parsed` will contain DOM         * otherwise it will be rendered as `DefaultDisplayType.TEXT` (default)         */        return new SpellResult(parsed, DefaultDisplayType.HTML);    }}You might want to manipulate DOM directly (e.g google d3.js), then refer Flowchart SpellYou might want to return promise not string (e.g API call), then refer Google Translation API Spell3. Create Helium package file for local 
 deploymentYou don't want to publish your package every time you make a change in your spell. Zeppelin provides local deploy.The only thing you need to do is creating a Helium Package file (JSON) for local deploy.It's automatically created when you publish to npm repository but in local case, you should make it by yourself.{  "type" : "SPELL",  "name" : "zeppelin-echo-spell",  "version": "1.0.0",  "description" : "Return just what receive (example)",  "artifact" : "./zeppelin-examples/zeppelin-example-spell-echo",  "license" : "Apache-2.0",  "spell": {    "magic": "%echo",    "usage": "%echo <TEXT>"  }}Place this file in your local registry directory (de
 fault $ZEPPELIN_HOME/helium).type should be SPELLMake sure that artifact should be same as your spell directory.You can get information about other fields in Helium Visualization#3-create-helium-package-file-and-locally-deploy.4. Run in dev modecd zeppelin-webyarn run dev:heliumYou can browse localhost:9000. Every time refresh your browser, Zeppelin will rebuild your spell and reload changes.5. Publish your spell to the npm repositorySee Publishing npm packages",
-      "url": " /development/helium/writing_spell.html",
-      "group": "development/helium",
-      "excerpt": "Spell is a kind of interpreter that runs on browser not on backend. So, technically it's the frontend interpreter. "
+    "/interpreter/mahout.html": {
+      "title": "Mahout Interpreter for Apache Zeppelin",
+      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Mahout Interpreter for Apache ZeppelinInstallationApache Mahout is a collection of packages that enable machine learning and matrix algebra on underlying engines such as Apache Flink or Apache Spark.  A convenience script for creating and configuring two Mahout enabled interpreters exists.  The %sparkMahout and %flinkMahout interpreters do not exist by default but can be easily created using this script.  Easy InstallationTo
  quickly and easily get up and running using Apache Mahout, run the following command from the top-level directory of the Zeppelin install:bashpython scripts/mahout/add_mahout.pyThis will create the %sparkMahout and %flinkMahout interpreters, and restart Zeppelin.Advanced InstallationThe add_mahout.py script contains several command line arguments for advanced users.      Argument    Description    Example        --zeppelin_home    This is the path to the Zeppelin installation.  This flag is not needed if the script is run from the top-level installation directory or from the `zeppelin/scripts/mahout` directory.    /path/to/zeppelin        --mahout_home    If the user has already installed Mahout, this flag can set the path to `MAHOUT_HOME`.  If this is set, downloading Mahout will be skipped.    /path/to/mahout_home        --restart_later    Restarting is necessary for updates to take effect. By default the script will restart Zeppelin for you- restart will be skipped if this flag 
 is set.    NA        --force_download    This flag will force the script to re-download the binary even if it already exists.  This is useful for previously failed downloads.    NA          --overwrite_existing      This flag will force the script to overwrite existing `%sparkMahout` and `%flinkMahout` interpreters. Useful when you want to just start over.      NA    NOTE 1: Apache Mahout at this time only supports Spark 1.5 and Spark 1.6 and Scala 2.10.  If the user is using another version of Spark (e.g. 2.0), the %sparkMahout will likely not work.  The %flinkMahout interpreter will still work and the user is encouraged to develop with that engine as the code can be ported via copy and paste, as is evidenced by the tutorial notebook.NOTE 2: If using Apache Flink in cluster mode, the following libraries will also need to be coppied to ${FLINK_HOME}/lib- mahout-math-0.12.2.jar- mahout-math-scala2.10-0.12.2.jar- mahout-flink2.10-0.12.2.jar- mahout-hdfs-0.12.2.jar- com.google.guava:gu
 ava:14.0.1OverviewThe Apache Mahout™ project's goal is to build an environment for quickly creating scalable performant machine learning applications.Apache Mahout software provides three major features:A simple and extensible programming environment and framework for building scalable algorithmsA wide variety of premade algorithms for Scala + Apache Spark, H2O, Apache FlinkSamsara, a vector math experimentation environment with R-like syntax which works at scaleIn other words:Apache Mahout provides a unified API for quickly creating machine learning algorithms on a variety of engines.How to useWhen starting a session with Apache Mahout, depending on which engine you are using (Spark or Flink), a few imports must be made and a Distributed Context must be declared.  Copy and paste the following code and run once to get started.Flink%flinkMahoutimport org.apache.flink.api.scala._import org.apache.mahout.math.drm._import org.apache.mahout.math.drm.RLikeDrmOps._import org.a
 pache.mahout.flinkbindings._import org.apache.mahout.math._import scalabindings._import RLikeOps._implicit val ctx = new FlinkDistributedContext(benv)Spark%sparkMahoutimport org.apache.mahout.math._import org.apache.mahout.math.scalabindings._import org.apache.mahout.math.drm._import org.apache.mahout.math.scalabindings.RLikeOps._import org.apache.mahout.math.drm.RLikeDrmOps._import org.apache.mahout.sparkbindings._implicit val sdc: org.apache.mahout.sparkbindings.SparkDistributedContext = sc2sdc(sc)Same Code, Different EnginesAfter importing and setting up the distributed context, the Mahout R-Like DSL is consistent across engines.  The following code will run in both %flinkMahout and %sparkMahoutval drmData = drmParallelize(dense(  (2, 2, 10.5, 10, 29.509541),  // Apple Cinnamon Cheerios  (1, 2, 12,   12, 18.042851),  // Cap'n'Crunch  (1, 1, 12,   13, 22.736446),  // Cocoa Puffs  (2, 1, 11,   13, 32.207582),  // Froot Loops  (1, 2, 12,   11, 21.871292),  // Honey G
 raham Ohs  (2, 1, 16,   8,  36.187559),  // Wheaties Honey Gold  (6, 2, 17,   1,  50.764999),  // Cheerios  (3, 2, 13,   7,  40.400208),  // Clusters  (3, 3, 13,   4,  45.811716)), numPartitions = 2)drmData.collect(::, 0 until 4)val drmX = drmData(::, 0 until 4)val y = drmData.collect(::, 4)val drmXtX = drmX.t %*% drmXval drmXty = drmX.t %*% yval XtX = drmXtX.collectval Xty = drmXty.collect(::, 0)val beta = solve(XtX, Xty)Leveraging Resource Pools and R for VisualizationResource Pools are a powerful Zeppelin feature that lets us share information between interpreters. A fun trick is to take the output of our work in Mahout and analyze it in other languages.Setting up a Resource Pool in FlinkIn Spark based interpreters resource pools are accessed via the ZeppelinContext API.  To put and get things from the resource pool one can be done simplescalaval myVal = 1z.put("foo", myVal)val myFetchedVal = z.get("foo")To add this functionality to a Flink bas
 ed interpreter we declare the follwoing%flinkMahoutimport org.apache.zeppelin.interpreter.InterpreterContextval z = InterpreterContext.get().getResourcePool()Now we can access the resource pool in a consistent manner from the %flinkMahout interpreter.Passing a variable from Mahout to R and PlottingIn this simple example, we use Mahout (on Flink or Spark, the code is the same) to create a random matrix and then take the Sin of each element. We then randomly sample the matrix and create a tab separated string. Finally we pass that string to R where it is read as a .tsv file, and a DataFrame is created and plotted using native R plotting libraries.val mxRnd = Matrices.symmetricUniformView(5000, 2, 1234)val drmRand = drmParallelize(mxRnd)val drmSin = drmRand.mapBlock() {case (keys, block) =>    val blockB = block.like()  for (i <- 0 until block.nrow) {    blockB(i, 0) = block(i, 0)    blockB(i, 1) = Math.sin((block(i, 0) * 8))  }  keys -> blockB}z.put("sinD
 rm", org.apache.mahout.math.drm.drmSampleToTSV(drmSin, 0.85))And then in an R paragraph...%spark.r {"imageWidth": "400px"}library("ggplot2")sinStr = z.get("flinkSinDrm")data <- read.table(text= sinStr, sep="t", header=FALSE)plot(data,  col="red")",
+      "url": " /interpreter/mahout.html",
+      "group": "interpreter",
+      "excerpt": "Apache Mahout provides a unified API (the R-Like Scala DSL) for quickly creating machine learning algorithms on a variety of engines."
     }
     ,
     
   
 
-    "/development/helium/writing_visualization_basic.html": {
-      "title": "Writing a new Helium Visualization: basic",
-      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Writing a new VisualizationWhat is Apache Zeppelin VisualizationApache Zeppelin Visualization is a pluggable package that can be loaded/unloaded on runtime through Helium framework in Zeppelin. A Visualization is a javascript npm package and user can use them just like any other built-in visualization in notebook.How it works1. Load Helium package files from registryZeppelin needs to know what Visualization packages are available. 
 Zeppelin will read information of packages from both online and local registry.Registries are configurable through ZEPPELIN_HELIUM_LOCALREGISTRY_DEFAULT env variable or zeppelin.helium.localregistry.default property.2. Enable packagesOnce Zeppelin loads Helium package files from registries, available packages are displayed in Helium menu.Click 'enable' button.3. Create and load visualization bundle on the flyOnce a Visualization package is enabled, HeliumBundleFactory creates a js bundle. The js bundle is served by helium/bundle/load rest api endpoint.4. Run visualizationZeppelin shows additional button for loaded Visualizations.User can use just like any other built-in visualizations.Write new Visualization1. Create a npm packageCreate a package.json in your new Visualization directory. You can add any dependencies in package.json, but you must include two dependencies: zeppelin-vis and zeppelin-tabledata.Here's an example{  "name": &qu
 ot;zeppelin_horizontalbar",  "description" : "Horizontal Bar chart",  "version": "1.0.0",  "main": "horizontalbar",  "author": "",  "license": "Apache-2.0",  "dependencies": {    "zeppelin-tabledata": "*",    "zeppelin-vis": "*"  }}2. Create your own visualizationTo create your own visualization, you need to create a js file and import Visualization class from zeppelin-vis package and extend the class. zeppelin-tabledata package provides some useful transformations, like pivot, you can use in your visualization. (you can create your own transformation, too).Visualization class, there're several methods that you need to override and implement. Here's simple visualization that just prints Hello 
 world.import Visualization from 'zeppelin-vis'import PassthroughTransformation from 'zeppelin-tabledata/passthrough'export default class helloworld extends Visualization {  constructor(targetEl, config) {    super(targetEl, config)    this.passthrough = new PassthroughTransformation(config);  }  render(tableData) {    this.targetEl.html('Hello world!')  }  getTransformation() {    return this.passthrough  }}To learn more about Visualization class, check visualization.js.You can check complete visualization package example here.Zeppelin's built-in visualization uses the same API, so you can check built-in visualizations as additional examples.3. Create Helium package file and locally deployHelium Package file is a json file that provides information about the application.Json file contains the following information{  "type" : "VISUALIZATION",  "name" : "zeppelin_hori
 zontalbar",  "description" : "Horizontal Bar chart (example)",  "license" : "Apache-2.0",  "artifact" : "./zeppelin-examples/zeppelin-example-horizontalbar",  "icon" : "<i class='fa fa-bar-chart rotate90flipX'></i>"}Place this file in your local registry directory (default ./helium).typeWhen you're creating a visualization, 'type' should be 'VISUALIZATION'. Check these types as well.Helium ApplicationHelium SpellnameName of visualization. Should be unique. Allows [A-Za-z90-9_].descriptionA short description about visualization.artifactLocation of the visualization npm package. Support npm package with version or local filesystem path.e.g.When artifact exists in npm repository"artifact": "my-visualiztion@1.0.0"When 
 artifact exists in local file system"artifact": "/path/to/my/visualization"licenseLicense information.e.g."license": "Apache-2.0"iconIcon to be used in visualization select button. String in this field will be rendered as a HTML tag.e.g."icon": "<i class='fa fa-coffee'></i>"4. Run in dev modePlace your Helium package file in local registry (ZEPPELIN_HOME/helium).Run Zeppelin. And then run zeppelin-web in visualization dev mode.cd zeppelin-webyarn run dev:heliumYou can browse localhost:9000. Everytime refresh your browser, Zeppelin will rebuild your visualization and reload changes.5. Publish your visualizationOnce it's done, publish your visualization package using npm publish.That's it. With in an hour, your visualization will be available in Zeppelin's helium menu.See MoreCheck Helium Visualization: Transfor
 mation for more complex examples.",
-      "url": " /development/helium/writing_visualization_basic.html",
-      "group": "development/helium",
-      "excerpt": "Apache Zeppelin Visualization is a pluggable package that can be loaded/unloaded on runtime through Helium framework in Zeppelin. A Visualization is a javascript npm package and user can use them just like any other built-in visualization in a note."
+    "/interpreter/spark.html": {
+      "title": "Apache Spark Interpreter for Apache Zeppelin",
+      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Spark Interpreter for Apache ZeppelinOverviewApache Spark is a fast and general-purpose cluster computing system.It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs.Apache Spark is supported in Zeppelin with Spark interpreter group which consists of below five interpreters.      Name    Class    Description        %spark    SparkInterpreter    Creates a SparkConte
 xt and provides a Scala environment        %spark.pyspark    PySparkInterpreter    Provides a Python environment        %spark.r    SparkRInterpreter    Provides an R environment with SparkR support        %spark.sql    SparkSQLInterpreter    Provides a SQL environment        %spark.dep    DepInterpreter    Dependency loader  ConfigurationThe Spark interpreter can be configured with properties provided by Zeppelin.You can also set other Spark properties which are not listed in the table. For a list of additional properties, refer to Spark Available Properties.      Property    Default    Description        args        Spark commandline args      master    local[*]    Spark master uri.  ex) spark://masterhost:7077      spark.app.name    Zeppelin    The name of spark application.        spark.cores.max        Total number of cores to use.  Empty value uses all available core.        spark.executor.memory     1g    Executor memory per worker instance.  ex) 512m, 32g        zeppelin.dep
 .additionalRemoteRepository    spark-packages,  http://dl.bintray.com/spark-packages/maven,  false;    A list of id,remote-repository-URL,is-snapshot;  for each remote repository.        zeppelin.dep.localrepo    local-repo    Local repository for dependency loader        PYSPARKPYTHON    python    Python binary executable to use for PySpark in both driver and workers (default is python).            Property spark.pyspark.python take precedence if it is set        PYSPARKDRIVERPYTHON    python    Python binary executable to use for PySpark in driver only (default is PYSPARKPYTHON).            Property spark.pyspark.driver.python take precedence if it is set        zeppelin.spark.concurrentSQL    false    Execute multiple SQL concurrently if set true.        zeppelin.spark.maxResult    1000    Max number of Spark SQL result to display.        zeppelin.spark.printREPLOutput    true    Print REPL output        zeppelin.spark.useHiveContext    true    Use HiveContext instead of SQLConte
 xt if it is true.        zeppelin.spark.importImplicit    true    Import implicits, UDF collection, and sql if set true.        zeppelin.spark.enableSupportedVersionCheck    true    Do not change - developer only setting, not for production use      zeppelin.spark.uiWebUrl        Overrides Spark UI default URL. Value should be a full URL (ex: http://{hostName}/{uniquePath}  Without any configuration, Spark interpreter works out of box in local mode. But if you want to connect to your Spark cluster, you'll need to follow below two simple steps.1. Export SPARK_HOMEIn conf/zeppelin-env.sh, export SPARK_HOME environment variable with your Spark installation path.For example,export SPARK_HOME=/usr/lib/sparkYou can optionally set more environment variables# set hadoop conf direxport HADOOP_CONF_DIR=/usr/lib/hadoop# set options to pass spark-submit commandexport SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0"# extra classpath. e.g. set classp
 ath for hive-site.xmlexport ZEPPELIN_INTP_CLASSPATH_OVERRIDES=/etc/hive/confFor Windows, ensure you have winutils.exe in %HADOOP_HOME%bin. Please see Problems running Hadoop on Windows for the details.2. Set master in Interpreter menuAfter start Zeppelin, go to Interpreter menu and edit master property in your Spark interpreter setting. The value may vary depending on your Spark cluster deployment type.For example,local[*] in local modespark://master:7077 in standalone clusteryarn-client in Yarn client modeyarn-cluster in Yarn cluster modemesos://host:5050 in Mesos clusterThat's it. Zeppelin will work with any version of Spark and any deployment type without rebuilding Zeppelin in this way.For the further information about Spark & Zeppelin version compatibility, please refer to "Available Interpreters" section in Zeppelin download page.Note that without exporting SPARK_HOME, it's running in local mode with included version of Spark. The incl
 uded version may vary depending on the build profile.3. Yarn modeZeppelin support both yarn client and yarn cluster mode (yarn cluster mode is supported from 0.8.0). For yarn mode, you must specify SPARK_HOME & HADOOP_CONF_DIR.You can either specify them in zeppelin-env.sh, or in interpreter setting page. Specifying them in zeppelin-env.sh means you can use only one version of spark & hadoop. Specifying themin interpreter setting page means you can use multiple versions of spark & hadoop in one zeppelin instance.SparkContext, SQLContext, SparkSession, ZeppelinContextSparkContext, SQLContext and ZeppelinContext are automatically created and exposed as variable names sc, sqlContext and z, respectively, in Scala, Python and R environments.Staring from 0.6.1 SparkSession is available as variable spark when you are using Spark 2.x.Note that Scala/Python/R environment shares the same SparkContext, SQLContext and ZeppelinContext instance. How to pass property to Spa
 rkConfThere're 2 kinds of properties that would be passed to SparkConfStandard spark property (prefix with spark.). e.g. spark.executor.memory will be passed to SparkConfNon-standard spark property (prefix with zeppelin.spark.).  e.g. zeppelin.spark.property_1, property_1 will be passed to SparkConfDependency ManagementThere are two ways to load external libraries in Spark interpreter. First is using interpreter setting menu and second is loading Spark properties.1. Setting Dependencies via Interpreter SettingPlease see Dependency Management for the details.2. Loading Spark PropertiesOnce SPARK_HOME is set in conf/zeppelin-env.sh, Zeppelin uses spark-submit as spark interpreter runner. spark-submit supports two ways to load configurations.The first is command line options such as --master and Zeppelin can pass these options to spark-submit by exporting SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.sh. Second is reading configuration options from SPARK_HOME/conf/spark-defaults.co
 nf. Spark properties that user can set to distribute libraries are:      spark-defaults.conf    SPARK_SUBMIT_OPTIONS    Description        spark.jars    --jars    Comma-separated list of local jars to include on the driver and executor classpaths.        spark.jars.packages    --packages    Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version.        spark.files    --files    Comma-separated list of files to be placed in the working directory of each executor.  Here are few examples:SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.shexport SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0 --jars /path/mylib1.jar,/path/mylib2.jar --files /path/mylib1.py,/path/mylib2.zip,/path/mylib3.egg"SPARK_HOME/conf/spark-defaults.confspark
 .jars        /path/mylib1.jar,/path/mylib2.jarspark.jars.packages   com.databricks:spark-csv_2.10:1.2.0spark.files       /path/mylib1.py,/path/mylib2.egg,/path/mylib3.zip3. Dynamic Dependency Loading via %spark.dep interpreterNote: %spark.dep interpreter loads libraries to %spark and %spark.pyspark but not to  %spark.sql interpreter. So we recommend you to use the first option instead.When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs using %spark.dep interpreter.Load libraries recursively from maven repositoryLoad libraries from local filesystemAdd additional maven repositoryAutomatically add libraries to SparkCluster (You can turn off)Dep interpreter leverages Scala environment. So you can write any Scala code here.Note that %spark.dep interpreter should be used before %spark, %spark.pyspark, %spark.sql.Here's usages.%spark.depz.reset() // clean up previously added artifact and repository// add maven 
 repositoryz.addRepo("RepoName").url("RepoURL")// add maven snapshot repositoryz.addRepo("RepoName").url("RepoURL").snapshot()// add credentials for private maven repositoryz.addRepo("RepoName").url("RepoURL").username("username").password("password")// add artifact from filesystemz.load("/path/to.jar")// add artifact from maven repository, with no dependencyz.load("groupId:artifactId:version").excludeAll()// add artifact recursivelyz.load("groupId:artifactId:version")// add artifact recursively except comma separated GroupID:ArtifactId listz.load("groupId:artifactId:version").exclude("groupId:artifactId,groupId:artifactId, ...")// exclude with patternz.load("groupId:artifactId:version").exclude(*)z.load("groupId:artifactId:ver
 sion").exclude("groupId:artifactId:*")z.load("groupId:artifactId:version").exclude("groupId:*")// local() skips adding artifact to spark clusters (skipping sc.addJar())z.load("groupId:artifactId:version").local()ZeppelinContextZeppelin automatically injects ZeppelinContext as variable z in your Scala/Python environment. ZeppelinContext provides some additional functions and utilities.Exploring Spark DataFramesZeppelinContext provides a show method, which, using Zeppelin's table feature, can be used to nicely display a Spark DataFrame:df = spark.read.csv('/path/to/csv')z.show(df)Object ExchangeZeppelinContext extends map and it's shared between Scala and Python environment.So you can put some objects from Scala and read it from Python, vice versa.  // Put object from scala%sparkval myObject = ...z.put("objName", myObject)// Exchanging data framesmyScalaDa
 taFrame = ...z.put("myScalaDataFrame", myScalaDataFrame)val myPythonDataFrame = z.get("myPythonDataFrame").asInstanceOf[DataFrame]    # Get object from python%spark.pysparkmyObject = z.get("objName")# Exchanging data framesmyPythonDataFrame = ...z.put("myPythonDataFrame", postsDf._jdf)myScalaDataFrame = DataFrame(z.get("myScalaDataFrame"), sqlContext)  Form CreationZeppelinContext provides functions for creating forms.In Scala and Python environments, you can create forms programmatically.  %spark/* Create text input form */z.input("formName")/* Create text input form with default value */z.input("formName", "defaultValue")/* Create select form */z.select("formName", Seq(("option1", "option1DisplayName"),                         ("option2", "option2DisplayName&a
 mp;quot;)))/* Create select form with default value*/z.select("formName", "option1", Seq(("option1", "option1DisplayName"),                                    ("option2", "option2DisplayName")))    %spark.pyspark# Create text input formz.input("formName")# Create text input form with default valuez.input("formName", "defaultValue")# Create select formz.select("formName", [("option1", "option1DisplayName"),                      ("option2", "option2DisplayName")])# Create select form with default valuez.select("formName", [("option1", "option1DisplayName"),                      ("option2", "option2DisplayName")], "option1")  In sql
  environment, you can create form in simple template.%spark.sqlselect * from ${table=defaultTableName} where text like '%${search}%'To learn more about dynamic form, checkout Dynamic Form.Matplotlib Integration (pyspark)Both the python and pyspark interpreters have built-in support for inline visualization using matplotlib,a popular plotting library for python. More details can be found in the python interpreter documentation,since matplotlib support is identical. More advanced interactive plotting can be done with pyspark throughutilizing Zeppelin's built-in Angular Display System, as shown below:Interpreter setting optionYou can choose one of shared, scoped and isolated options wheh you configure Spark interpreter.Spark interpreter creates separated Scala compiler per each notebook but share a single SparkContext in scoped mode (experimental).It creates separated SparkContext per each notebook in isolated mode.IPython supportBy default, zeppelin would use I
 Python in pyspark when IPython is available, Otherwise it would fall back to the original PySpark implementation.If you don't want to use IPython, then you can set zeppelin.pyspark.useIPython as false in interpreter setting. For the IPython features, you can refer docPython InterpreterSetting up Zeppelin with KerberosLogical setup with Zeppelin, Kerberos Key Distribution Center (KDC), and Spark on YARN:Configuration SetupOn the server that Zeppelin is installed, install Kerberos client modules and configuration, krb5.conf.This is to make the server communicate with KDC.Set SPARK_HOME in [ZEPPELIN_HOME]/conf/zeppelin-env.sh to use spark-submit(Additionally, you might have to set export HADOOP_CONF_DIR=/etc/hadoop/conf)Add the two properties below to Spark configuration ([SPARK_HOME]/conf/spark-defaults.conf):spark.yarn.principalspark.yarn.keytabNOTE: If you do not have permission to access for the above spark-defaults.conf file, optionally, you can add the above lines to the 
 Spark Interpreter setting through the Interpreter tab in the Zeppelin UI.That's it. Play with Zeppelin!",
+      "url": " /interpreter/spark.html",
+      "group": "interpreter",
+      "excerpt": "Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution engine."
     }
     ,
     
   
 
-    "/development/helium/writing_visualization_transformation.html": {
-      "title": "Transformations in Zeppelin Visualization",
-      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Transformations for Zeppelin VisualizationOverviewTransformations renders setting which allows users to set columns and transforms table rows according to the configured columns.Zeppelin provides 4 types of transformations.1. PassthroughTransformationPassthroughTransformation is the simple transformation which does not convert original tabledata at all.See passthrough.js2. ColumnselectorTransformationColumnselectorTransformation is
  uses when you need N axes but do not need aggregation. See columnselector.js3. PivotTransformationPivotTransformation provides group by and aggregation. Every chart using PivotTransformation has 3 axes. Keys, Groups and Values.See pivot.js4. AdvancedTransformationAdvancedTransformation has more detailed options while providing existing features of PivotTransformation and ColumnselectorTransformationmultiple sub chartsconfigurable chart axesparameter widgets: input, checkbox, option, textareaparsing parameters automatically based on their typesexpand / fold axis and parameter panelsmultiple transformation methods while supporting lazy converting re-initialize the whole configuration based on spec hash.SpecAdvancedTransformation requires spec which includes axis and parameter details for charts.Let's create 2 sub-charts called line and no-group. Each sub chart can have different axis and parameter depending on their requirements.class AwesomeVisualization extends Visualizatio
 n {  constructor(targetEl, config) {    super(targetEl, config)    const spec = {      charts: {        'line': {          transform: { method: 'object', },          sharedAxis: false, /** set if you want to share axes between sub charts, default is `false` */          axis: {            'xAxis': { dimension: 'multiple', axisType: 'key', description: 'serial', },            'yAxis': { dimension: 'multiple', axisType: 'aggregator', description: 'serial', },            'category': { dimension: 'multiple', axisType: 'group', description: 'categorical', },          },          parameter: {            'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', },            &#3
 9;yAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of yAxis', },            'lineWidth': { valueType: 'int', defaultValue: 0, description: 'width of line', },          },        },        'no-group': {          transform: { method: 'object', },          sharedAxis: false,          axis: {            'xAxis': { dimension: 'single', axisType: 'key', },            'yAxis': { dimension: 'multiple', axisType: 'value', },          },          parameter: {            'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', },            'yAxisUnit': { valueType: 'string', defaultValue: '', description: 
 'unit of yAxis', },        },      },    }    this.transformation = new AdvancedTransformation(config, spec)  }  ...  // `render` will be called whenever `axis` or `parameter` is changed   render(data) {    const { chart, parameter, column, transformer, } = data    if (chart === 'line') {      const transformed = transformer()      // draw line chart     } else if (chart === 'no-group') {      const transformed = transformer()      // draw no-group chart     }  }}Spec: axisField NameAvailable Values (type)DescriptiondimensionsingleAxis can contains only 1 columndimensionmultipleAxis can contains multiple columnsaxisTypekeyColumn(s) in this axis will be used as key like in PivotTransformation. These columns will be served in column.keyaxisTypeaggregatorColumn(s) in this axis will be used as value like in PivotTransformation. These columns will be served in column.aggregatoraxisTypegroupColumn(s) in this axis will be used as group like i
 n PivotTransformation. These columns will be served in column.groupaxisType(string)Any string value can be used here. These columns will be served in column.custommaxAxisCount (optional)(int)The max number of columns that this axis can contain. (unlimited if undefined)minAxisCount (optional)(int)The min number of columns that this axis should contain to draw chart. (1 in case of single dimension)description (optional)(string)Description for the axis.Here is an example.axis: {  'xAxis': { dimension: 'multiple', axisType: 'key',  },  'yAxis': { dimension: 'multiple', axisType: 'aggregator'},  'category': { dimension: 'multiple', axisType: 'group', maxAxisCount: 2, valueType: 'string', },},Spec: sharedAxisIf you set sharedAxis: false for sub charts, then their axes are persisted in global space (shared). It's useful for 
 when you creating multiple sub charts sharing their axes but have different parameters. For example, basic-column, stacked-column, percent-columnpie and donutHere is an example.    const spec = {      charts: {        'column': {          transform: { method: 'array', },          sharedAxis: true,          axis: { ... },          parameter: { ... },        },        'stacked': {          transform: { method: 'array', },          sharedAxis: true,          axis: { ... }          parameter: { ... },        },Spec: parameterField NameAvailable Values (type)DescriptionvalueTypestringParameter which has string valuevalueTypeintParameter which has int valuevalueTypefloatParameter which has float valuevalueTypebooleanParameter which has boolean value used with checkbox widget usuallyvalueTypeJSONParameter which has JSON value used with textarea widget usually. defaultValue should be "" (empty string). Thisdes
 cription(string)Description of this parameter. This value will be parsed as HTML for pretty outputwidgetinputUse input widget. This is the default widget (if widget is undefined)widgetcheckboxUse checkbox widget.widgettextareaUse textarea widget.widgetoptionUse select + option widget. This parameter should have optionValues field as well.optionValues(Array)Available option values used with the option widgetHere is an example.parameter: {  // string type, input widget  'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', },  // boolean type, checkbox widget  'inverted': { widget: 'checkbox', valueType: 'boolean', defaultValue: false, description: 'invert x and y axes', },  // string type, option widget with `optionValues`  'graphType': { widget: 'option', valueType: 'string', defa
 ultValue: 'line', description: 'graph type', optionValues: [ 'line', 'smoothedLine', 'step', ], },  // HTML in `description`  'dateFormat': { valueType: 'string', defaultValue: '', description: 'format of date (<a href="https://docs.amcharts.com/3/javascriptcharts/AmGraph#dateFormat">doc</a>) (e.g YYYY-MM-DD)', },  // JSON type, textarea widget  'yAxisGuides': { widget: 'textarea', valueType: 'JSON', defaultValue: '', description: 'guides of yAxis ', },Spec: transformField NameAvailable Values (type)Descriptionmethodobjectdesigned for rows requiring object manipulationmethodarraydesigned for rows requiring array manipulationmethodarray:2-keydesigned for xyz charts (e.g bubble chart)methoddrill-downdesigned for drill-d
 own chartsmethodrawwill return the original tableData.rowsWhatever you specified as transform.method, the transformer value will be always function for lazy computation. // advanced-transformation.util#getTransformerif (transformSpec.method === 'raw') {  transformer = () => { return rows; }} else if (transformSpec.method === 'array') {  transformer = () => {    ...    return { ... }  }}Here is actual usage.class AwesomeVisualization extends Visualization {  constructor(...) { /** setup your spec */ }  ...   // `render` will be called whenever `axis` or `parameter` are changed  render(data) {    const { chart, parameter, column, transformer, } = data    if (chart === 'line') {      const transformed = transformer()      // draw line chart     } else if (chart === 'no-group') {      const transformed = transformer()      // draw no-group chart     }  }  ...}",
-      "url": " /development/helium/writing_visualization_transformation.html",
-      "group": "development/helium",
-      "excerpt": "Description for Transformations"
+    "/interpreter/python.html": {
+      "title": "Python 2 & 3 Interpreter for Apache Zeppelin",
+      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Python 2 & 3 Interpreter for Apache ZeppelinConfiguration      Property    Default    Description        zeppelin.python    python    Path of the already installed Python binary (could be python2 or python3).    If python is not in your $PATH you can set the absolute directory (example : /usr/bin/python)            zeppelin.python.maxResult    1000    Max number of dataframe rows to display.  Enabling Python InterpreterIn a
  notebook, to enable the Python interpreter, click on the Gear icon and select PythonUsing the Python InterpreterIn a paragraph, use %python to select the Python interpreter and then input all commands.The interpreter can only work if you already have python installed (the interpreter doesn't bring it own python binaries).To access the help, type help()Python environmentsDefaultBy default, PythonInterpreter will use python command defined in zeppelin.python property to run python process.The interpreter can use all modules already installed (with pip, easy_install...)CondaConda is an package management system and environment management system for python.%python.conda interpreter lets you change between environments.Usageget the Conda Infomation: %python.conda infolist the Conda environments: %python.conda env listcreate a conda enviornment: %python.conda create --name [ENV NAME]activate an environment (python interpreter will be restarted): %python.conda activate [ENV NAME]d
 eactivate%python.conda deactivateget installed package list inside the current environment%python.conda listinstall package%python.conda install [PACKAGE NAME]uninstall package%python.conda uninstall [PACKAGE NAME]Docker%python.docker interpreter allows PythonInterpreter creates python process in a specified docker container.Usageactivate an environment%python.docker activate [Repository]%python.docker activate [Repository:Tag]%python.docker activate [Image Id]deactivate%python.docker deactivateHere is an example# activate latest tensorflow image as a python environment%python.docker activate gcr.io/tensorflow/tensorflow:latestUsing Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your Python code.Zeppelin Dynamic Form can only be used if py4j Python library is installed in your system. If not, you can install it with pip install py4j.Example : %python### Input formprint (z.input("f1","defaultValue"))### Select formprint (z.sele
 ct("f1",[("o1","1"),("o2","2")],"2"))### Checkbox formprint("".join(z.checkbox("f3", [("o1","1"), ("o2","2")],["1"])))Matplotlib integrationThe python interpreter can display matplotlib figures inline automatically using the pyplot module:%pythonimport matplotlib.pyplot as pltplt.plot([1, 2, 3])This is the recommended method for using matplotlib from within a Zeppelin notebook. The output of this command will by default be converted to HTML by implicitly making use of the %html magic. Additional configuration can be achieved using the builtin z.configure_mpl() method. For example, z.configure_mpl(width=400, height=300, fmt='svg')plt.plot([1, 2, 3])Will produce a 400x300 image in SVG format, which by default are normally 600x400 and PNG r
 espectively. In the future, another option called angular can be used to make it possible to update a plot produced from one paragraph directly from another (the output will be %angular instead of %html). However, this feature is already available in the pyspark interpreter. More details can be found in the included "Zeppelin Tutorial: Python - matplotlib basic" tutorial notebook. If Zeppelin cannot find the matplotlib backend files (which should usually be found in $ZEPPELIN_HOME/interpreter/lib/python) in your PYTHONPATH, then the backend will automatically be set to agg, and the (otherwise deprecated) instructions below can be used for more limited inline plotting.If you are unable to load the inline backend, use z.show(plt): python%pythonimport matplotlib.pyplot as pltplt.figure()(.. ..)z.show(plt)plt.close()The z.show() function can take optional parameters to adapt graph dimensions (width and height) as well as output format (png or optionally svg).%pythonz.s
 how(plt, width='50px')z.show(plt, height='150px', fmt='svg')Pandas integrationApache Zeppelin Table Display System provides built-in data visualization capabilities. Python interpreter leverages it to visualize Pandas DataFrames though similar z.show() API, same as with Matplotlib integration.Example:import pandas as pdrates = pd.read_csv("bank.csv", sep=";")z.show(rates)SQL over Pandas DataFramesThere is a convenience %python.sql interpreter that matches Apache Spark experience in Zeppelin and enables usage of SQL language to query Pandas DataFrames and visualization of results though built-in Table Display System.Pre-requestsPandas pip install pandasPandaSQL pip install -U pandasqlIn case default binded interpreter is Python (first in the interpreter list, under the Gear Icon), you can just use it as %sql i.efirst paragraphimport pandas as pdrates = pd.read_csv("bank.csv", sep=&am
 p;quot;;")next paragraph%sqlSELECT * FROM rates WHERE age < 40Otherwise it can be referred to as %python.sqlIPython SupportIPython is more powerful than the default python interpreter with extra functionality. You can use IPython with Python2 or Python3 which depends on which python you set zeppelin.python.Pre-requests- Jupyter `pip install jupyter`- grpcio `pip install grpcio`If you already install anaconda, then you just need to install grpcio as Jupyter is already included in anaconda.In addition to all basic functions of the python interpreter, you can use all the IPython advanced features as you use it in Jupyter Notebook.e.g. Use IPython magic%python.ipython#python helprange?#timeit%timeit range(100)Use matplotlib %python.ipython%matplotlib inlineimport matplotlib.pyplot as pltprint("hello world")data=[1,2,3,4]plt.figure()plt.plot(data)We also make ZeppelinContext available in IPython Interpreter. You can use ZeppelinContext to create dynamic 
 forms and display pandas DataFrame.e.g.Create dynamic formz.input(name='my_name', defaultValue='hello')Show pandas dataframeimport pandas as pddf = pd.DataFrame({'id':[1,2,3], 'name':['a','b','c']})z.show(df)By default, we would use IPython in %python.python if IPython is available. Otherwise it would fall back to the original Python implementation.If you don't want to use IPython, then you can set zeppelin.python.useIPython as false in interpreter setting.Technical descriptionFor in-depth technical details on current implementation please refer to python/README.md.Some features not yet implemented in the Python InterpreterInterrupt a paragraph execution (cancel() method) is currently only supported in Linux and MacOs. If interpreter runs in another operating system (for instance MS Windows) , interrupt a paragraph will close the whole interpreter. A JIRA ticket (Z
 EPPELIN-893) is opened to implement this feature in a next release of the interpreter.Progression bar in webUI  (getProgress() method) is currently not implemented.Code-completion is currently not implemented.",
+      "url": " /interpreter/python.html",
+      "group": "interpreter",
+      "excerpt": "Python is a programming language that lets you work quickly and integrate systems more effectively."
     }
     ,
     
   
 
-    "/development/writing_zeppelin_interpreter.html": {
-      "title": "Writing a New Interpreter",

[... 1111 lines stripped ...]