You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by zj...@apache.org on 2020/02/10 02:18:51 UTC
[zeppelin] branch master updated: [ZEPPELIN-4490]. Update
ZeppelinContext document
This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new ff3662f [ZEPPELIN-4490]. Update ZeppelinContext document
ff3662f is described below
commit ff3662fcfb4de780f1a7e03e7e3ea437d7d3b2ac
Author: Jeff Zhang <zj...@apache.org>
AuthorDate: Tue Dec 17 18:10:02 2019 +0800
[ZEPPELIN-4490]. Update ZeppelinContext document
### What is this PR for?
This is a straightforward PR which update the ZeppelinContext document, as ZeppelinContext has changed a lots, many new features and refactoring is done on this. So we should update document accordingly.
### What type of PR is it?
[ Documentation ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4490
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zj...@apache.org>
Closes #3559 from zjffdu/ZEPPELIN-4490 and squashes the following commits:
adeed7969 [Jeff Zhang] [ZEPPELIN-4490]. Update ZeppelinContext document
---
.../img/docs-img/zeppelin_context_share_string.png | Bin 0 -> 125751 bytes
.../img/docs-img/zeppelin_context_share_table.png | Bin 0 -> 242987 bytes
docs/usage/other_features/zeppelin_context.md | 166 +++++++++++++++------
3 files changed, 120 insertions(+), 46 deletions(-)
diff --git a/docs/assets/themes/zeppelin/img/docs-img/zeppelin_context_share_string.png b/docs/assets/themes/zeppelin/img/docs-img/zeppelin_context_share_string.png
new file mode 100644
index 0000000..9ea31f5
Binary files /dev/null and b/docs/assets/themes/zeppelin/img/docs-img/zeppelin_context_share_string.png differ
diff --git a/docs/assets/themes/zeppelin/img/docs-img/zeppelin_context_share_table.png b/docs/assets/themes/zeppelin/img/docs-img/zeppelin_context_share_table.png
new file mode 100644
index 0000000..8e14ebb
Binary files /dev/null and b/docs/assets/themes/zeppelin/img/docs-img/zeppelin_context_share_table.png differ
diff --git a/docs/usage/other_features/zeppelin_context.md b/docs/usage/other_features/zeppelin_context.md
index ced400a..e5db9a2 100644
--- a/docs/usage/other_features/zeppelin_context.md
+++ b/docs/usage/other_features/zeppelin_context.md
@@ -38,7 +38,7 @@ environments is described below.
In many programming-language interpreters (e.g. Apache Spark, Python, R) the zeppelin-context is available
as a predefined variable `z` that can be used by directly invoking its methods.
The methods available on the `z` object are described below.
-Other interpreters based on programming languages like spark.dep, Apache Beam, etc. also provide the
+Other interpreters based on programming languages like Apache Beam, etc. also provide the
predefined variable `z`.
### Exploring Spark DataFrames
@@ -51,7 +51,7 @@ z.show(df)
```
This display functionality using the `show` method is planned to be extended uniformly to
-other interpreters that can access the `z` object.
+other interpreters that can access the `z` object (Flink already support to show table too).
### Object Exchange
`ZeppelinContext` extends map and it's shared between the Apache Spark and Python environments.
@@ -61,36 +61,103 @@ So you can put some objects using Scala (in an Apache Spark cell) and read it fr
<div data-lang="scala" markdown="1">
{% highlight scala %}
-// Put object from scala
+// Put/Get object from scala
%spark
-val myObject = ...
-z.put("objName", myObject)
-// Exchanging data frames
-myScalaDataFrame = ...
-z.put("myScalaDataFrame", myScalaDataFrame)
+val myObject = "hello'
+z.put("objName", myObject)
+z.get("objName")
-val myPythonDataFrame = z.get("myPythonDataFrame").asInstanceOf[DataFrame]
{% endhighlight %}
</div>
<div data-lang="python" markdown="1">
{% highlight python %}
-# Get object from python
+# Put/Get object from python
%spark.pyspark
+
+val myObject = "hello"
+z.put("objName", myObject)
myObject = z.get("objName")
-# Exchanging data frames
-myPythonDataFrame = ...
-z.put("myPythonDataFrame", postsDf._jdf)
+# df is Python pandas DataFrame
+# "table_name" must be table type. Currently only sql interpreter (%spark.sql or %jdbc) result is supported.
+df = z.getAsDataFrame("table_name")
+{% endhighlight %}
+
+ </div>
+
+<div data-lang="R" markdown="1">
+
+{% highlight python %}
+# Get/Put object from R
+%spark.r
+
+z.put("objName", myObject)
+myObject <- z.get("objName")
+
+# df is R DataFrame
+# "table_name" must be table type. Currently only sql interpreter (%spark.sql or %jdbc) result is supported.
+df <- z.getAsDataFrame("table_name")
-myScalaDataFrame = DataFrame(z.get("myScalaDataFrame"), sqlContext)
{% endhighlight %}
</div>
+
</div>
+Currently, there're two types of data could be shared across interpreters:
+
+* String Data
+* Table Data
+
+#### Share String Object
+
+Here's one example we share one String object `maxAge` between Spark interpreter and jdbc interpreter.
+
+```scala
+%spark
+
+z.put("maxAge", 83)
+```
+
+```sql
+%jdbc(interpolate=true)
+
+select * from bank where age = {maxAge}
+```
+
+<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/zeppelin_context_share_string.png" height="100%" width="100%">
+
+
+#### Share Table Object
+Here's one example we share one Table object between jdbc interpreter and python interpreter.
+
+```sql
+%jdbc(saveAs=bank)
+
+select * from bank
+```
+
+
+```python
+%python.ipython
+
+%matplotlib inline
+
+import warnings
+warnings.filterwarnings("ignore")
+from plotnine import ggplot, geom_histogram, aes, facet_wrap
+
+bank = z.getAsDataFrame('bank')
+(ggplot(bank, aes(x='age'))
+```
+
+<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/zeppelin_context_share_table.png" height="100%" width="100%">
+
+
+
### Form Creation
`ZeppelinContext` provides functions for creating forms.
@@ -100,18 +167,19 @@ In Scala and Python environments, you can create forms programmatically.
{% highlight scala %}
%spark
+
/* Create text input form */
-z.input("formName")
+z.input("input_1")
/* Create text input form with default value */
-z.input("formName", "defaultValue")
+z.input("input_2", "defaultValue")
/* Create select form */
-z.select("formName", Seq(("option1", "option1DisplayName"),
+z.select("select_1", Seq(("option1", "option1DisplayName"),
("option2", "option2DisplayName")))
/* Create select form with default value*/
-z.select("formName", "option1", Seq(("option1", "option1DisplayName"),
+z.select("select_2", "option1", Seq(("option1", "option1DisplayName"),
("option2", "option2DisplayName")))
{% endhighlight %}
@@ -120,38 +188,44 @@ z.select("formName", "option1", Seq(("option1", "option1DisplayName"),
{% highlight python %}
%spark.pyspark
+
# Create text input form
-z.input("formName")
+z.input("input_1")
# Create text input form with default value
-z.input("formName", "defaultValue")
+z.input("input_2", "defaultValue")
# Create select form
-z.select("formName", [("option1", "option1DisplayName"),
+z.select("select_1", [("option1", "option1DisplayName"),
("option2", "option2DisplayName")])
# Create select form with default value
-z.select("formName", [("option1", "option1DisplayName"),
+z.select("select_2", [("option1", "option1DisplayName"),
("option2", "option2DisplayName")], "option1")
{% endhighlight %}
</div>
</div>
-In sql environment, you can create form in simple template.
+Patterns of the form ${ ... } are used to dynamically create additional HTML elements
+for requesting user input (that replaces the corresponding pattern in the paragraph text).
+Currently only [text](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/text),
+[select](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/select) with
+[options](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/option), and
+[checkbox](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/checkbox) are supported.
+
+Dynamic forms are described in detail here: [Dynamic Form](../usage/dynamic_form/intro.html).
+
+In sql environment, you can create dynamic form in simple template.
```sql
%spark.sql
+
select * from ${table=defaultTableName} where text like '%${search}%'
```
To learn more about dynamic form, checkout [Dynamic Form](../usage/dynamic_form/intro.html).
-### Interpreter-Specific Functions
-
-Some interpreters use a subclass of `BaseZepplinContext` augmented with interpreter-specific functions.
-For example functions of the dependency loader (%spark.dep) can be invoked as `z.addRepo()`, `z.load()`, etc.
-Such interpreter-specific functions are described within each interpreter's documentation.
## Usage with Embedded Commands
@@ -168,17 +242,6 @@ but object interpolation is only available in a small, but growing, list of inte
(marked with an asterisk in the table above).
Both these zeppelin-context features are described below.
-### Dynamic Forms
-
-Patterns of the form ${ ... } are used to dynamically create additional HTML elements
-for requesting user input (that replaces the corresponding pattern in the paragraph text).
-Currently only [text](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/text),
-[select](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/select) with
-[options](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/option), and
-[checkbox](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/checkbox) are supported.
-
-Dynamic forms are described in detail here: [Dynamic Form](../usage/dynamic_form/intro.html).
-
### Object Interpolation
Some interpreters can interpolate object values from `z` into the paragraph text by using the
`{variable-name}` syntax. The value of any object previously `put` into `z` can be
@@ -188,18 +251,22 @@ The following example shows one use of this facility:
####In Scala cell:
```scala
+%spark
+
z.put("minAge", 35)
```
####In later SQL cell:
```sql
-%sql select * from members where age >= {minAge}
+%spark.sql
+
+select * from members where age >= {minAge}
```
The interpolation of a `{var-name}` pattern is performed only when `z` contains an object with the specified name.
But the pattern is left unchanged if the named object does not exist in `z`.
-Further, all `{var-name}` patterns within the paragraph text must must be translatable for any interpolation to occur --
+Further, all `{var-name}` patterns within the paragraph text must be translatable for any interpolation to occur --
translation of only some of the patterns in a paragraph text is never done.
In some situations, it is necessary to use { and } characters in a paragraph text without invoking the
@@ -208,15 +275,18 @@ doubled braces {{ and }} should be used. The following example shows the use of
regular expression containing just { and } into the paragraph text.
```sql
-%sql select * from members where name rlike '[aeiou]{{3}}'
+%spark.sql
+{% raw %}
+select * from members where name rlike '[aeiou]{{3}}'
+{% endraw %}
```
To summarize, patterns of the form `{var-name}` within the paragraph text will be interpolated only if a predefined
object of the specified name exists. Additionally, all such patterns within the paragraph text should also
-be translatable for any interpolation to occur. Patterns of the form `{{any-text}}` are translated into `{any-text}`.
-These translations are performed only when all occurrences of `{`, `}`, `{{`, and `}}` in the paragraph text conform
+be translatable for any interpolation to occur. Patterns of the form {% raw %} `{{any-text}}` {% endraw %} are translated into `{any-text}`.
+These translations are performed only when all occurrences of `{`, `}`, {% raw %} `{{`, and `}}`{% endraw %} in the paragraph text conform
to one of the two forms described above. Paragraph text containing `{` and/or `}` characters used in any other way
-(than `{var-name}` and `{{any-text}}`) is used as-is without any changes.
+(than `{var-name}` and {% raw %} `{{any-text}}` {% endraw %} ) is used as-is without any changes.
No error is flagged in any case. This behavior is identical to the implementation of a similar feature in
Jupyter's shell invocation using the `!` magic command.
@@ -224,12 +294,16 @@ This feature is disabled by default, and must be explicitly turned on for each i
by setting the value of an interpreter-specific property to `true`.
Consult the _Configuration_ section of each interpreter's documentation
to find out if object interpolation is implemented, and the name of the parameter that must be set to `true` to
-enable the feature. The name of the parameter used to enable this feature it is different for each interpreter.
+enable the feature. The name of the parameter used to enable this feature is different for each interpreter.
For example, the SparkSQL and Shell interpreters use the parameter names `zeppelin.spark.sql.interpolation` and
`zeppelin.shell.interpolation` respectively.
At present only the SparkSQL, JDBC, and Shell interpreters support object interpolation.
+### Interpreter-Specific Functions
+
+Some interpreters use a subclass of `BaseZepplinContext` augmented with interpreter-specific functions.
+Such interpreter-specific functions are described within each interpreter's documentation.