You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by zj...@apache.org on 2019/11/01 01:43:58 UTC
[zeppelin] branch master updated: [ZEPPELIN-4375]. Add more flink
tutorial notes
This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 0566354 [ZEPPELIN-4375]. Add more flink tutorial notes
0566354 is described below
commit 056635409a39ba7916a4865cc75c4d712dda21a6
Author: Jeff Zhang <zj...@apache.org>
AuthorDate: Thu Oct 17 19:33:51 2019 +0800
[ZEPPELIN-4375]. Add more flink tutorial notes
### What is this PR for?
This PR add 2 tutorial note for flink，one for batch，another for streaming. Besides that, I also update the flink documentation for some improvement.
### What type of PR is it?
[ Improvement | Documentation ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4375
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zj...@apache.org>
Closes #3481 from zjffdu/ZEPPELIN-4375 and squashes the following commits:
d1ee659b0 [Jeff Zhang] address comment
1e8db163a [Jeff Zhang] [ZEPPELIN-4375]. Add more flink tutorial notes
---
docs/interpreter/flink.md | 35 +-
.../Flink Batch Tutorial_2EN1E1ATY.zpln | 602 +++++++++++++++++++++
.../Flink Stream Tutorial_2ER62Y5VJ.zpln | 427 +++++++++++++++
3 files changed, 1061 insertions(+), 3 deletions(-)
diff --git a/docs/interpreter/flink.md b/docs/interpreter/flink.md
index ad71487..3fc75c9 100644
--- a/docs/interpreter/flink.md
+++ b/docs/interpreter/flink.md
@@ -78,7 +78,7 @@ You can also set other flink properties which are not listed in the table. For a
<tr>
<td>flink.execution.mode</td>
<td>local</td>
- <td>Execution mode of flink, e.g. local/yarn/remote</td>
+ <td>Execution mode of flink, e.g. local | yarn | remote</td>
</tr>
<tr>
<td>flink.execution.remote.host</td>
@@ -148,13 +148,18 @@ You can also set other flink properties which are not listed in the table. For a
<tr>
<td>zeppelin.flink.planner</td>
<td>blink</td>
- <td>planner or flink table api, blink or flink</td>
+ <td>planner of flink table api, <em>blink</em> or <em>flink</em></td>
</tr>
<tr>
<td>zeppelin.pyflink.python</td>
<td>python</td>
<td>python executable for pyflink</td>
</tr>
+ <tr>
+ <td>HADOOP_CONF_DIR</td>
+ <td></td>
+ <td>location of hadoop conf, this is must be set if running in yarn mode</td>
+ </tr>
</table>
@@ -166,6 +171,25 @@ Zeppelin will create 4 variables to represent flink's entrypoint:
* `stenv` (StreamTableEnvironment)
* `btenv` (BatchTableEnvironment)
+## Flink Planner
+
+Starting from Flink 1.9, there're 2 planners supported by Flink's table api: `flink` & `blink`.
+* If you want to use DataSet api, then please use flink planner (specify `zeppelin.flink.planner` to `flink`).
+* In other cases, we would always recommend you to use `blink` planner which is also the default value of `zeppelin.flink.planner`.
+
+## How to use Hive
+
+In order to use Hive in Flink, you have to do several setting.
+* Set `zeppelin.flink.enableHive` to `true`
+* Copy necessary dependencies to flink's lib folder, check this [link](https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/hive/#depedencies) for more details
+ * flink-connector-hive_{scala_version}-{flink.version}.jar
+ * flink-hadoop-compatibility_{scala_version}-{flink.version}.jar
+ * flink-shaded-hadoop-2-uber-{hadoop.version}-{flink-shaded.version}.jar
+ * hive-exec-2.x.jar (for Hive 1.x, you need to copy hive-exec-1.x.jar, hive-metastore-1.x.jar, libfb303-0.9.2.jar and libthrift-0.9.2.jar)
+* Specify `HIVE_CONF_DIR` either in flink interpreter setting or `zeppelin-env.sh`
+* Specify `zeppelin.flink.hive.version`, by default it is 2.3.4. If you are using Hive 1.2.x, then you need to set it as `1.2.2`
+
+After these settings, you will be able to query hive table via either table api `%flink` or batch sql `%flink.bsql`
## ZeppelinContext
Zeppelin automatically injects `ZeppelinContext` as variable `z` in your Scala/Python environment. `ZeppelinContext` provides some additional functions and utilities.
@@ -173,8 +197,13 @@ See [Zeppelin-Context](../usage/other_features/zeppelin_context.html) for more d
## IPython support
-By default, zeppelin would use IPython in `pyflink` when IPython is available, Otherwise it would fall back to the original PyFlink implementation.
+By default, zeppelin would use IPython in `%flink.pyflink` when IPython is available, Otherwise it would fall back to the original python implementation.
If you don't want to use IPython, then you can set `zeppelin.pyflink.useIPython` as `false` in interpreter setting. For the IPython features, you can refer doc
[Python Interpreter](python.html)
+## Tutorial Notes
+
+Zeppelin is shipped with several Flink tutorial notes which may be helpful for you.
+
+
diff --git a/notebook/Zeppelin Tutorial/Flink Batch Tutorial_2EN1E1ATY.zpln b/notebook/Zeppelin Tutorial/Flink Batch Tutorial_2EN1E1ATY.zpln
new file mode 100644
index 0000000..018089e
--- /dev/null
+++ b/notebook/Zeppelin Tutorial/Flink Batch Tutorial_2EN1E1ATY.zpln
@@ -0,0 +1,602 @@
+{
+ "paragraphs": [
+ {
+ "title": "Introduction",
+ "text": "%md\n\nThis is a tutorial note for Flink batch scenario (`To be noticed, you need to use flink 1.9 or afterwards`), . You can run flink scala api via `%flink` and run flink batch sql via `%flink.bsql`. \nThis note use flink\u0027s DataSet api to demonstrate flink\u0027s batch capablity. DataSet is only supported by flink planner, so here we have to specify the planner `zeppelin.flink.planner` as `flink`, otherwise it would use blink planner by default which doesn\u0027t su [...]
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:15:54.910",
+ "config": {
+ "tableHide": false,
+ "editorSetting": {
+ "language": "markdown",
+ "editOnDblClick": true,
+ "completionKey": "TAB",
+ "completionSupport": false
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/markdown",
+ "fontSize": 9.0,
+ "editorHide": true,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "HTML",
+ "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis is a tutorial note for Flink batch scenario (\u003ccode\u003eTo be noticed, you need to use flink 1.9 or afterwards\u003c/code\u003e), . You can run flink scala api via \u003ccode\u003e%flink\u003c/code\u003e and run flink batch sql via \u003ccode\u003e%flink.bsql\u003c/code\u003e.\u003cbr/\u003eThis note use flink\u0026rsquo;s DataSet api to demonstrate flink\u0026rsquo;s batch capablity. DataSet is onl [...]
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641095_-188362229",
+ "id": "paragraph_1547794482637_957545547",
+ "dateCreated": "2019-09-26 17:20:41.095",
+ "dateStarted": "2019-10-08 15:15:54.910",
+ "dateFinished": "2019-10-08 15:15:54.923",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Configure Flink Interpreter",
+ "text": "%flink.conf\n\nFLINK_HOME \u003cFLINK_INSTALLATION\u003e\n# DataSet is only supported in flink planner, so here we use flink planner. By default it is blink planner\nzeppelin.flink.planner flink\n\n\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:37:39.948",
+ "config": {
+ "editorSetting": {
+ "language": "text",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/text",
+ "fontSize": 9.0,
+ "editorHide": false,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": []
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641097_1856810430",
+ "id": "paragraph_1546565092490_1952685806",
+ "dateCreated": "2019-09-26 17:20:41.097",
+ "dateStarted": "2019-10-11 10:00:42.031",
+ "dateFinished": "2019-10-11 10:00:42.052",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%sh\n\ncd /tmp\nwget https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:16:45.127",
+ "config": {
+ "runOnSelectionChange": true,
+ "title": false,
+ "checkEmpty": true,
+ "colWidth": 12.0,
+ "fontSize": 9.0,
+ "enabled": true,
+ "results": {},
+ "editorSetting": {
+ "language": "sh",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": false
+ },
+ "editorMode": "ace/mode/sh"
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TEXT",
+ "data": "--2019-10-08 15:16:46-- https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv\nResolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.120.2\nConnecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.120.2|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 461474 (451K) [application/octet-stream]\nSaving to: \u0027bank.csv\u0027\n\n 0K .......... .......... .......... .......... .......... 11% 91.0K 4s\n 50K .......... ...... [...]
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489915993_-1377803690",
+ "id": "paragraph_1569489915993_-1377803690",
+ "dateCreated": "2019-09-26 17:25:15.993",
+ "dateStarted": "2019-10-08 15:16:45.132",
+ "dateFinished": "2019-10-08 15:16:50.387",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Load Bank Data",
+ "text": "%flink\n\nval bankText \u003d benv.readTextFile(\"/tmp/bank.csv\")\nval bank \u003d bankText.map(s \u003d\u003e s.split(\";\")).filter(s \u003d\u003e s(0) !\u003d \"\\\"age\\\"\").map(\n s \u003d\u003e (s(0).toInt,\n s(1).replaceAll(\"\\\"\", \"\"),\n s(2).replaceAll(\"\\\"\", \"\"),\n s(3).replaceAll(\"\\\"\", \"\"),\n s(5).replaceAll(\"\\\"\", \"\").toInt\n )\n )\n\nbtenv.registerDataSet(\"bank\", bank, \u0027age, \u0027j [...]
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:01:44.897",
+ "config": {
+ "editorSetting": {
+ "language": "scala",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/scala",
+ "fontSize": 9.0,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "table",
+ "height": 111.0,
+ "optionOpen": false
+ }
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TEXT",
+ "data": "\u001b[1m\u001b[34mbankText\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.api.scala.DataSet[String]\u001b[0m \u003d org.apache.flink.api.scala.DataSet@683ccfd\n\u001b[1m\u001b[34mbank\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.api.scala.DataSet[(Int, String, String, String, Int)]\u001b[0m \u003d org.apache.flink.api.scala.DataSet@35e6c08f\n"
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641099_-2030350664",
+ "id": "paragraph_1546584347815_1533642635",
+ "dateCreated": "2019-09-26 17:20:41.099",
+ "dateStarted": "2019-10-11 10:01:44.904",
+ "dateFinished": "2019-10-11 10:02:12.333",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%flink.bsql\n\ndescribe bank\n\n\n\n\n\n\n\n\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:07:51.800",
+ "config": {
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "runOnSelectionChange": true,
+ "title": false,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "table",
+ "height": 300.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "table": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ }
+ },
+ "commonSetting": {}
+ }
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TEXT",
+ "data": "Column\tType\nOptional[age]\tOptional[INT]\nOptional[job]\tOptional[STRING]\nOptional[marital]\tOptional[STRING]\nOptional[education]\tOptional[STRING]\nOptional[balance]\tOptional[INT]\n"
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491358206_1385745589",
+ "id": "paragraph_1569491358206_1385745589",
+ "dateCreated": "2019-09-26 17:49:18.206",
+ "dateStarted": "2019-10-11 10:07:51.808",
+ "dateFinished": "2019-10-11 10:07:52.076",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%flink.bsql\n\nselect age, count(1) as v\nfrom bank \nwhere age \u003c 30 \ngroup by age \norder by age",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:07:53.645",
+ "config": {
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "runOnSelectionChange": true,
+ "title": false,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "table",
+ "height": 300.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "age": "string",
+ "v": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ }
+ },
+ "commonSetting": {}
+ }
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TABLE",
+ "data": "age\tv\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n"
+ },
+ {
+ "type": "TEXT",
+ "data": ""
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491511585_-677666348",
+ "id": "paragraph_1569491511585_-677666348",
+ "dateCreated": "2019-09-26 17:51:51.585",
+ "dateStarted": "2019-10-11 10:07:53.654",
+ "dateFinished": "2019-10-11 10:08:10.439",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%flink.bsql\n\nselect age, count(1) as v \nfrom bank \nwhere age \u003c ${maxAge\u003d30} \ngroup by age \norder by age",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:08:13.910",
+ "config": {
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "runOnSelectionChange": true,
+ "title": false,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "pieChart",
+ "height": 300.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "age": "string",
+ "v": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ },
+ "multiBarChart": {
+ "rotate": {
+ "degree": "-45"
+ },
+ "xLabelStatus": "default"
+ }
+ },
+ "commonSetting": {},
+ "keys": [
+ {
+ "name": "age",
+ "index": 0.0,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [],
+ "values": [
+ {
+ "name": "v",
+ "index": 1.0,
+ "aggr": "sum"
+ }
+ ]
+ },
+ "helium": {}
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {
+ "maxAge": "40"
+ },
+ "forms": {
+ "maxAge": {
+ "type": "TextBox",
+ "name": "maxAge",
+ "defaultValue": "30",
+ "hidden": false
+ }
+ }
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TABLE",
+ "data": "age\tv\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n30\t150\n31\t199\n32\t224\n33\t186\n34\t231\n35\t180\n36\t188\n37\t161\n38\t159\n39\t130\n"
+ },
+ {
+ "type": "TEXT",
+ "data": ""
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641100_115468969",
+ "id": "paragraph_1546592465802_-1181051373",
+ "dateCreated": "2019-09-26 17:20:41.100",
+ "dateStarted": "2019-10-11 10:08:13.923",
+ "dateFinished": "2019-10-11 10:08:14.876",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%flink.bsql\n\nselect age, count(1) as v \nfrom bank \nwhere marital\u003d\u0027${marital\u003dsingle,single|divorced|married}\u0027 \ngroup by age \norder by age",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:37:59.493",
+ "config": {
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "runOnSelectionChange": true,
+ "title": false,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "multiBarChart",
+ "height": 300.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "age": "string",
+ "v": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ },
+ "multiBarChart": {
+ "rotate": {
+ "degree": "-45"
+ },
+ "xLabelStatus": "default"
+ },
+ "lineChart": {
+ "rotate": {
+ "degree": "-45"
+ },
+ "xLabelStatus": "default"
+ },
+ "stackedAreaChart": {
+ "rotate": {
+ "degree": "-45"
+ },
+ "xLabelStatus": "default"
+ }
+ },
+ "commonSetting": {},
+ "keys": [
+ {
+ "name": "age",
+ "index": 0.0,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [],
+ "values": [
+ {
+ "name": "v",
+ "index": 1.0,
+ "aggr": "sum"
+ }
+ ]
+ },
+ "helium": {}
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {
+ "marital": "married"
+ },
+ "forms": {
+ "marital": {
+ "type": "Select",
+ "options": [
+ {
+ "value": "single"
+ },
+ {
+ "value": "divorced"
+ },
+ {
+ "value": "married"
+ }
+ ],
+ "name": "marital",
+ "defaultValue": "single",
+ "hidden": false
+ }
+ }
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TABLE",
+ "data": "age\tv\n23\t3\n24\t11\n25\t11\n26\t18\n27\t26\n28\t23\n29\t37\n30\t56\n31\t104\n32\t105\n33\t103\n34\t142\n35\t109\n36\t117\n37\t100\n38\t99\n39\t88\n40\t105\n41\t97\n42\t91\n43\t79\n44\t68\n45\t76\n46\t82\n47\t78\n48\t91\n49\t87\n50\t74\n51\t63\n52\t66\n53\t75\n54\t56\n55\t68\n56\t50\n57\t78\n58\t67\n59\t56\n60\t36\n61\t15\n62\t5\n63\t7\n64\t6\n65\t4\n66\t7\n67\t5\n68\t1\n69\t5\n70\t5\n71\t5\n72\t4\n73\t6\n74\t2\n75\t3\n76\t1\n77\t5\n78\t2\n79\t3\n80\t6\n81\t1\n83\t [...]
+ },
+ {
+ "type": "TEXT",
+ "data": ""
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641100_-1622522691",
+ "id": "paragraph_1546592478596_-1766740165",
+ "dateCreated": "2019-09-26 17:20:41.100",
+ "dateStarted": "2019-10-11 10:08:16.423",
+ "dateFinished": "2019-10-11 10:08:17.295",
+ "status": "FINISHED"
+ },
+ {
+ "text": "\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-09-26 17:54:48.292",
+ "config": {
+ "colWidth": 12.0,
+ "fontSize": 9.0,
+ "enabled": true,
+ "results": {},
+ "editorSetting": {
+ "language": "scala",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "editorMode": "ace/mode/scala"
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569489641101_-1182720873",
+ "id": "paragraph_1553093710610_-1734599499",
+ "dateCreated": "2019-09-26 17:20:41.101",
+ "status": "READY"
+ }
+ ],
+ "name": "Flink Batch Tutorial",
+ "id": "2EN1E1ATY",
+ "defaultInterpreterGroup": "spark",
+ "version": "0.9.0-SNAPSHOT",
+ "permissions": {},
+ "noteParams": {},
+ "noteForms": {},
+ "angularObjects": {},
+ "config": {
+ "isZeppelinNotebookCronEnable": false
+ },
+ "info": {},
+ "path": "/Flink Batch Tutorial"
+}
\ No newline at end of file
diff --git a/notebook/Zeppelin Tutorial/Flink Stream Tutorial_2ER62Y5VJ.zpln b/notebook/Zeppelin Tutorial/Flink Stream Tutorial_2ER62Y5VJ.zpln
new file mode 100644
index 0000000..b8f3658
--- /dev/null
+++ b/notebook/Zeppelin Tutorial/Flink Stream Tutorial_2ER62Y5VJ.zpln
@@ -0,0 +1,427 @@
+{
+ "paragraphs": [
+ {
+ "title": "Introduction",
+ "text": "%md\n\nThis is a tutorial note for Flink Streaming application. You can run flink scala api via `%flink` and run flink stream sql via `%flink.ssql`. We provide 2 examples in this tutorial:\n\n* Classical word count example via Flink streaming\n* We simulate a web log stream source, and then query and visualize this streaming data in Zeppelin.\n\n\nFor now, the capability of supporting flink streaming job is very limited in Zeppelin. For example, canceling job is not suppor [...]
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:18:50.038",
+ "config": {
+ "tableHide": false,
+ "editorSetting": {
+ "language": "markdown",
+ "editOnDblClick": true,
+ "completionKey": "TAB",
+ "completionSupport": false
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/markdown",
+ "fontSize": 9.0,
+ "editorHide": false,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "HTML",
+ "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis is a tutorial note for Flink Streaming application. You can run flink scala api via \u003ccode\u003e%flink\u003c/code\u003e and run flink stream sql via \u003ccode\u003e%flink.ssql\u003c/code\u003e. We provide 2 examples in this tutorial:\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003eClassical word count example via Flink streaming\u003c/li\u003e\n \u003cli\u003eWe simulate a web log stream source, and [...]
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705460_-1023073277",
+ "id": "paragraph_1548052720723_1943177100",
+ "dateCreated": "2019-09-26 17:55:05.460",
+ "dateStarted": "2019-10-08 15:18:47.711",
+ "dateFinished": "2019-10-08 15:18:47.724",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Configure Flink Interpreter",
+ "text": "%flink.conf\n\nFLINK_HOME \u003cFLINK_INSTALLATION\u003e\n# Use blink planner for flink streaming scenario\nzeppelin.flink.planner blink\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:38:09.734",
+ "config": {
+ "editorSetting": {
+ "language": "text",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/text",
+ "fontSize": 9.0,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": []
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705461_327101264",
+ "id": "paragraph_1546571299955_275296580",
+ "dateCreated": "2019-09-26 17:55:05.461",
+ "dateStarted": "2019-10-08 15:21:48.153",
+ "dateFinished": "2019-10-08 15:21:48.165",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Stream WordCount",
+ "text": "%flink\n\nval data \u003d senv.fromElements(\"hello world\", \"hello flink\", \"hello hadoop\")\ndata.flatMap(line \u003d\u003e line.split(\"\\\\s\"))\n .map(w \u003d\u003e (w, 1))\n .keyBy(0)\n .sum(1)\n .print\n\nsenv.execute()\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:19:07.936",
+ "config": {
+ "editorSetting": {
+ "language": "scala",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/scala",
+ "fontSize": 9.0,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TEXT",
+ "data": "\u001b[1m\u001b[34mdata\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.streaming.api.scala.DataStream[String]\u001b[0m \u003d org.apache.flink.streaming.api.scala.DataStream@5a099566\n\u001b[1m\u001b[34mres0\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.streaming.api.datastream.DataStreamSink[(String, Int)]\u001b[0m \u003d org.apache.flink.streaming.api.datastream.DataStreamSink@58805cef\n(hello,1)\n(world,1)\n(hello,2)\n(flink,1)\n(hello,3)\n(hadoop,1)\n\u001b[1m\u00 [...]
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705461_377981606",
+ "id": "paragraph_1546571324670_-435705916",
+ "dateCreated": "2019-09-26 17:55:05.461",
+ "dateStarted": "2019-10-08 15:19:07.941",
+ "dateFinished": "2019-10-08 15:19:23.511",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Register a Stream DataSource to simulate web log",
+ "text": "%flink\n\nimport org.apache.flink.streaming.api.functions.source.SourceFunction\nimport org.apache.flink.table.api.TableEnvironment\nimport org.apache.flink.streaming.api.TimeCharacteristic\nimport org.apache.flink.streaming.api.checkpoint.ListCheckpointed\nimport java.util.Collections\nimport scala.collection.JavaConversions._\n\nsenv.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)\nsenv.enableCheckpointing(1000)\n\nval data \u003d senv.addSource(new SourceFunct [...]
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:19:31.503",
+ "config": {
+ "editorSetting": {
+ "language": "scala",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/scala",
+ "fontSize": 9.0,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TEXT",
+ "data": "import org.apache.flink.streaming.api.functions.source.SourceFunction\nimport org.apache.flink.table.api.TableEnvironment\nimport org.apache.flink.streaming.api.TimeCharacteristic\nimport org.apache.flink.streaming.api.checkpoint.ListCheckpointed\nimport java.util.Collections\nimport scala.collection.JavaConversions._\n\u001b[1m\u001b[34mres4\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.streaming.api.scala.StreamExecutionEnvironment\u001b[0m \u003d org.apache.flink. [...]
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705461_-1986381705",
+ "id": "paragraph_1546571333074_1869171983",
+ "dateCreated": "2019-09-26 17:55:05.461",
+ "dateStarted": "2019-10-08 15:19:31.507",
+ "dateFinished": "2019-10-08 15:19:34.040",
+ "status": "FINISHED"
+ },
+ {
+ "title": "Total Page View",
+ "text": "%flink.ssql(type\u003dsingle, parallelism\u003d1, refreshInterval\u003d3000, template\u003d\u003ch1\u003e{1}\u003c/h1\u003e until \u003ch2\u003e{0}\u003c/h2\u003e, enableSavePoint\u003dfalse, runWithSavePoint\u003dfalse)\n\nselect max(rowtime), count(1) from log\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:19:41.998",
+ "config": {
+ "savepointPath": "file:/tmp/save_point/savepoint-13e681-f552013d6184",
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "editorHide": false,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "table",
+ "height": 141.0,
+ "optionOpen": false
+ }
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705461_616808409",
+ "id": "paragraph_1546571459644_596843735",
+ "dateCreated": "2019-09-26 17:55:05.461",
+ "dateStarted": "2019-10-08 15:19:42.003",
+ "dateFinished": "2019-09-26 17:57:24.042",
+ "status": "ABORT"
+ },
+ {
+ "title": "Page View by Page",
+ "text": "%flink.ssql(type\u003dretract, refreshInterval\u003d2000, parallelism\u003d1, enableSavePoint\u003dfalse, runWithSavePoint\u003dfalse)\n\nselect url, count(1) as pv from log group by url",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-08 15:20:02.708",
+ "config": {
+ "savepointPath": "file:/flink/save_point/savepoint-e4f781-996290516ef1",
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 6.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "editorHide": false,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "multiBarChart",
+ "height": 198.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "url": "string",
+ "pv": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ },
+ "multiBarChart": {
+ "xLabelStatus": "default",
+ "rotate": {
+ "degree": "-45"
+ }
+ }
+ },
+ "commonSetting": {},
+ "keys": [],
+ "groups": [],
+ "values": []
+ },
+ "helium": {}
+ },
+ "1": {
+ "graph": {
+ "mode": "table",
+ "height": 86.0,
+ "optionOpen": false
+ }
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705462_1478852202",
+ "id": "paragraph_1546571485092_-716357716",
+ "dateCreated": "2019-09-26 17:55:05.462",
+ "dateStarted": "2019-09-26 17:56:45.502",
+ "dateFinished": "2019-09-26 17:57:24.042",
+ "status": "ABORT"
+ },
+ {
+ "title": "Page View Per Page for each 5 Seconds",
+ "text": "%flink.ssql(type\u003dts, parallelism\u003d1, refreshInterval\u003d2000, enableSavePoint\u003dfalse, runWithSavePoint\u003dfalse, threshold\u003d60000)\n\nselect TUMBLE_START(rowtime, INTERVAL \u00275\u0027 SECOND) as start_time, url, count(1) as pv from log group by TUMBLE(rowtime, INTERVAL \u00275\u0027 SECOND), url",
+ "user": "anonymous",
+ "dateUpdated": "2019-10-11 10:38:03.669",
+ "config": {
+ "editorSetting": {
+ "language": "sql",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/sql",
+ "fontSize": 9.0,
+ "title": false,
+ "runOnSelectionChange": true,
+ "checkEmpty": true,
+ "results": {
+ "0": {
+ "graph": {
+ "mode": "lineChart",
+ "height": 300.0,
+ "optionOpen": false,
+ "setting": {
+ "table": {
+ "tableGridState": {},
+ "tableColumnTypeState": {
+ "names": {
+ "start_time": "string",
+ "url": "string",
+ "pv": "string"
+ },
+ "updated": false
+ },
+ "tableOptionSpecHash": "[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer [...]
+ "tableOptionValue": {
+ "useFilter": false,
+ "showPagination": false,
+ "showAggregationFooter": false
+ },
+ "updated": false,
+ "initialized": false
+ },
+ "lineChart": {
+ "rotate": {
+ "degree": "-45"
+ },
+ "xLabelStatus": "rotate"
+ }
+ },
+ "commonSetting": {},
+ "keys": [
+ {
+ "name": "start_time",
+ "index": 0.0,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [
+ {
+ "name": "url",
+ "index": 1.0,
+ "aggr": "sum"
+ }
+ ],
+ "values": [
+ {
+ "name": "pv",
+ "index": 2.0,
+ "aggr": "sum"
+ }
+ ]
+ },
+ "helium": {}
+ }
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "results": {
+ "code": "SUCCESS",
+ "msg": [
+ {
+ "type": "TABLE",
+ "data": "start_time\turl\tpv\n2018-01-01 00:00:00.0\thome\t9\n2018-01-01 00:00:00.0\tsearch\t17\n2018-01-01 00:00:00.0\tproduct\t24\n2018-01-01 00:00:05.0\tsearch\t17\n2018-01-01 00:00:05.0\thome\t8\n2018-01-01 00:00:05.0\tproduct\t25\n"
+ }
+ ]
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705462_-861601454",
+ "id": "paragraph_1546571542468_1571709353",
+ "dateCreated": "2019-09-26 17:55:05.462",
+ "status": "READY"
+ },
+ {
+ "text": "%flink.ssql\n",
+ "user": "anonymous",
+ "dateUpdated": "2019-09-26 17:55:05.462",
+ "config": {},
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "apps": [],
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1569491705462_-1279435256",
+ "id": "paragraph_1546571603746_-749250139",
+ "dateCreated": "2019-09-26 17:55:05.462",
+ "status": "READY"
+ }
+ ],
+ "name": "Flink Stream Tutorial",
+ "id": "2ER62Y5VJ",
+ "defaultInterpreterGroup": "spark",
+ "version": "0.9.0-SNAPSHOT",
+ "permissions": {},
+ "noteParams": {},
+ "noteForms": {},
+ "angularObjects": {},
+ "config": {
+ "isZeppelinNotebookCronEnable": false
+ },
+ "info": {},
+ "path": "/Flink Stream Tutorial"
+}
\ No newline at end of file