You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by zj...@apache.org on 2021/02/08 11:49:22 UTC
[zeppelin] branch master updated: [ZEPPELIN-5220]. Update flink
turtorial note
This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new f5eb4ce [ZEPPELIN-5220]. Update flink turtorial note
f5eb4ce is described below
commit f5eb4cec32e54ba40da492483fc5d9afbd376405
Author: Jeff Zhang <zj...@apache.org>
AuthorDate: Mon Jan 25 14:32:51 2021 +0800
[ZEPPELIN-5220]. Update flink turtorial note
### What is this PR for?
Some update about flink tutorial note.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5220
### How should this be tested?
* No test
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zj...@apache.org>
Closes #4033 from zjffdu/ZEPPELIN-5220 and squashes the following commits:
ff3bed1c3 [Jeff Zhang] [ZEPPELIN-5220]. Update flink turtorial note
---
.../Flink Tutorial/1. Flink Basics_2F2YS7PCE.zpln | 53 ++------
...ial Steps for Building Flink Job_2F7SKEHPA.zpln | 77 +++--------
.../3. Flink Job Control Tutorial_2F5RKHCDV.zpln | 58 ++------
.../Flink Tutorial/4. Streaming ETL_2EYD56B9B.zpln | 44 ++----
.../5. Streaming Data Analytics_2EYT7Q6R8.zpln | 151 ++++++++++++++-------
.../Flink Tutorial/6. Batch ETL_2EW19CSPA.zpln | 139 ++++---------------
.../7. Batch Data Analytics_2EZ9G3JJU.zpln | 92 ++-----------
.../8. Logistic Regression (Alink)_2F4HJNWVN.zpln | 71 ++--------
8 files changed, 205 insertions(+), 480 deletions(-)
diff --git a/notebook/Flink Tutorial/1. Flink Basics_2F2YS7PCE.zpln b/notebook/Flink Tutorial/1. Flink Basics_2F2YS7PCE.zpln
index dec2c63..908d37e 100644
--- a/notebook/Flink Tutorial/1. Flink Basics_2F2YS7PCE.zpln
+++ b/notebook/Flink Tutorial/1. Flink Basics_2F2YS7PCE.zpln
@@ -2,9 +2,10 @@
"paragraphs": [
{
"title": "Introduction",
- "text": "%md\n\n# Introduction\n\n[Apache Flink](https://flink.apache.org/) is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. This is Flink tutorial for running classical wordcount in both batch and streaming mode. \n\nThere\u0027re 3 things you need to do before using flink in Zeppelin.\n\n* Download [Flink 1.10](https://flink.apache.org/downloads.html) for scala 2.11 (Only scala-2.11 is supported, scala-2.12 is not [...]
+ "text": "%md\n\n# Introduction\n\n[Apache Flink](https://flink.apache.org/) is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. This is Flink tutorial for running classical wordcount in both batch and streaming mode. \n\nThere\u0027re 3 things you need to do before using flink in Zeppelin.\n\n* Download [Flink 1.10](https://flink.apache.org/downloads.html) (Only 1.10 afterwards are supported) for scala 2.11 (Only scal [...]
"user": "anonymous",
- "dateUpdated": "2020-07-14 22:20:39.929",
+ "dateUpdated": "2021-01-25 14:14:41.500",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -25,23 +26,14 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003ch1\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003e\u003ca href\u003d\"https://flink.apache.org/\"\u003eApache Flink\u003c/a\u003e is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. This is Flink tutorial for running classical wordcount in both batch and streaming mode.\u003c/p\u003e\n\u003cp\u003eThere\u0026rsquo;re 3 things you need to do before using [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1580997898536_-1239502599",
"id": "paragraph_1580997898536_-1239502599",
"dateCreated": "2020-02-06 22:04:58.536",
- "dateStarted": "2020-07-14 22:20:39.929",
- "dateFinished": "2020-07-14 22:20:39.954",
+ "dateStarted": "2021-01-25 14:14:41.500",
+ "dateFinished": "2021-01-25 14:14:41.515",
"status": "FINISHED"
},
{
@@ -49,6 +41,7 @@
"text": "%flink\n\nval data \u003d benv.fromElements(\"hello world\", \"hello flink\", \"hello hadoop\")\ndata.flatMap(line \u003d\u003e line.split(\"\\\\s\"))\n .map(w \u003d\u003e (w, 1))\n .groupBy(0)\n .sum(1)\n .print()\n",
"user": "anonymous",
"dateUpdated": "2020-04-29 10:57:40.471",
+ "progress": 0,
"config": {
"colWidth": 6.0,
"fontSize": 9.0,
@@ -67,15 +60,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "\u001b[1m\u001b[34mdata\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.api.scala.DataSet[String]\u001b[0m \u003d org.apache.flink.api.scala.DataSet@177908f2\n(flink,1)\n(hadoop,1)\n(hello,3)\n(world,1)\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -91,6 +75,7 @@
"text": "%flink\n\nval data \u003d senv.fromElements(\"hello world\", \"hello flink\", \"hello hadoop\")\ndata.flatMap(line \u003d\u003e line.split(\"\\\\s\"))\n .map(w \u003d\u003e (w, 1))\n .keyBy(0)\n .sum(1)\n .print\n\nsenv.execute()",
"user": "anonymous",
"dateUpdated": "2020-04-29 10:58:47.117",
+ "progress": 0,
"config": {
"colWidth": 6.0,
"fontSize": 9.0,
@@ -109,15 +94,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "\u001b[1m\u001b[34mdata\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.streaming.api.scala.DataStream[String]\u001b[0m \u003d org.apache.flink.streaming.api.scala.DataStream@614839e8\n\u001b[1m\u001b[34mres2\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.streaming.api.datastream.DataStreamSink[(String, Int)]\u001b[0m \u003d org.apache.flink.streaming.api.datastream.DataStreamSink@1ead6506\n(hello,1)\n(world,1)\n(hello,2)\n(flink,1)\n(hello,3)\n(hadoop,1)\n\u001b[1m\u00 [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -133,6 +109,7 @@
"text": "%flink\n\nval data \u003d benv.readTextFile(\"hdfs:///tmp/bank.csv\")\ndata.print()\n",
"user": "anonymous",
"dateUpdated": "2020-02-25 11:11:54.292",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -151,19 +128,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "\u001b[1m\u001b[34mdata\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.api.scala.DataSet[String]\u001b[0m \u003d org.apache.flink.api.scala.DataSet@4721b6ff\n\"age\";\"job\";\"marital\";\"education\";\"default\";\"balance\";\"housing\";\"loan\";\"contact\";\"day\";\"month\";\"duration\";\"campaign\";\"pdays\";\"previous\";\"poutcome\";\"y\"\n30;\"unemployed\";\"married\";\"primary\";\"no\";1787;\"no\";\"no\";\"cellular\";19;\"oct\";79;1;-1;0;\"unknown\";\"no\"\n33;\"s [...]
- },
- {
- "type": "HTML",
- "data": "\u003cdiv class\u003d\"result-alert alert-warning\" role\u003d\"alert\"\u003e\u003cbutton type\u003d\"button\" class\u003d\"close\" data-dismiss\u003d\"alert\" aria-label\u003d\"Close\"\u003e\u003cspan aria-hidden\u003d\"true\"\u003e\u0026times;\u003c/span\u003e\u003c/button\u003e\u003cstrong\u003eOutput is truncated\u003c/strong\u003e to 102400 bytes. Learn more about \u003cstrong\u003eZEPPELIN_INTERPRETER_OUTPUT_LIMIT\u003c/strong\u003e\u003c/div\u003e"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -178,6 +142,7 @@
"text": "%flink\n",
"user": "anonymous",
"dateUpdated": "2020-02-25 11:10:14.096",
+ "progress": 0,
"config": {},
"settings": {
"params": {},
diff --git a/notebook/Flink Tutorial/2. Three Essential Steps for Building Flink Job_2F7SKEHPA.zpln b/notebook/Flink Tutorial/2. Three Essential Steps for Building Flink Job_2F7SKEHPA.zpln
index aec2347..903f696 100644
--- a/notebook/Flink Tutorial/2. Three Essential Steps for Building Flink Job_2F7SKEHPA.zpln
+++ b/notebook/Flink Tutorial/2. Three Essential Steps for Building Flink Job_2F7SKEHPA.zpln
@@ -4,7 +4,8 @@
"title": "Introduction",
"text": "%md\n\n# Introduction\n\n\nTypically there\u0027re 3 essential steps for building one flink job. And each step has its favorite tools.\n\n* Define source/sink (SQL DDL)\n* Define data flow (Table Api / SQL)\n* Implement business logic (UDF)\n\nThis tutorial demonstrates how to build one typical flinkjob via these 3 steps and their favorite tools.\nIn this demo, we will do real time analysis of cdn access data. First we read cdn access log from kafka queue and do some proc [...]
"user": "anonymous",
- "dateUpdated": "2020-07-14 22:22:38.673",
+ "dateUpdated": "2021-01-25 14:25:27.359",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -25,23 +26,14 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003ch1\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eTypically there\u0026rsquo;re 3 essential steps for building one flink job. And each step has its favorite tools.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDefine source/sink (SQL DDL)\u003c/li\u003e\n\u003cli\u003eDefine data flow (Table Api / SQL)\u003c/li\u003e\n\u003cli\u003eImplement business logic (UDF)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis tuto [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1587965294481_785664297",
"id": "paragraph_1587965294481_785664297",
"dateCreated": "2020-04-27 13:28:14.481",
- "dateStarted": "2020-07-14 22:22:38.673",
- "dateFinished": "2020-07-14 22:22:38.688",
+ "dateStarted": "2021-01-25 14:25:27.360",
+ "dateFinished": "2021-01-25 14:25:30.018",
"status": "FINISHED"
},
{
@@ -49,6 +41,7 @@
"text": "%flink.conf\n\n# This example use kafka as source and mysql as sink, so you need to specify flink kafka connector and flink jdbc connector first.\n\nflink.execution.packages\torg.apache.flink:flink-connector-kafka_2.11:1.10.0,org.apache.flink:flink-connector-kafka-base_2.11:1.10.0,org.apache.flink:flink-json:1.10.0,org.apache.flink:flink-jdbc_2.11:1.10.0,mysql:mysql-connector-java:5.1.38\n\n# Set taskmanager.memory.segment-size to be the smallest value just for this demo, [...]
"user": "anonymous",
"dateUpdated": "2020-04-29 15:07:01.902",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -67,10 +60,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": []
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -86,6 +75,7 @@
"text": "%flink.ssql\n\nDROP table if exists cdn_access_log;\n\nCREATE TABLE cdn_access_log (\n uuid VARCHAR,\n client_ip VARCHAR,\n request_time BIGINT,\n response_size BIGINT,\n uri VARCHAR,\n event_ts BIGINT\n) WITH (\n \u0027connector.type\u0027 \u003d \u0027kafka\u0027,\n \u0027connector.version\u0027 \u003d \u0027universal\u0027,\n \u0027connector.topic\u0027 \u003d \u0027cdn_events\u0027,\n \u0027connector.properties.zookeeper.connect\u0027 \u003d \u0027localhost:2181\u0027, [...]
"user": "anonymous",
"dateUpdated": "2020-04-29 15:07:03.989",
+ "progress": 0,
"config": {
"colWidth": 6.0,
"fontSize": 9.0,
@@ -104,15 +94,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "Table has been dropped.\nTable has been created.\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -128,6 +109,7 @@
"text": "%flink.ssql\n\nDROP table if exists cdn_access_statistic;\n\n-- Please create this mysql table first in your mysql instance. Flink won\u0027t create mysql table for you.\n\nCREATE TABLE cdn_access_statistic (\n province VARCHAR,\n access_count BIGINT,\n total_download BIGINT,\n download_speed DOUBLE\n) WITH (\n \u0027connector.type\u0027 \u003d \u0027jdbc\u0027,\n \u0027connector.url\u0027 \u003d \u0027jdbc:mysql://localhost:3306/flink_cdn\u0027,\n \u0027connector.table\u0 [...]
"user": "anonymous",
"dateUpdated": "2020-04-29 15:07:05.522",
+ "progress": 0,
"config": {
"colWidth": 6.0,
"fontSize": 9.0,
@@ -146,15 +128,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "Table has been dropped.\nTable has been created.\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -170,6 +143,7 @@
"text": "%flink.ipyflink\n\nimport re\nimport json\nfrom pyflink.table import DataTypes\nfrom pyflink.table.udf import udf\nfrom urllib.parse import quote_plus\nfrom urllib.request import urlopen\n\n# This UDF is to convert ip address to country. Here we simply the logic just for demo purpose.\n\n@udf(input_types\u003d[DataTypes.STRING()], result_type\u003dDataTypes.STRING())\ndef ip_to_country(ip):\n\n countries \u003d [\u0027USA\u0027, \u0027China\u0027, \u0027Japan\u0027, \u0 [...]
"user": "anonymous",
"dateUpdated": "2020-04-29 15:07:24.842",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -188,10 +162,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": []
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -207,6 +177,7 @@
"text": "%flink.bsql\n\nselect ip_to_country(\u00272.10.01.1\u0027)",
"user": "anonymous",
"dateUpdated": "2020-04-29 14:17:35.942",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -253,15 +224,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "EXPR$0\nChina\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -277,6 +239,7 @@
"text": "%flink.ipyflink\n\nt \u003d st_env.from_path(\"cdn_access_log\")\\\n .select(\"uuid, \"\n \"ip_to_country(client_ip) as country, \" \n \"response_size, request_time\")\\\n .group_by(\"country\")\\\n .select(\n \"country, count(uuid) as access_count, \" \n \"sum(response_size) as total_download, \" \n \"sum(response_size) * 1.0 / sum(request_time) as download_speed\")\n #.insert_into(\"cdn_access_statistic\")\n\n# z.show [...]
"user": "anonymous",
"dateUpdated": "2020-04-29 14:19:00.530",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -367,6 +330,7 @@
"text": "%flink\n\nval t \u003d stenv.from(\"cdn_access_log\")\n .select(\"uuid, ip_to_country(client_ip) as country, response_size, request_time\")\n .groupBy(\"country\")\n .select( \"country, count(uuid) as access_count, sum(response_size) as total_download, sum(response_size) * 1.0 / sum(request_time) as download_speed\")\n //.insertInto(\"cdn_access_statistic\")\n\n// z.show will display the result in zeppelin front end in table format, you can uncomment the above ins [...]
"user": "anonymous",
"dateUpdated": "2020-04-29 15:14:29.646",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -456,6 +420,7 @@
"text": "%flink.ssql\n\ninsert into cdn_access_statistic\nselect ip_to_country(client_ip) as country, \n count(uuid) as access_count, \n sum(response_size) as total_download ,\n sum(response_size) * 1.0 / sum(request_time) as download_speed\nfrom cdn_access_log\n group by ip_to_country(client_ip)\n",
"user": "anonymous",
"dateUpdated": "2020-04-29 15:12:26.431",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -489,7 +454,8 @@
{
"text": "%md\n\n# Query sink table via jdbc interpreter\n\nYou can also query the sink table (mysql) directly via jdbc interpreter. Here I will create a jdbc interpreter named `mysql` and use it to query the sink table. Regarding how to connect mysql in Zeppelin, refer this [link](http://zeppelin.apache.org/docs/0.9.0-preview1/interpreter/jdbc.html#mysql)",
"user": "anonymous",
- "dateUpdated": "2020-07-14 22:23:28.657",
+ "dateUpdated": "2021-01-25 14:25:41.163",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -509,23 +475,14 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003ch1\u003eQuery sink table via jdbc interpreter\u003c/h1\u003e\n\u003cp\u003eYou can also query the sink table (mysql) directly via jdbc interpreter. Here I will create a jdbc interpreter named \u003ccode\u003emysql\u003c/code\u003e and use it to query the sink table. Regarding how to connect mysql in Zeppelin, refer this \u003ca href\u003d\"http://zeppelin.apache.org/docs/0.9.0-preview1/interpreter/jdbc.html#mysql\"\ [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1587976725546_2073084823",
"id": "paragraph_1587976725546_2073084823",
"dateCreated": "2020-04-27 16:38:45.548",
- "dateStarted": "2020-07-14 22:23:28.657",
- "dateFinished": "2020-07-14 22:23:28.670",
+ "dateStarted": "2021-01-25 14:25:41.162",
+ "dateFinished": "2021-01-25 14:25:41.175",
"status": "FINISHED"
},
{
@@ -533,6 +490,7 @@
"text": "%mysql\n\nselect * from flink_cdn.cdn_access_statistic",
"user": "anonymous",
"dateUpdated": "2020-04-29 15:22:13.333",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -630,6 +588,7 @@
"text": "%flink.ipyflink\n",
"user": "anonymous",
"dateUpdated": "2020-04-27 16:38:45.464",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
diff --git a/notebook/Flink Tutorial/3. Flink Job Control Tutorial_2F5RKHCDV.zpln b/notebook/Flink Tutorial/3. Flink Job Control Tutorial_2F5RKHCDV.zpln
index 9a70026..e533972 100644
--- a/notebook/Flink Tutorial/3. Flink Job Control Tutorial_2F5RKHCDV.zpln
+++ b/notebook/Flink Tutorial/3. Flink Job Control Tutorial_2F5RKHCDV.zpln
@@ -4,7 +4,8 @@
"title": "Introduction",
"text": "%md\n\n# Introduction\n\nThis tutorial is to demonstrate how to do job control in flink (job submission/cancel/resume).\n2 steps:\n1. Create custom data stream and register it as flink table. The custom data stream is a simulated web access logs. \n2. Query this flink table (pv for each page type), you can cancel it and then resume it again w/o savepoint.\n",
"user": "anonymous",
- "dateUpdated": "2020-07-14 22:24:11.629",
+ "dateUpdated": "2021-01-25 14:25:48.903",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -25,23 +26,14 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003ch1\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eThis tutorial is to demonstrate how to do job control in flink (job submission/cancel/resume).\u003cbr /\u003e\n2 steps:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eCreate custom data stream and register it as flink table. The custom data stream is a simulated web access logs.\u003c/li\u003e\n\u003cli\u003eQuery this flink table (pv for each page type), you can canc [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1587964310955_443124874",
"id": "paragraph_1587964310955_443124874",
"dateCreated": "2020-04-27 13:11:50.955",
- "dateStarted": "2020-07-14 22:24:11.630",
- "dateFinished": "2020-07-14 22:24:11.642",
+ "dateStarted": "2021-01-25 14:25:48.903",
+ "dateFinished": "2021-01-25 14:25:48.916",
"status": "FINISHED"
},
{
@@ -49,6 +41,7 @@
"text": "%flink \n\nimport org.apache.flink.streaming.api.functions.source.SourceFunction\nimport org.apache.flink.table.api.TableEnvironment\nimport org.apache.flink.streaming.api.TimeCharacteristic\nimport org.apache.flink.streaming.api.checkpoint.ListCheckpointed\nimport java.util.Collections\nimport scala.collection.JavaConversions._\n\nsenv.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)\nsenv.enableCheckpointing(1000)\n\nval data \u003d senv.addSource(new SourceFunc [...]
"user": "anonymous",
"dateUpdated": "2020-04-29 14:21:46.802",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -67,15 +60,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "import org.apache.flink.streaming.api.functions.source.SourceFunction\nimport org.apache.flink.table.api.TableEnvironment\nimport org.apache.flink.streaming.api.TimeCharacteristic\nimport org.apache.flink.streaming.api.checkpoint.ListCheckpointed\nimport java.util.Collections\nimport scala.collection.JavaConversions._\n\u001b[1m\u001b[34mres1\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.streaming.api.scala.StreamExecutionEnvironment\u001b[0m \u003d org.apache.flink. [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -91,6 +75,7 @@
"text": "%flink.ssql(type\u003dupdate)\n\nselect url, count(1) as c from log group by url",
"user": "anonymous",
"dateUpdated": "2020-04-29 14:22:40.594",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -176,6 +161,7 @@
"text": "%flink.ssql(type\u003dupdate,parallelism\u003d2,maxParallelism\u003d10,savepointDir\u003d/tmp/flink_a)\n\nselect url, count(1) as pv from log group by url",
"user": "anonymous",
"dateUpdated": "2020-04-29 14:24:26.768",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -229,15 +215,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "url\tpv\nhome\t500\nproduct\t1500\nsearch\t1000\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -253,6 +230,7 @@
"text": "%flink(parallelism\u003d1,maxParallelism\u003d10,savepointDir\u003d/tmp/flink_b)\n\nval table \u003d stenv.sqlQuery(\"select url, count(1) as pv from log group by url\")\n\nz.show(table, streamType\u003d\"update\")\n",
"user": "anonymous",
"dateUpdated": "2020-04-15 16:11:54.114",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -304,15 +282,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "url\tpv\nhome\t34\nproduct\t100\nsearch\t68\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -328,6 +297,7 @@
"text": "%flink.ipyflink(parallelism\u003d1,maxParallelism\u003d10,savepointDir\u003d/tmp/flink_c)\n\ntable \u003d st_env.sql_query(\"select url, count(1) as pv from log group by url\")\n\nz.show(table, stream_type\u003d\"update\")",
"user": "anonymous",
"dateUpdated": "2020-04-27 14:45:28.045",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -378,15 +348,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "url\tpv\nhome\t17\nproduct\t49\nsearch\t34\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -401,6 +362,7 @@
"text": "%flink.ssql\n",
"user": "anonymous",
"dateUpdated": "2020-04-18 19:04:17.969",
+ "progress": 0,
"config": {},
"settings": {
"params": {},
diff --git a/notebook/Flink Tutorial/4. Streaming ETL_2EYD56B9B.zpln b/notebook/Flink Tutorial/4. Streaming ETL_2EYD56B9B.zpln
index 11501b4..9e8b92f 100644
--- a/notebook/Flink Tutorial/4. Streaming ETL_2EYD56B9B.zpln
+++ b/notebook/Flink Tutorial/4. Streaming ETL_2EYD56B9B.zpln
@@ -4,7 +4,8 @@
"title": "Overview",
"text": "%md\n\nThis tutorial demonstrate how to use Flink do streaming processing via its streaming sql + udf. In this tutorial, we read data from kafka queue and do some simple processing (just filtering here) and then write it back to another kafka queue. We use this [docker](https://zeppelin-kafka-connect-datagen.readthedocs.io/en/latest/) to create kafka cluster and source data \n\n* Make sure you add the following ip host name mapping to your hosts file, otherwise you may not [...]
"user": "anonymous",
- "dateUpdated": "2020-04-27 13:44:26.806",
+ "dateUpdated": "2021-01-25 14:26:04.106",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -27,23 +28,14 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis tutorial demonstrate how to use Flink do streaming processing via its streaming sql + udf. In this tutorial, we read data from kafka queue and do some simple processing (just filtering here) and then write it back to another kafka queue. We use this \u003ca href\u003d\"https://zeppelin-kafka-connect-datagen.readthedocs.io/en/latest/\"\u003edocker\u003c/a\u003e to create kafka cluster and source data\u003 [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1579054287919_-61477360",
"id": "paragraph_1579054287919_-61477360",
"dateCreated": "2020-01-15 10:11:27.919",
- "dateStarted": "2020-04-27 11:49:46.116",
- "dateFinished": "2020-04-27 11:49:46.128",
+ "dateStarted": "2021-01-25 14:26:04.106",
+ "dateFinished": "2021-01-25 14:26:04.118",
"status": "FINISHED"
},
{
@@ -51,6 +43,7 @@
"text": "%flink.conf\n\n# You need to run this paragraph first before running any flink code.\n\nflink.execution.packages\torg.apache.flink:flink-connector-kafka_2.11:1.10.0,org.apache.flink:flink-connector-kafka-base_2.11:1.10.0,org.apache.flink:flink-json:1.10.0",
"user": "anonymous",
"dateUpdated": "2020-04-29 15:45:27.361",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -69,10 +62,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": []
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -88,6 +77,7 @@
"text": "%flink.ssql\n\nDROP TABLE IF EXISTS source_kafka;\n\nCREATE TABLE source_kafka (\n status STRING,\n direction STRING,\n event_ts BIGINT\n) WITH (\n \u0027connector.type\u0027 \u003d \u0027kafka\u0027, \n \u0027connector.version\u0027 \u003d \u0027universal\u0027,\n \u0027connector.topic\u0027 \u003d \u0027generated.events\u0027,\n \u0027connector.startup-mode\u0027 \u003d \u0027earliest-offset\u0027,\n \u0027connector.properties.zookeeper.connect\u0027 [...]
"user": "anonymous",
"dateUpdated": "2020-04-29 15:45:29.234",
+ "progress": 0,
"config": {
"colWidth": 6.0,
"fontSize": 9.0,
@@ -108,15 +98,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "Table has been dropped.\nTable has been created.\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -132,6 +113,7 @@
"text": "%flink.ssql\n\nDROP TABLE IF EXISTS sink_kafka;\n\nCREATE TABLE sink_kafka (\n status STRING,\n direction STRING,\n event_ts TIMESTAMP(3),\n WATERMARK FOR event_ts AS event_ts - INTERVAL \u00275\u0027 SECOND\n) WITH (\n \u0027connector.type\u0027 \u003d \u0027kafka\u0027, \n \u0027connector.version\u0027 \u003d \u0027universal\u0027, \n \u0027connector.topic\u0027 \u003d \u0027generated.events2\u0027,\n \u0027connector.properties.zookeeper.connect [...]
"user": "anonymous",
"dateUpdated": "2020-04-29 15:45:30.663",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -152,15 +134,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "Table has been dropped.\nTable has been created.\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -176,6 +149,7 @@
"text": "%flink.ssql\n\ninsert into sink_kafka select status, direction, cast(event_ts/1000000000 as timestamp(3)) from source_kafka where status \u003c\u003e \u0027foo\u0027\n",
"user": "anonymous",
"dateUpdated": "2020-04-29 15:45:43.388",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -212,6 +186,7 @@
"text": "%flink.ssql(type\u003dupdate)\n\nselect * from sink_kafka order by event_ts desc limit 10;",
"user": "anonymous",
"dateUpdated": "2020-04-29 15:28:01.122",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -325,6 +300,7 @@
"text": "%flink.ssql\n",
"user": "anonymous",
"dateUpdated": "2020-04-29 15:27:31.430",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
diff --git a/notebook/Flink Tutorial/5. Streaming Data Analytics_2EYT7Q6R8.zpln b/notebook/Flink Tutorial/5. Streaming Data Analytics_2EYT7Q6R8.zpln
index 5fcbaf9..6d94d14 100644
--- a/notebook/Flink Tutorial/5. Streaming Data Analytics_2EYT7Q6R8.zpln
+++ b/notebook/Flink Tutorial/5. Streaming Data Analytics_2EYT7Q6R8.zpln
@@ -4,7 +4,8 @@
"title": "Overview",
"text": "%md\n\nThis tutorial demonstrate how to use Flink do streaming analytics via its streaming sql + udf. Zeppelin now support 3 kinds of streaming visualization.\n\n* Single - Single mode is for the case when the result of sql statement is always one row, such as the following example\n* Update - Update mode is suitable for the case when the output is more than one rows, and always will be updated continuously. \n* Append - Append mode is suitable for the scenario where ou [...]
"user": "anonymous",
- "dateUpdated": "2020-04-29 15:31:49.376",
+ "dateUpdated": "2021-01-25 14:26:13.376",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -27,30 +28,54 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis tutorial demonstrate how to use Flink do streaming analytics via its streaming sql + udf. Zeppelin now support 3 kinds of streaming visualization.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSingle - Single mode is for the case when the result of sql statement is always one row, such as the following example\u003c/li\u003e\n\u003cli\u003eUpdate - Update mode is suitable for the case when the output is [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1579054784565_2122156822",
"id": "paragraph_1579054784565_2122156822",
"dateCreated": "2020-01-15 10:19:44.565",
- "dateStarted": "2020-04-29 15:31:49.379",
- "dateFinished": "2020-04-29 15:31:49.549",
+ "dateStarted": "2021-01-25 14:26:13.376",
+ "dateFinished": "2021-01-25 14:26:13.394",
+ "status": "FINISHED"
+ },
+ {
+ "text": "%flink \n\nimport org.apache.flink.streaming.api.functions.source.SourceFunction\nimport org.apache.flink.table.api.TableEnvironment\nimport org.apache.flink.streaming.api.TimeCharacteristic\nimport org.apache.flink.streaming.api.checkpoint.ListCheckpointed\nimport java.util.Collections\nimport scala.collection.JavaConversions._\n\nsenv.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)\nsenv.enableCheckpointing(1000)\n\nval data \u003d senv.addSource(new SourceFunc [...]
+ "user": "anonymous",
+ "dateUpdated": "2021-01-25 14:27:02.042",
+ "progress": 0,
+ "config": {
+ "editorSetting": {
+ "language": "scala",
+ "editOnDblClick": false,
+ "completionKey": "TAB",
+ "completionSupport": true
+ },
+ "colWidth": 12.0,
+ "editorMode": "ace/mode/scala",
+ "fontSize": 9.0,
+ "results": {},
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "apps": [],
+ "runtimeInfos": {},
+ "progressUpdateIntervalMs": 500,
+ "jobName": "paragraph_1611556011274_1848600588",
+ "id": "paragraph_1611556011274_1848600588",
+ "dateCreated": "2021-01-25 14:26:51.274",
+ "dateStarted": "2021-01-25 14:27:02.045",
+ "dateFinished": "2021-01-25 14:27:36.849",
"status": "FINISHED"
},
{
"title": "Single row mode of Output",
- "text": "%flink.ssql(type\u003dsingle, parallelism\u003d1, refreshInterval\u003d3000, template\u003d\u003ch1\u003e{1}\u003c/h1\u003e until \u003ch2\u003e{0}\u003c/h2\u003e)\n\nselect max(event_ts), count(1) from sink_kafka\n",
+ "text": "%flink.ssql(type\u003dsingle,parallelism\u003d1,refreshInterval\u003d1000,template\u003d\u003ch1\u003e{1}\u003c/h1\u003e until \u003ch2\u003e{0}\u003c/h2\u003e)\n\nselect max(rowtime),count(1) from log\n",
"user": "anonymous",
- "dateUpdated": "2020-04-29 15:45:47.116",
+ "dateUpdated": "2021-01-25 14:28:01.427",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -69,27 +94,42 @@
"template": "\u003ch1\u003e{1}\u003c/h1\u003e until \u003ch2\u003e{0}\u003c/h2\u003e",
"refreshInterval": "3000",
"parallelism": "1",
- "type": "single"
+ "type": "single",
+ "latest_checkpoint_path": "\u003ccheckpoint-not-externally-addressable\u003e"
},
"settings": {
"params": {},
"forms": {}
},
"apps": [],
- "runtimeInfos": {},
+ "runtimeInfos": {
+ "jobUrl": {
+ "propertyName": "jobUrl",
+ "label": "FLINK JOB",
+ "tooltip": "View in Flink web UI",
+ "group": "flink",
+ "values": [
+ {
+ "jobUrl": "http://localhost:8081#/job/1086e2a18ab8e8402b8969412c9b0e8e"
+ }
+ ],
+ "interpreterSettingId": "flink"
+ }
+ },
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1578909960516_-1812187661",
"id": "paragraph_1578909960516_-1812187661",
"dateCreated": "2020-01-13 18:06:00.516",
- "dateStarted": "2020-04-29 15:45:47.127",
- "dateFinished": "2020-04-29 15:47:14.738",
+ "dateStarted": "2021-01-25 14:28:01.430",
+ "dateFinished": "2021-01-25 14:30:03.648",
"status": "ABORT"
},
{
"title": "Update mode of Output",
- "text": "%flink.ssql(type\u003dupdate, refreshInterval\u003d2000, parallelism\u003d1)\n\nselect status, count(1) as pv from sink_kafka group by status",
+ "text": "%flink.ssql(type\u003dupdate,parallelism\u003d1,refreshInterval\u003d2000)\n\nselect url,count(1) as pv from log group by url\n",
"user": "anonymous",
- "dateUpdated": "2020-04-29 15:46:09.485",
+ "dateUpdated": "2021-01-25 14:28:48.633",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -114,7 +154,7 @@
"commonSetting": {},
"keys": [
{
- "name": "status",
+ "name": "url",
"index": 0.0,
"aggr": "sum"
}
@@ -142,27 +182,42 @@
"parallelism": "1",
"type": "update",
"savepointDir": "/tmp/flink_2",
- "editorHide": false
+ "editorHide": false,
+ "latest_checkpoint_path": "\u003ccheckpoint-not-externally-addressable\u003e"
},
"settings": {
"params": {},
"forms": {}
},
"apps": [],
- "runtimeInfos": {},
+ "runtimeInfos": {
+ "jobUrl": {
+ "propertyName": "jobUrl",
+ "label": "FLINK JOB",
+ "tooltip": "View in Flink web UI",
+ "group": "flink",
+ "values": [
+ {
+ "jobUrl": "http://localhost:8081#/job/026f68085b98b16a9a677c2b1ba47bb3"
+ }
+ ],
+ "interpreterSettingId": "flink"
+ }
+ },
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1578910004762_-286113604",
"id": "paragraph_1578910004762_-286113604",
"dateCreated": "2020-01-13 18:06:44.762",
- "dateStarted": "2020-04-29 15:46:09.492",
- "dateFinished": "2020-04-29 15:47:13.312",
+ "dateStarted": "2021-01-25 14:28:48.637",
+ "dateFinished": "2021-01-25 14:30:01.879",
"status": "ABORT"
},
{
"title": "Append mode of Output",
- "text": "%flink.ssql(type\u003dappend, parallelism\u003d1, refreshInterval\u003d2000, threshold\u003d60000)\n\nselect TUMBLE_START(event_ts, INTERVAL \u00275\u0027 SECOND) as start_time, status, count(1) from sink_kafka\ngroup by TUMBLE(event_ts, INTERVAL \u00275\u0027 SECOND), status\n",
+ "text": "%flink.ssql(type\u003dappend,parallelism\u003d1,refreshInterval\u003d2000,threshold\u003d60000)\n\nselect TUMBLE_START(rowtime,INTERVAL \u00275\u0027 SECOND) start_time,url,count(1) as pv from log\ngroup by TUMBLE(rowtime,INTERVAL \u00275\u0027 SECOND),url\n",
"user": "anonymous",
- "dateUpdated": "2020-04-29 15:46:40.174",
+ "dateUpdated": "2021-01-25 14:29:33.485",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -194,14 +249,14 @@
],
"groups": [
{
- "name": "status",
+ "name": "url",
"index": 1.0,
"aggr": "sum"
}
],
"values": [
{
- "name": "EXPR$2",
+ "name": "pv",
"index": 2.0,
"aggr": "sum"
}
@@ -221,39 +276,28 @@
"parallelism": "1",
"threshold": "60000",
"type": "append",
- "savepointDir": "/tmp/flink_3"
+ "savepointDir": "/tmp/flink_3",
+ "latest_checkpoint_path": "\u003ccheckpoint-not-externally-addressable\u003e"
},
"settings": {
"params": {},
"forms": {}
},
- "results": {
- "code": "ERROR",
- "msg": [
- {
- "type": "TABLE",
- "data": "start_time\tstatus\tEXPR$2\n2020-04-29 07:46:20.0\tbaz\t3\n2020-04-29 07:46:20.0\tbar\t3\n2020-04-29 07:46:25.0\tbaz\t2\n2020-04-29 07:46:25.0\tbar\t4\n2020-04-29 07:46:30.0\tbar\t4\n2020-04-29 07:46:30.0\tbaz\t2\n2020-04-29 07:46:35.0\tbar\t4\n2020-04-29 07:46:40.0\tbar\t5\n2020-04-29 07:46:40.0\tbaz\t2\n2020-04-29 07:46:45.0\tbar\t4\n2020-04-29 07:46:45.0\tbaz\t4\n2020-04-29 07:46:50.0\tbar\t4\n2020-04-29 07:46:50.0\tbaz\t3\n2020-04-29 07:46:55.0\tbaz\t3\n2020-04-2 [...]
- },
- {
- "type": "TEXT",
- "data": "Fail to run sql command: select TUMBLE_START(event_ts, INTERVAL \u00275\u0027 SECOND) as start_time, status, count(1) from sink_kafka\ngroup by TUMBLE(event_ts, INTERVAL \u00275\u0027 SECOND), status\njava.io.IOException: Fail to run stream sql job\n\tat org.apache.zeppelin.flink.sql.AbstractStreamSqlJob.run(AbstractStreamSqlJob.java:166)\n\tat org.apache.zeppelin.flink.sql.AbstractStreamSqlJob.run(AbstractStreamSqlJob.java:101)\n\tat org.apache.zeppelin.flink.FlinkS [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1578910016872_1942851900",
"id": "paragraph_1578910016872_1942851900",
"dateCreated": "2020-01-13 18:06:56.872",
- "dateStarted": "2020-04-29 15:46:18.958",
- "dateFinished": "2020-04-29 15:47:22.960",
+ "dateStarted": "2021-01-25 14:29:17.122",
+ "dateFinished": "2021-01-25 14:29:53.805",
"status": "ABORT"
},
{
"text": "%flink.ssql\n",
"user": "anonymous",
"dateUpdated": "2020-01-13 21:17:35.739",
+ "progress": 0,
"config": {},
"settings": {
"params": {},
@@ -274,7 +318,22 @@
"version": "0.9.0-SNAPSHOT",
"noteParams": {},
"noteForms": {},
- "angularObjects": {},
+ "angularObjects": {
+ "flink-shared_process": [
+ {
+ "name": "value_0",
+ "object": "2019-12-31 16:01:44.1",
+ "noteId": "2EYT7Q6R8",
+ "paragraphId": "paragraph_1578909960516_-1812187661"
+ },
+ {
+ "name": "value_1",
+ "object": "1042",
+ "noteId": "2EYT7Q6R8",
+ "paragraphId": "paragraph_1578909960516_-1812187661"
+ }
+ ]
+ },
"config": {
"isZeppelinNotebookCronEnable": false
},
diff --git a/notebook/Flink Tutorial/6. Batch ETL_2EW19CSPA.zpln b/notebook/Flink Tutorial/6. Batch ETL_2EW19CSPA.zpln
index 40c8cc2..93f11bb 100644
--- a/notebook/Flink Tutorial/6. Batch ETL_2EW19CSPA.zpln
+++ b/notebook/Flink Tutorial/6. Batch ETL_2EW19CSPA.zpln
@@ -4,7 +4,8 @@
"title": "Overview",
"text": "%md\n\nThis tutorial demonstrates how to use Flink do batch ETL via its batch sql + udf (scala, python \u0026 hive). Here\u0027s what we do in this tutorial\n\n* Download [bank](https://archive.ics.uci.edu/ml/datasets/bank+marketing) data via shell interpreter to local\n* Process the raw data via flink batch sql \u0026 scala udf which parse and clean the raw data\n* Write the structured and cleaned data to another flink table via sql\n",
"user": "anonymous",
- "dateUpdated": "2020-04-29 16:08:52.383",
+ "dateUpdated": "2021-01-25 14:26:19.720",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -27,22 +28,14 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis tutorial demonstrates how to use Flink do batch ETL via its batch sql + udf (scala, python \u0026amp; hive). Here\u0026rsquo;s what we do in this tutorial\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDownload \u003ca href\u003d\"https://archive.ics.uci.edu/ml/datasets/bank+marketing\"\u003ebank\u003c/a\u003e data via shell interpreter to local\u003c/li\u003e\n\u003cli\u003eProcess the raw data via flink [...]
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1579052523153_721650872",
"id": "paragraph_1579052523153_721650872",
"dateCreated": "2020-01-15 09:42:03.156",
- "dateStarted": "2020-04-27 11:41:03.630",
- "dateFinished": "2020-04-27 11:41:04.735",
+ "dateStarted": "2021-01-25 14:26:19.720",
+ "dateFinished": "2021-01-25 14:26:19.732",
"status": "FINISHED"
},
{
@@ -50,6 +43,7 @@
"text": "%sh\n\ncd /tmp\nrm -rf bank*\nwget https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank.zip\nunzip bank.zip\n# upload data to hdfs if you want to run it in yarn mode\n# hadoop fs -put /tmp/bank.csv /tmp/bank.csv\n",
"user": "anonymous",
"dateUpdated": "2020-05-05 22:52:51.576",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -70,16 +64,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "--2020-05-05 22:52:51-- https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank.zip\nResolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252\nConnecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 579043 (565K) [application/x-httpd-php]\nSaving to: ‘bank.zip’\n\n 0K .......... .......... .......... .......... .......... 8% 105K 5s\n 50K ... [...]
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1578045094400_1030344935",
"id": "paragraph_1578045094400_1030344935",
@@ -93,6 +79,7 @@
"text": "%sh\n\nhead -n 10 /tmp/bank.csv",
"user": "anonymous",
"dateUpdated": "2020-04-27 11:37:44.952",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -113,16 +100,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "\"age\";\"job\";\"marital\";\"education\";\"default\";\"balance\";\"housing\";\"loan\";\"contact\";\"day\";\"month\";\"duration\";\"campaign\";\"pdays\";\"previous\";\"poutcome\";\"y\"\n30;\"unemployed\";\"married\";\"primary\";\"no\";1787;\"no\";\"no\";\"cellular\";19;\"oct\";79;1;-1;0;\"unknown\";\"no\"\n33;\"services\";\"married\";\"secondary\";\"no\";4789;\"yes\";\"yes\";\"cellular\";11;\"may\";220;1;339;4;\"failure\";\"no\"\n35;\"management\";\"single\";\"tertia [...]
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1579053112778_2010129053",
"id": "paragraph_1579053112778_2010129053",
@@ -136,6 +115,7 @@
"text": "%flink.bsql\n\nDROP TABLE IF EXISTS bank_raw;\nCREATE TABLE bank_raw (\n content STRING\n) WITH (\n\u0027format.field-delimiter\u0027\u003d\u0027\\n\u0027,\n\u0027connector.type\u0027\u003d\u0027filesystem\u0027,\n\u0027format.derive-schema\u0027\u003d\u0027true\u0027,\n\u0027connector.path\u0027\u003d\u0027/tmp/bank.csv\u0027,\n\u0027format.type\u0027\u003d\u0027csv\u0027\n);",
"user": "anonymous",
"dateUpdated": "2020-05-05 23:01:30.843",
+ "progress": 0,
"config": {
"colWidth": 6.0,
"fontSize": 9.0,
@@ -156,16 +136,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "Table has been dropped.\nTable has been created.\n"
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1578044954921_-1188487356",
"id": "paragraph_1578044954921_-1188487356",
@@ -179,6 +151,7 @@
"text": "%flink.bsql\n\nDROP TABLE IF EXISTS bank;\nCREATE TABLE bank (\n age int, \n job string,\n marital string,\n education string,\n `default` string,\n balance string,\n housing string,\n loan string,\n contact string, \n `day` string,\n `month` string,\n duration int,\n campaign int,\n pdays int,\n previous int,\n poutcome string,\n y string\n) WITH (\n\u0027format.field-delimiter\u0027\u003d\u0027,\u0027,\n\u0027connector.t [...]
"user": "anonymous",
"dateUpdated": "2020-05-05 23:01:32.335",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -199,16 +172,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "Table has been dropped.\nTable has been created.\n"
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1578045204379_-1427374232",
"id": "paragraph_1578045204379_-1427374232",
@@ -222,6 +187,7 @@
"text": "%flink.bsql\n\nshow tables",
"user": "anonymous",
"dateUpdated": "2020-05-05 22:52:11.086",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -268,16 +234,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "table\nbank\nbank_raw\n"
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1587958743728_1444404682",
"id": "paragraph_1587958743728_1444404682",
@@ -291,6 +249,7 @@
"text": "%flink\n\nimport org.apache.flink.api.java.typeutils.RowTypeInfo\nimport org.apache.flink.api.common.typeinfo.Types\nimport org.apache.flink.api.java.typeutils._\nimport org.apache.flink.api.scala.typeutils._\nimport org.apache.flink.api.scala._\n\nclass Person(val age:Int, val job: String, val marital: String, val education: String, val default: String, val balance: String, val housing: String, val loan: String, val contact: String, val day: String, val month: String, val [...]
"user": "anonymous",
"dateUpdated": "2020-05-05 23:01:45.688",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -311,16 +270,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "import org.apache.flink.api.java.typeutils.RowTypeInfo\nimport org.apache.flink.api.common.typeinfo.Types\nimport org.apache.flink.api.java.typeutils._\nimport org.apache.flink.api.scala.typeutils._\nimport org.apache.flink.api.scala._\ndefined class Person\ndefined class ParseFunction\n"
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1578888628353_1621411444",
"id": "paragraph_1578888628353_1621411444",
@@ -334,6 +285,7 @@
"text": "%sh\n\nrm -rf /tmp/bank_cleaned\n#hadoop fs -rmr /tmp/bank_cleaned",
"user": "anonymous",
"dateUpdated": "2020-05-05 22:52:42.774",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -354,11 +306,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": []
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1579061020460_-113987164",
"id": "paragraph_1579061020460_-113987164",
@@ -372,6 +321,7 @@
"text": "%flink.bsql\n\ninsert into bank select T.* from bank_raw, LATERAL TABLE(parse(content)) as T(age, job, marital, education, `default`, balance, housing, loan, contact, `day`, `month`, duration, campaign, pdays, previous, poutcome, y) ",
"user": "anonymous",
"dateUpdated": "2020-05-05 22:53:04.445",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -392,16 +342,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "Insertion successfully.\n"
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1578669828368_-1923137601",
"id": "paragraph_1578669828368_-1923137601",
@@ -415,6 +357,7 @@
"text": "%flink.bsql\n\nselect * from bank limit 10\n",
"user": "anonymous",
"dateUpdated": "2020-05-05 23:01:51.464",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -479,16 +422,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "age\tjob\tmarital\teducation\tdefault\tbalance\thousing\tloan\tcontact\tday\tmonth\tduration\tcampaign\tpdays\tprevious\tpoutcome\ty\n30\tunemployed\tmarried\tprimary\tno\t1787\tno\tno\tcellular\t19\toct\t79\t1\t-1\t0\tunknown\tno\n33\tservices\tmarried\tsecondary\tno\t4789\tyes\tyes\tcellular\t11\tmay\t220\t1\t339\t4\tfailure\tno\n35\tmanagement\tsingle\ttertiary\tno\t1350\tyes\tno\tcellular\t16\tapr\t185\t1\t330\t1\tfailure\tno\n30\tmanagement\tmarried\ttertiary\tn [...]
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1578068480238_-1678045273",
"id": "paragraph_1578068480238_-1678045273",
@@ -502,6 +437,7 @@
"text": "%flink\n\nval table \u003d btenv.sqlQuery(\"select * from bank limit 10\")\nz.show(table)",
"user": "anonymous",
"dateUpdated": "2020-05-05 23:33:44.788",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -564,20 +500,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "\u001b[1m\u001b[34mtable\u001b[0m: \u001b[1m\u001b[32morg.apache.flink.table.api.Table\u001b[0m \u003d UnnamedTable$3\n"
- },
- {
- "type": "TABLE",
- "data": "age\tjob\tmarital\teducation\tdefault\tbalance\thousing\tloan\tcontact\tday\tmonth\tduration\tcampaign\tpdays\tprevious\tpoutcome\ty\n30\tunemployed\tmarried\tprimary\tno\t1787\tno\tno\tcellular\t19\toct\t79\t1\t-1\t0\tunknown\tno\n33\tservices\tmarried\tsecondary\tno\t4789\tyes\tyes\tcellular\t11\tmay\t220\t1\t339\t4\tfailure\tno\n35\tmanagement\tsingle\ttertiary\tno\t1350\tyes\tno\tcellular\t16\tapr\t185\t1\t330\t1\tfailure\tno\n30\tmanagement\tmarried\ttertiary\tn [...]
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1579061037737_-1577558456",
"id": "paragraph_1579061037737_-1577558456",
@@ -591,6 +515,7 @@
"text": "%flink.pyflink\n\ntable \u003d bt_env.sql_query(\"select * from bank limit 10\")\nz.show(table)",
"user": "anonymous",
"dateUpdated": "2020-05-05 23:37:48.878",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -653,16 +578,8 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "age\tjob\tmarital\teducation\tdefault\tbalance\thousing\tloan\tcontact\tday\tmonth\tduration\tcampaign\tpdays\tprevious\tpoutcome\ty\n30\tunemployed\tmarried\tprimary\tno\t1787\tno\tno\tcellular\t19\toct\t79\t1\t-1\t0\tunknown\tno\n33\tservices\tmarried\tsecondary\tno\t4789\tyes\tyes\tcellular\t11\tmay\t220\t1\t339\t4\tfailure\tno\n35\tmanagement\tsingle\ttertiary\tno\t1350\tyes\tno\tcellular\t16\tapr\t185\t1\t330\t1\tfailure\tno\n30\tmanagement\tmarried\ttertiary\tn [...]
- }
- ]
- },
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1588690392097_1159956807",
"id": "paragraph_1588690392097_1159956807",
@@ -675,12 +592,14 @@
"text": "%flink.pyflink\n",
"user": "anonymous",
"dateUpdated": "2020-05-05 23:37:07.989",
+ "progress": 0,
"config": {},
"settings": {
"params": {},
"forms": {}
},
"apps": [],
+ "runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1588693027989_1331448600",
"id": "paragraph_1588693027989_1331448600",
diff --git a/notebook/Flink Tutorial/7. Batch Data Analytics_2EZ9G3JJU.zpln b/notebook/Flink Tutorial/7. Batch Data Analytics_2EZ9G3JJU.zpln
index f66031a..dd3501b 100644
--- a/notebook/Flink Tutorial/7. Batch Data Analytics_2EZ9G3JJU.zpln
+++ b/notebook/Flink Tutorial/7. Batch Data Analytics_2EZ9G3JJU.zpln
@@ -4,7 +4,8 @@
"title": "Overview",
"text": "%md\n\nThis tutorial demonstrates how to use Flink do data exploration analytics via its.\n\n* batch sql \n* udf (scala, python \u0026 hive) \n* Zeppelin\u0027s dynamic forms and builtin visualization\n\nWe use the bank data registered in another tutorial note. You can also use any existed hive table. \n",
"user": "anonymous",
- "dateUpdated": "2020-04-29 22:12:26.213",
+ "dateUpdated": "2021-01-25 14:30:29.848",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -27,23 +28,14 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003cp\u003eThis tutorial demonstrates how to use Flink do data exploration analytics via its.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ebatch sql\u003c/li\u003e\n\u003cli\u003eudf (scala, python \u0026amp; hive)\u003c/li\u003e\n\u003cli\u003eZeppelin\u0026rsquo;s dynamic forms and builtin visualization\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWe use the bank data registered in another tutorial note. You can also [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1579053946947_-1754951794",
"id": "paragraph_1579053946947_-1754951794",
"dateCreated": "2020-01-15 10:05:46.947",
- "dateStarted": "2020-04-29 22:12:26.216",
- "dateFinished": "2020-04-29 22:12:26.371",
+ "dateStarted": "2021-01-25 14:30:29.848",
+ "dateFinished": "2021-01-25 14:30:29.873",
"status": "FINISHED"
},
{
@@ -51,6 +43,7 @@
"text": "%flink.bsql\n\nselect age, count(1) as aval\nfrom bank \nwhere age \u003c 30 \ngroup by age \norder by age\n",
"user": "anonymous",
"dateUpdated": "2020-04-27 11:41:22.697",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -122,15 +115,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "age\taval\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -146,6 +130,7 @@
"text": "%flink.bsql\n\nselect age, count(1) as val \nfrom bank \nwhere age \u003c ${maxAge\u003d30} \ngroup by age \norder by age",
"user": "anonymous",
"dateUpdated": "2020-04-27 11:41:24.988",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -226,15 +211,6 @@
}
}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "age\tval\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n30\t150\n31\t199\n32\t224\n33\t186\n34\t231\n35\t180\n36\t188\n37\t161\n38\t159\n39\t130\n40\t142\n41\t135\n42\t141\n43\t115\n44\t105\n45\t112\n46\t119\n47\t108\n48\t114\n49\t112\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -250,6 +226,7 @@
"text": "%flink.bsql\n\nselect age, count(1) as val \nfrom bank \nwhere marital\u003d\u0027${marital\u003dsingle,single|divorced|married}\u0027 \ngroup by age \norder by age",
"user": "anonymous",
"dateUpdated": "2020-04-27 11:41:26.761",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -341,15 +318,6 @@
}
}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "age\tval\n23\t3\n24\t11\n25\t11\n26\t18\n27\t26\n28\t23\n29\t37\n30\t56\n31\t104\n32\t105\n33\t103\n34\t142\n35\t109\n36\t117\n37\t100\n38\t99\n39\t88\n40\t105\n41\t97\n42\t91\n43\t79\n44\t68\n45\t76\n46\t82\n47\t78\n48\t91\n49\t87\n50\t74\n51\t63\n52\t66\n53\t75\n54\t56\n55\t68\n56\t50\n57\t78\n58\t67\n59\t56\n60\t36\n61\t15\n62\t5\n63\t7\n64\t6\n65\t4\n66\t7\n67\t5\n68\t1\n69\t5\n70\t5\n71\t5\n72\t4\n73\t6\n74\t2\n75\t3\n76\t1\n77\t5\n78\t2\n79\t3\n80\t6\n81\t1\n83 [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -365,6 +333,7 @@
"text": "%flink\n\n\nclass ScalaUpper extends ScalarFunction {\n def eval(str: String) \u003d str.toUpperCase\n}\n\nbtenv.registerFunction(\"scala_upper\", new ScalaUpper())\n",
"user": "anonymous",
"dateUpdated": "2020-04-27 11:41:30.878",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -385,15 +354,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "defined class ScalaUpper\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -409,6 +369,7 @@
"text": "%flink.pyflink\n\nclass PythonUpper(ScalarFunction):\n def eval(self, s):\n return s.upper()\n\nbt_env.register_function(\"python_upper\", udf(PythonUpper(), DataTypes.STRING(), DataTypes.STRING()))\n\n\n\n\n",
"user": "anonymous",
"dateUpdated": "2020-04-27 11:41:33.020",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -429,10 +390,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": []
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -448,6 +405,7 @@
"text": "%flink.bsql\n\nhelp",
"user": "anonymous",
"dateUpdated": "2020-04-27 11:41:37.384",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -474,15 +432,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "The following commands are available:\n\nCREATE TABLE\t\tCreate table under current catalog and database.\nDROP TABLE\t\tDrop table with optional catalog and database. Syntax: \u0027DROP TABLE [IF EXISTS] \u003cname\u003e;\u0027\nCREATE VIEW\t\tCreates a virtual table from a SQL query. Syntax: \u0027CREATE VIEW \u003cname\u003e AS \u003cquery\u003e;\u0027\nDESCRIBE\t\tDescribes the schema of a table with the given name.\nDROP VIEW\t\tDeletes a previously created virt [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -498,6 +447,7 @@
"text": "%flink.bsql\n\nselect scala_upper(education), count(1) from bank group by education\n",
"user": "anonymous",
"dateUpdated": "2020-04-27 11:41:42.397",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -569,15 +519,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "EXPR$0\tEXPR$1\nPRIMARY\t678\nSECONDARY\t2306\nTERTIARY\t1350\nUNKNOWN\t187\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -593,6 +534,7 @@
"text": "%flink.bsql\n\nselect python_upper(education) as edu, count(1) as c from bank group by education\n\n",
"user": "anonymous",
"dateUpdated": "2020-04-27 11:41:44.032",
+ "progress": 0,
"config": {
"runOnSelectionChange": true,
"title": true,
@@ -677,15 +619,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "edu\tc\nPRIMARY\t678\nSECONDARY\t2306\nTERTIARY\t1350\nUNKNOWN\t187\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -700,6 +633,7 @@
"text": "%flink.bsql\n",
"user": "anonymous",
"dateUpdated": "2020-02-05 16:10:30.318",
+ "progress": 0,
"config": {},
"settings": {
"params": {},
diff --git a/notebook/Flink Tutorial/8. Logistic Regression (Alink)_2F4HJNWVN.zpln b/notebook/Flink Tutorial/8. Logistic Regression (Alink)_2F4HJNWVN.zpln
index 097ddf8..a29dbb9 100644
--- a/notebook/Flink Tutorial/8. Logistic Regression (Alink)_2F4HJNWVN.zpln
+++ b/notebook/Flink Tutorial/8. Logistic Regression (Alink)_2F4HJNWVN.zpln
@@ -3,7 +3,8 @@
{
"text": "%md\n\n# Introduction\n\nThis note is to demonstrate how to do machine learning in flink. Here we use [Alink](https://github.com/alibaba/Alink/). \nWe use logics regression to do classification task. We use the same data as other tutorials [bank](https://archive.ics.uci.edu/ml/datasets/bank+marketing).\n",
"user": "anonymous",
- "dateUpdated": "2020-04-29 16:09:44.080",
+ "dateUpdated": "2021-01-25 14:30:35.800",
+ "progress": 0,
"config": {
"colWidth": 12.0,
"fontSize": 9.0,
@@ -23,29 +24,21 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "HTML",
- "data": "\u003cdiv class\u003d\"markdown-body\"\u003e\n\u003ch1\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eThis note is to demonstrate how to do machine learning in flink. Here we use \u003ca href\u003d\"https://github.com/alibaba/Alink/\"\u003eAlink\u003c/a\u003e.\u003cbr /\u003e\nWe use logics regression to do classification task. We use the same data as other tutorials \u003ca href\u003d\"https://archive.ics.uci.edu/ml/datasets/bank+marketing\"\u003ebank\u003c/a\u003e. [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
"jobName": "paragraph_1588147625869_1181490991",
"id": "paragraph_1588147625869_1181490991",
"dateCreated": "2020-04-29 16:07:05.869",
- "dateStarted": "2020-04-29 16:09:44.080",
- "dateFinished": "2020-04-29 16:09:44.102",
+ "dateStarted": "2021-01-25 14:30:35.800",
+ "dateFinished": "2021-01-25 14:30:35.807",
"status": "FINISHED"
},
{
"text": "%flink.pyflink\n\nimport pyflink\nfrom pyflink.dataset import ExecutionEnvironment\nfrom pyflink.datastream import StreamExecutionEnvironment\nfrom pyalink.alink.env import useCustomEnv\nmlenv \u003d useCustomEnv(gateway,\n b_env,bt_env_2, s_env, st_env_2)\nfrom pyalink.alink import *\n\nt \u003d bt_env_2.from_elements([(1, 2), (2, 5), (3, 1)], [\u0027a\u0027, \u0027b\u0027])\nsource \u003d TableSourceBatchOp(t)\nsource.print()",
"user": "anonymous",
"dateUpdated": "2020-04-27 13:48:06.523",
+ "progress": 0,
"config": {
"editorMode": "ace/mode/python",
"editorHide": false,
@@ -64,15 +57,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "\nUse one of the following commands to start using PyAlink:\n - useLocalEnv(parallelism, flinkHome\u003dNone, config\u003dNone)\n - useRemoteEnv(host, port, parallelism, flinkHome\u003dNone, localIp\u003d\"localhost\", config\u003dNone)\nCall resetEnv() to reset environment and switch to another.\n\n a b\n0 1 2\n1 2 5\n2 3 1\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -87,6 +71,7 @@
"text": "%flink.pyflink\n\n\ntest_data_path \u003d \"/tmp/bank.csv\"\nfull_data_path \u003d \"/tmp/bank-full.csv\"\nschema_str \u003d \"age int, job string, marital string, education string, default string, balance string, housing string, loan string, contact string, day string, month string, duration int, campaign int, pdays int, previous int, poutcome string, y string\"\n\ntest_data \u003d CsvSourceBatchOp() \\\n .setFilePath(test_data_path) \\\n .setSchemaStr(schema_str) \ [...]
"user": "anonymous",
"dateUpdated": "2020-04-29 16:10:03.433",
+ "progress": 0,
"config": {
"editorMode": "ace/mode/python",
"editorHide": false,
@@ -105,10 +90,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": []
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -123,6 +104,7 @@
"text": "%flink.pyflink\n\n\ncategoricalColNames \u003d [\"job\", \"marital\", \"education\", \"default\",\n \"balance\", \"housing\", \"loan\", \"contact\", \"poutcome\" ]\nnumerialColNames \u003d [\"age\", \"duration\", \"campaign\", \"pdays\",\n \"previous\"]\nlabelColName \u003d \"y\"\n\nonehot \u003d OneHotEncoder().setSelectedCols(categoricalColNames) \\\n .setOutputCols([\"output\"])\nassembler \u003d VectorAssembler().setSelec [...]
"user": "anonymous",
"dateUpdated": "2020-04-27 13:48:37.221",
+ "progress": 0,
"config": {
"editorMode": "ace/mode/python",
"editorHide": false,
@@ -141,10 +123,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": []
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -159,6 +137,7 @@
"text": "%flink.pyflink\n\n\nlogistic \u003d LogisticRegression().setVectorCol(\"vec\").setLabelCol(labelColName) \\\n .setPredictionCol(\"pred\").setPredictionDetailCol(\"detail\")\nmodel \u003d pipeline.add(logistic).fit(train_set)\n\npredict \u003d model.transform(test_data)\n\nmetrics \u003d EvalBinaryClassBatchOp().setLabelCol(labelColName) \\\n .setPredictionDetailCol(\"detail\").linkFrom(predict).collectMetrics()\n \n \n ",
"user": "anonymous",
"dateUpdated": "2020-04-27 13:48:39.232",
+ "progress": 0,
"config": {
"editorMode": "ace/mode/python",
"editorHide": false,
@@ -177,10 +156,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": []
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -195,6 +170,7 @@
"text": "%flink.pyflink\n\nprint(\"AUC:\", metrics.getAuc())\nprint(\"KS:\", metrics.getKs())\nprint(\"PRC:\", metrics.getPrc())\nprint(\"Precision:\", metrics.getPrecision())\nprint(\"Recall:\", metrics.getRecall())\nprint(\"F1:\", metrics.getF1())\nprint(\"ConfusionMatrix:\", metrics.getConfusionMatrix())\nprint(\"LabelArray:\", metrics.getLabelArray())\nprint(\"LogLoss:\", metrics.getLogLoss())\nprint(\"TotalSamples:\", metrics.getTotalSamples())\nprint(\"ActualLabelProportion:\ [...]
"user": "anonymous",
"dateUpdated": "2020-04-27 13:48:51.441",
+ "progress": 0,
"config": {
"editorMode": "ace/mode/python",
"editorHide": false,
@@ -213,15 +189,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "AUC: 0.9076425143954201\nKS: 0.6844817658349329\nPRC: 0.5349186786420718\nPrecision: 0.4748803827751196\nRecall: 0.761996161228407\nF1: 0.5851142225497421\nConfusionMatrix: [[397, 439], [124, 3561]]\nLabelArray: [\u0027yes\u0027, \u0027no\u0027]\nLogLoss: 0.3427333415817235\nTotalSamples: 4521\nActualLabelProportion: [0.11523999115239991, 0.8847600088476001]\nActualLabelFrequency: [521, 4000]\nAccuracy: 0.8754700287547003\nKappa: 0.5164554316821129\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -236,6 +203,7 @@
"text": "%flink.pyflink\n\ndf \u003d predict.filter(\"y\u003c\u003epred\").firstN(300).collectToDataframe()\n\nz.show(df)",
"user": "anonymous",
"dateUpdated": "2020-04-27 13:48:54.873",
+ "progress": 0,
"config": {
"editorMode": "ace/mode/python",
"editorHide": false,
@@ -302,15 +270,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TABLE",
- "data": "age\tjob\tmarital\teducation\tdefault\tbalance\thousing\tloan\tcontact\tday\tmonth\tduration\tcampaign\tpdays\tprevious\tpoutcome\ty\toutput\tvec\tpred\tdetail\n36\tself-employed\tmarried\ttertiary\tno\t307\tyes\tno\tcellular\t14\tmay\t341\t1\t330\t2\tother\tno\t$4563$6:1.0 13:1.0 17:1.0 19:1.0 2518:1.0 4554:1.0 4556:1.0 4560:1.0\t$4568$6:1.0 13:1.0 17:1.0 19:1.0 2518:1.0 4554:1.0 4556:1.0 4560:1.0 4563:36.0 4564:341.0 4565:1.0 4566:330.0 4567:2.0\tyes\t{\"no\":\"0.2 [...]
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,
@@ -325,6 +284,7 @@
"text": "%flink.pyflink\n",
"user": "anonymous",
"dateUpdated": "2020-03-10 11:04:48.771",
+ "progress": 0,
"config": {
"editorMode": "ace/mode/python",
"editorHide": false,
@@ -343,15 +303,6 @@
"params": {},
"forms": {}
},
- "results": {
- "code": "SUCCESS",
- "msg": [
- {
- "type": "TEXT",
- "data": "UsageError: Line magic function `%flink.pyflink` not found.\n"
- }
- ]
- },
"apps": [],
"runtimeInfos": {},
"progressUpdateIntervalMs": 500,