You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by sh...@apache.org on 2017/02/20 05:26:19 UTC
[2/2] kylin git commit: Document format review on flink.md and
squirrel.md
Document format review on flink.md and squirrel.md
Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/e3d43f72
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/e3d43f72
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/e3d43f72
Branch: refs/heads/document
Commit: e3d43f725e7c9e9e28290b44d4390e990916c58c
Parents: 159678c
Author: shaofengshi <sh...@apache.org>
Authored: Mon Feb 20 13:26:09 2017 +0800
Committer: shaofengshi <sh...@apache.org>
Committed: Mon Feb 20 13:26:09 2017 +0800
----------------------------------------------------------------------
website/_docs16/tutorial/flink.md | 146 ++++++++++++------------------
website/_docs16/tutorial/squirrel.md | 66 +++++---------
2 files changed, 82 insertions(+), 130 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kylin/blob/e3d43f72/website/_docs16/tutorial/flink.md
----------------------------------------------------------------------
diff --git a/website/_docs16/tutorial/flink.md b/website/_docs16/tutorial/flink.md
index 579bb08..f3cb99f 100644
--- a/website/_docs16/tutorial/flink.md
+++ b/website/_docs16/tutorial/flink.md
@@ -1,45 +1,39 @@
---
layout: docs16
-title: Apache Flink
+title: Connect from Apache Flink
categories: tutorial
permalink: /docs16/tutorial/flink.html
---
-### Intro
-There are several Attempts to use this in Scala and JDBC [Attempt1](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/JDBCInputFormat-preparation-with-Flink-1-1-SNAPSHOT-and-Scala-2-11-td5371.html) [Attempt2](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Type-of-TypeVariable-OT-in-class-org-apache-flink-api-common-io-RichInputFormat-could-not-be-determi-td7287.html) [Attempt3](http://stackoverflow.com/questions/36067881/create-dataset-from-jdbc-source-in-flink-using-scala) [Attempt4](https://codegists.com/snippet/scala/jdbcissuescala_zeitgeist_scala) but none works ...
+### Introduction
-**Problem 1:** There aren\u2019t any doc about connect Kylin with Flink
+This document describes how to use Kylin as a data source in Apache Flink;
-**Problem 2:** We will try use CreateInput and [JDBCInputFormat](https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/batch/index.html) in batch mode and access via JDBC to Kylin. But isn\u2019t implemented in Scala is only in Java [MailList](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/jdbc-JDBCInputFormat-td9393.html)
-
+There were several attempts to do this in Scala and JDBC, but none of them works:
-Then , We will go step by step solving problems
+* [attempt1](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/JDBCInputFormat-preparation-with-Flink-1-1-SNAPSHOT-and-Scala-2-11-td5371.html)
+* [attempt2](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Type-of-TypeVariable-OT-in-class-org-apache-flink-api-common-io-RichInputFormat-could-not-be-determi-td7287.html)
+* [attempt3](http://stackoverflow.com/questions/36067881/create-dataset-from-jdbc-source-in-flink-using-scala)
+* [attempt4](https://codegists.com/snippet/scala/jdbcissuescala_zeitgeist_scala);
-
-### Pre-requisites
-* We need an instance of Kylin, with a cube: [Quick Start with Sample Cube](http://kylin.apache.org/docs15/tutorial/kylin_sample.html), will be enough
-
- You can check:
-
- 
+We will try use CreateInput and [JDBCInputFormat](https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/batch/index.html) in batch mode and access via JDBC to Kylin. But it isn\u2019t implemented in Scala, is only in Java [MailList](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/jdbc-JDBCInputFormat-td9393.html). This doc will go step by step solving these problems.
+### Pre-requisites
+* Need an instance of Kylin, with a Cube; [Sample Cube](/docs16/tutorial/kylin_sample.html) will be good enough.
* [Scala](http://www.scala-lang.org/) and [Apache Flink](http://flink.apache.org/) Installed
+* [IntelliJ](https://www.jetbrains.com/idea/) Installed and configured for Scala/Flink (see [Flink IDE setup guide](https://ci.apache.org/projects/flink/flink-docs-release-1.1/internals/ide_setup.html) )
-* [IntelliJ](https://www.jetbrains.com/idea/) Installed and configured for Scala / Flink (See [Flink IDE setup guide](https://ci.apache.org/projects/flink/flink-docs-release-1.1/internals/ide_setup.html) )
+### Used software:
-
-### Used Software:
* [Apache Flink](http://flink.apache.org/downloads.html) v1.2-SNAPSHOT
-* [Apache Kylin](http://kylin.apache.org/download/) v1.5.2
-
- (In Nov 2016, only works with 1.5.4 or more by dependences with Calcite, I Tested with v1.6.0)
+* [Apache Kylin](http://kylin.apache.org/download/) v1.5.2 (v1.6.0 also works)
* [IntelliJ](https://www.jetbrains.com/idea/download/#section=linux) v2016.2
* [Scala](downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz) v2.11
-
### Starting point:
+
This can be out initial skeleton:
{% highlight Groff markup %}
@@ -57,18 +51,14 @@ val inputFormat = JDBCInputFormat.buildJDBCInputFormat()
The first error is: 
-We add to Scala:
+Add to Scala:
{% highlight Groff markup %}
import org.apache.flink.api.java.io.jdbc.JDBCInputFormat
{% endhighlight %}
-
-
Next error is 
-We can solve dependencies [(mvn repository: jdbc)](https://mvnrepository.com/artifact/org.apache.flink/flink-jdbc/1.1.2)
-
-Add this to your pom.xml:
+We can solve dependencies [(mvn repository: jdbc)](https://mvnrepository.com/artifact/org.apache.flink/flink-jdbc/1.1.2); Add this to your pom.xml:
{% highlight Groff markup %}
<dependency>
<groupId>org.apache.flink</groupId>
@@ -77,14 +67,14 @@ Add this to your pom.xml:
</dependency>
{% endhighlight %}
-
## Solve dependencies of row
+
Similar to previous point we need solve dependencies of Row Class [(mvn repository: Table) ](https://mvnrepository.com/artifact/org.apache.flink/flink-table_2.10/1.1.2):
-* In POM.XML
+* In pom.xml add:
{% highlight Groff markup %}
<dependency>
<groupId>org.apache.flink</groupId>
@@ -98,14 +88,14 @@ Similar to previous point we need solve dependencies of Row Class [(mvn reposito
import org.apache.flink.api.table.Row
{% endhighlight %}
-
## Solve RowTypeInfo property (and their new dependencies)
-This is the new error to solve
+
+This is the new error to solve:
-* If we check the code of [JDBCInputFormat.java](https://github.com/apache/flink/blob/master/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormat.java#L69), we can see [this new property](https://github.com/apache/flink/commit/09b428bd65819b946cf82ab1fdee305eb5a941f5#diff-9b49a5041d50d9f9fad3f8060b3d1310R69) (and mandatory) added on Apr 2016 by [FLINK-3750](https://issues.apache.org/jira/browse/FLINK-3750) Manual [JDBCInputFormat](https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormat.html) v1.2 in Java
+* If check the code of [JDBCInputFormat.java](https://github.com/apache/flink/blob/master/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormat.java#L69), we can see [this new property](https://github.com/apache/flink/commit/09b428bd65819b946cf82ab1fdee305eb5a941f5#diff-9b49a5041d50d9f9fad3f8060b3d1310R69) (and mandatory) added on Apr 2016 by [FLINK-3750](https://issues.apache.org/jira/browse/FLINK-3750) Manual [JDBCInputFormat](https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormat.html) v1.2 in Java
Add the new Property: **setRowTypeInfo**
@@ -120,51 +110,47 @@ val inputFormat = JDBCInputFormat.buildJDBCInputFormat()
.finish()
{% endhighlight %}
-* �How can configure this property in Scala? In [Attempt4](https://codegists.com/snippet/scala/jdbcissuescala_zeitgeist_scala), there is an incorrect solution
+* How can configure this property in Scala? In [Attempt4](https://codegists.com/snippet/scala/jdbcissuescala_zeitgeist_scala), there is an incorrect solution
We can check the types using the intellisense: 
- Then we will need add more dependences :(
-
- Add to scala:
+ Then we will need add more dependences; Add to scala:
+
{% highlight Groff markup %}
- import org.apache.flink.api.table.typeutils.RowTypeInfo
- import org.apache.flink.api.common.typeinfo.{BasicTypeInfo, TypeInformation}
+import org.apache.flink.api.table.typeutils.RowTypeInfo
+import org.apache.flink.api.common.typeinfo.{BasicTypeInfo, TypeInformation}
{% endhighlight %}
- We need create a Array or Seq of TypeInformation[ ]
+ Create a Array or Seq of TypeInformation[ ]
- My solution:
+ Solution:
{% highlight Groff markup %}
var stringColum: TypeInformation[String] = createTypeInformation[String]
val DB_ROWTYPE = new RowTypeInfo(Seq(stringColum))
{% endhighlight %}
-
-## Solve Class Not Found
-We need find the kylin-jdbc-x.x.x.jar and expose to flink
+## Solve ClassNotFoundException
+Need find the kylin-jdbc-x.x.x.jar and then expose to Flink
-1. We need to find the JAR Class for the JDBC Connector
+1. Find the Kylin JDBC jar
- From Kylin [Download](http://kylin.apache.org/download/) Choose **Binary** and the **correct version of Kylin and HBase**
+ From Kylin [Download](http://kylin.apache.org/download/) choose **Binary** and the **correct version of Kylin and HBase**
Download & Unpack: in ./lib:
-
-
2. Make this JAR accessible to Flink
- If you execute like service you need put this JAR in you Java ClassPATH using your .bashrc
+ If you execute like service you need put this JAR in you Java class path using your .bashrc
@@ -176,40 +162,37 @@ We need find the kylin-jdbc-x.x.x.jar and expose to flink
-
- If you are executing from IDE, you need add your ClassPath manually:
+ If you are executing from IDE, need add your class path manually:
On IntelliJ:  >  >  > 
The result, will be similar to: 
-
-## Solve Couldn\u2019t access resultSet
+## Solve "Couldn\u2019t access resultSet" error
-Is related with [Flink 4108](https://issues.apache.org/jira/browse/FLINK-4108) [(MailList)](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/jdbc-JDBCInputFormat-td9393.html#a9415) and Timo Walther [make a PR](https://github.com/apache/flink/pull/2619)
+It is related with [Flink 4108](https://issues.apache.org/jira/browse/FLINK-4108) [(MailList)](http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/jdbc-JDBCInputFormat-td9393.html#a9415) and Timo Walther [make a PR](https://github.com/apache/flink/pull/2619)
-If you are <= Flink 1.2 you will need apply this path and make clean install
+If you are running Flink <= 1.2 you will need apply this path and make clean install
-
-## Solve the casting
+## Solve the casting error
-In the error msg you have the problem and solution \u2026. nice ;) ��
+In the error message you have the problem and solution \u2026. nice ;) ��
-
## The result
-The out must be similar to this, print the result of query by standard output:
+
+The output must be similar to this, print the result of query by standard output:
-
-## Now, more complex . . .
-We can try with multi-colum and multi-type query:
+## Now, more complex
+
+Try with a multi-colum and multi-type query:
{% highlight Groff markup %}
select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers
@@ -218,60 +201,49 @@ group by part_dt
order by part_dt
{% endhighlight %}
-We will need changes in DB_ROWTYPE:
+Need changes in DB_ROWTYPE:
-And import lib of Java, to work with Data type of Java 
+And import lib of Java, to work with Data type of Java 
-The new Result will be: 
+The new result will be:
-
-
-
## Error: Reused Connection
+
-Check if your HBase and Kylin is working
-Also you can use Kylin UI for it
+Check if your HBase and Kylin is working. Also you can use Kylin UI for it.
-
## Error: java.lang.AbstractMethodError: \u2026.Avatica Connection
+
See [Kylin 1898](https://issues.apache.org/jira/browse/KYLIN-1898)
-It is a problem with kylin-jdbc-1.x.x. JAR, you need use Calcite 1.8 or more \u2192 use Kylin 1.5.4 or more
+It is a problem with kylin-jdbc-1.x.x. JAR, you need use Calcite 1.8 or above; The solution is to use Kylin 1.5.4 or above.
-
## Error: can't expand macros compiled by previous versions of scala
-Is a problem with versions of scala, check in with scala -version your actual version and choose your correct POM
-Perhaps you will need a IntelliJ > File > Invalidates Cache > Invalidate and Restart
+Is a problem with versions of scala, check in with "scala -version" your actual version and choose your correct POM.
-I Aded POM for Scala 2.11
+Perhaps you will need a IntelliJ > File > Invalidates Cache > Invalidate and Restart.
+I added POM for Scala 2.11
-
-## Final Words
-**Now we can read Kylin\u2019s data from Apache Flink, great News ��**
-
-[Full Code Example](https://github.com/albertoRamon/Flink/tree/master/ReadKylinFromFlink/flink-scala-project)
-We solved all integration problems, and tested with different types of data (Long, BigDecimal and Dates)
-
-Today (12 Oct 2016) Flink 1.2-SnapShot, you need download, apply path, compile and make install \u2026. but in new releases will not necessary
+## Final Words
-**Updated** The patch has been comited at 15 Oct, then, Will be part of Flink 1.2
+Now you can read Kylin\u2019s data from Apache Flink, great!
-
-
+[Full Code Example](https://github.com/albertoRamon/Flink/tree/master/ReadKylinFromFlink/flink-scala-project)
+Solved all integration problems, and tested with different types of data (Long, BigDecimal and Dates). The patch has been comited at 15 Oct, then, will be part of Flink 1.2.
http://git-wip-us.apache.org/repos/asf/kylin/blob/e3d43f72/website/_docs16/tutorial/squirrel.md
----------------------------------------------------------------------
diff --git a/website/_docs16/tutorial/squirrel.md b/website/_docs16/tutorial/squirrel.md
index 3a731cd..5e69780 100644
--- a/website/_docs16/tutorial/squirrel.md
+++ b/website/_docs16/tutorial/squirrel.md
@@ -1,45 +1,39 @@
---
layout: docs16
-title: SQuirreL
+title: Connect from SQuirreL
categories: tutorial
permalink: /docs16/tutorial/squirrel.html
---
-### Intro
-SquirreL SQL is a multi platform Universal SQL Client (GNU License)
-You can use it to access HBase + Phoenix and Hive
+### Introduction
+
+[SQuirreL SQL](http://www.squirrelsql.org/) is a multi platform Universal SQL Client (GNU License). You can use it to access HBase + Phoenix and Hive. This document introduces how to connect to Kylin from SQuirreL.
-
### Used Software
-* [Kylin v1.5.2](http://kylin.apache.org/download/) & ODBC 1.5
-* Update Dic 2016: Kylin 1.6.0 & ODBC 1.6 Works OK
+
+* [Kylin v1.6.0](/download/) & ODBC 1.6
* [SquirreL SQL v3.7.1](http://www.squirrelsql.org/)
-
## Pre-requisites
-* We need to find the JAR Class for the JDBC Connector
+* Find the Kylin JDBC driver jar
From Kylin Download, Choose Binary and the **correct version of Kylin and HBase**
-
Download & Unpack: in **./lib**:
-
-* We need an instance of Kylin, with a cube: Quick Start with Sample Cube, will be enough
-
- You can check:
+* Need an instance of Kylin, with a Cube; the [Sample Cube](/docs16/tutorial/kylin_sample.html) is enough.
-* [Dowload and install SquirreL](http://www.squirrelsql.org/#installation), you will need Java
+* [Dowload and install SquirreL](http://www.squirrelsql.org/#installation)
+
+## Add Kylin JDBC Driver
-
-## Add Kylin Driver
On left menu:  > >  > 
-And locale your JAR: 
+And locate the JAR: 
Configure this parameters:
@@ -48,12 +42,12 @@ Configure this parameters:
jdbc:kylin://172.17.0.2:7070/learn_kylin
* Put Class Name: 
- TIP: If auto complete not work, type you: org.apache.kylin.jdbc.Driver
+ Tip: If auto complete not work, type: org.apache.kylin.jdbc.Driver
-Check in your Driver List: 
+Check the Driver List: 
-
## Add Aliases
+
On left menu:  >  : (Login pass by default: ADMIN / KYLIN)
@@ -64,15 +58,14 @@ And automatically launch conection:
-
## Connect and Execute
-The startup window when you are connect:
- 
+The startup window when connected:
+ 
-Choose Tab: and write your querie (whe use Kylin\u2019s example cube):
+Choose Tab: and write a query (whe use Kylin\u2019s example cube):
@@ -88,45 +81,32 @@ Execute With: 
-And it\u2019s works,OK 
+And it\u2019s works!
-
-## Extra: Some tips:
-SquirreL isn\u2019t the most stable SQL Client, but its very flexible and get you a lot of info
+## Tips:
-I use it for PoC and try to solve / check connectivity problems
+SquirreL isn\u2019t the most stable SQL Client, but it is very flexible and get a lot of info; It can be used for PoC and checking connectivity issues.
-List of Tables:
+List of tables:
-
-
-List of Columns of table:
+List of columns of table:
-
-
List of column of Querie:
-
-
Export the result of queries:
-
-
Info about time query execution:
-
-
-