You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by NicoK <gi...@git.apache.org> on 2018/04/23 17:02:57 UTC
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/5900
[FLINK-9222][docs] add documentation for setting up Gradle projects
## What is the purpose of the change
Together with https://github.com/apache/flink-web/pull/105, this will add a project template for setting up Flink jobs via Gradle.
## Brief change log
- adapt documentation to cover Gradle for Java:
- add project file templates to the docs
- link the new quickstart
- adapt the description
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-9222
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5900.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5900
----
commit 2d5bdb141560c74393841e02b48c75024193f8eb
Author: Nico Kruber <ni...@...>
Date: 2018-04-23T16:58:15Z
[hotfix][docs] fix some typos
commit e5a3e52c660898106e615f0fbf99b2f8091e469c
Author: Nico Kruber <ni...@...>
Date: 2018-04-23T16:58:28Z
[FLINK-9222][docs] add documentation for setting up Gradle projects
----
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194901455
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ //myShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ testCompile "junit:junit:4.12"
+}
+
+// make compileOnly dependencies available for tests:
+sourceSets {
+ main.runtimeClasspath += configurations.myShadowJar
+ main.runtimeClasspath += configurations.myShadowJar
+
+ test.compileClasspath += configurations.myShadowJar
+ test.runtimeClasspath += configurations.myShadowJar
+}
+
+jar {
+ manifest {
+ attributes 'Built-By': System.getProperty('user.name'),
+ 'Build-Jdk': System.getProperty('java.version')
+ }
+}
+
+shadowJar {
+ configurations = [project.configurations.myShadowJar]
+}
+ {% endhighlight %}
+ </div>
+ <div class="tab-pane" id="gradle-settings">
+ {% highlight gradle %}
+rootProject.name = 'quickstart'
+ {% endhighlight %}
+ </div>
+ </div>
+ </div>
+
+ <div class="tab-pane" id="gradle-script">
+ {% highlight bash %}
+ $ curl https://flink.apache.org/q/gradle-quickstart.sh | bash
+ {% endhighlight %}
+ This allows you to <strong>name your newly created project</strong>. It will interactively ask
+ you for the project name, organization (also used for the package name), project version,
+ Scala and Flink version.
+ </div>
+</div>
+
+### Inspect Project
+
+There will be a new directory in your working directory based on the
+project name you provided, e.g. for `quickstart`:
+
+{% highlight bash %}
+$ tree quickstart/
+quickstart/
+├── README
+├── build.gradle
+├── settings.gradle
+└── src
+ └── main
+ ├── java
+ │ └── org
+ │ └── myorg
+ │ └── quickstart
+ │ ├── BatchJob.java
+ │ └── StreamingJob.java
+ └── resources
+ └── log4j.properties
+{% endhighlight %}
+
+The sample project is a __Gradle project__, which contains two classes: _StreamingJob_ and _BatchJob_ are the basic skeleton programs for a *DataStream* and *DataSet* program.
+The _main_ method is the entry point of the program, both for in-IDE testing/execution and for proper deployments.
+
+We recommend you __import this project into your IDE__ to develop and
+test it. IntelliJ IDEA supports Gradle projects after installing the `Gradle` plugin.
+Eclipse does so via the [Eclipse Buildship](https://projects.eclipse.org/projects/tools.buildship) plugin.
--- End diff --
no, sorry - that should of course be double-checked
---
[GitHub] flink issue #5900: [FLINK-9222][docs] add documentation for setting up Gradl...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5900
Nice addition!
A few things I would like to double check on the quickstart configuration (I am not fluent enough Gradle):
- We do not need to hide/shade any dependencies in the user code. In Maven, we use the shade plugin, but only to build an uber jar, not to actually relocate dependencies. Is that the same in the Gradle quickstart?
- The Flink core dependencies need to be in a scope equivalent to "provided", so they do not end up in the uber jar. Can we do something similar in Gradle? This has been a frequent source of unnecessarily bloated application jars.
- The Maven quickstart template uses a trick to make sure that the provided dependencies are still in the classpath when we run the program in the IDE: A profile that activates in IDEA (by a property variable) and alters the scope from *provided* to *compile*. Not sure if that is strictly necessary, but may be helpful.
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by imrel <gi...@git.apache.org>.
Github user imrel commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r197720653
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
--- End diff --
compileOnly for provided dependencies works good for me,
for running in IDEA i select checkbox "include provided dependencies" on run configuration and for tests i add this:
configurations {
testCompile.extendsFrom compileOnly
}
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194709349
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
--- End diff --
Shall we move the sections for which a user cares (Flink version and dependencies) to the beginning?
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r202066470
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -101,23 +111,210 @@ allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/refer
Some Eclipse bundles include that plugin by default, others require you
to install it manually.
-*A note to Mac OS X users*: The default JVM heapsize for Java may be too
+*Please note*: The default JVM heapsize for Java may be too
small for Flink. You have to manually increase it.
-In Eclipse, choose
-`Run Configurations -> Arguments` and write into the `VM Arguments`
-box: `-Xmx800m`.
+In Eclipse, choose `Run Configurations -> Arguments` and write into the `VM Arguments` box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.4'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '{{ site.version }}'
+ scalaBinaryVersion = '{{ site.scala_version }}'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "flinkShadowJar" configuration!
+configurations {
+ flinkShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ flinkShadowJar.exclude group: 'org.slf4j'
+ flinkShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ // Add connector dependencies here.
+ // They must be in the flinkShadowJar configuration in order to be included into the shaded jar.
+ //flinkShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ // Add test dependencies here.
+ // testCompile "junit:junit:4.12"
+}
+
+// make compileOnly dependencies available for tests:
+sourceSets {
+ main.runtimeClasspath += configurations.flinkShadowJar
--- End diff --
makes sense, I added this to the script, thanks
---
[GitHub] flink issue #5900: [FLINK-9222][docs] add documentation for setting up Gradl...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/5900
Thanks for the review, Timo.
Regarding the copy of the gradle build file: since we did not really have a pristine version (only one that is generated by the shell script) I wanted to put one somewhere for users who don't won't/trust the shell script :) Alternatively, we could put this up for download in flink-web as well, but the visibility is not the same as it is when having it included in the docs.
---
[GitHub] flink issue #5900: [FLINK-9222][docs] add documentation for setting up Gradl...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/5900
Hi @twalthr,
I just created a fixup commit for this PR and the one for `flink-web`. Can you have a second look?
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194708294
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ //myShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ testCompile "junit:junit:4.12"
+}
+
+// make compileOnly dependencies available for tests:
+sourceSets {
+ main.runtimeClasspath += configurations.myShadowJar
+ main.runtimeClasspath += configurations.myShadowJar
+
+ test.compileClasspath += configurations.myShadowJar
+ test.runtimeClasspath += configurations.myShadowJar
+}
+
+jar {
+ manifest {
+ attributes 'Built-By': System.getProperty('user.name'),
+ 'Build-Jdk': System.getProperty('java.version')
+ }
+}
+
+shadowJar {
+ configurations = [project.configurations.myShadowJar]
+}
+ {% endhighlight %}
+ </div>
+ <div class="tab-pane" id="gradle-settings">
+ {% highlight gradle %}
+rootProject.name = 'quickstart'
+ {% endhighlight %}
+ </div>
+ </div>
+ </div>
+
+ <div class="tab-pane" id="gradle-script">
+ {% highlight bash %}
+ $ curl https://flink.apache.org/q/gradle-quickstart.sh | bash
+ {% endhighlight %}
+ This allows you to <strong>name your newly created project</strong>. It will interactively ask
+ you for the project name, organization (also used for the package name), project version,
+ Scala and Flink version.
+ </div>
+</div>
+
+### Inspect Project
+
+There will be a new directory in your working directory based on the
+project name you provided, e.g. for `quickstart`:
+
+{% highlight bash %}
+$ tree quickstart/
+quickstart/
+├── README
+├── build.gradle
+├── settings.gradle
+└── src
+ └── main
+ ├── java
+ │ └── org
+ │ └── myorg
+ │ └── quickstart
+ │ ├── BatchJob.java
+ │ └── StreamingJob.java
+ └── resources
+ └── log4j.properties
+{% endhighlight %}
+
+The sample project is a __Gradle project__, which contains two classes: _StreamingJob_ and _BatchJob_ are the basic skeleton programs for a *DataStream* and *DataSet* program.
+The _main_ method is the entry point of the program, both for in-IDE testing/execution and for proper deployments.
+
+We recommend you __import this project into your IDE__ to develop and
+test it. IntelliJ IDEA supports Gradle projects after installing the `Gradle` plugin.
+Eclipse does so via the [Eclipse Buildship](https://projects.eclipse.org/projects/tools.buildship) plugin.
+You may also use [Gradle's IDE integration](https://docs.gradle.org/current/userguide/userguide.html#ide-integration)
+to create project files from Gradle.
+
+
+*A note to Mac OS X users*: The default JVM heapsize for Java may be too
--- End diff --
I think this does not only apply to Mac OS X users.
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194900211
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
--- End diff --
that's actually a good idea
---
[GitHub] flink issue #5900: [FLINK-9222][docs] add documentation for setting up Gradl...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/5900
I'm pretty sure I got all this covered by using a separate "configuration": `myShadowJar` - not the standard dependency configuration*, but does the trick:
Everything added to `myShadowJar` inside the `dependencies` gets added to the shaded jar (here: shadowJar), except for the explicit excludes which work with transitive dependencies as well (defined in the lines starting with `myShadowJar.exclude group:` where I included the same things as in the maven shade configuration of the quickstart). All user-code dependencies should be put into `myShadowJar` - maybe I should make this even more explicit in the gradle build file.
- nothing is relocated - there's stuff packed into the jar and other stuff that isn't, that's it :) (should be the same - I did compare the jar with the one from maven with and without the kafka connector dependency as a test)
- Flink core dependencies are excluded from the uber jar by not putting them into `myShadowJar`
- with the trick of using `myShadowJar`, not only is IntelliJ able to run the job and the tests, it also runs from commandline
* Unfortunately, I could not use the `shadow`/`compileOnly` dependency configurations which are standard for this in gradle because then the program would not run in IntelliJ or via `gradle run`. It would expect the environment to provide the dependencies which it does not there. Alternatives/fixes for this broke the transitive dependency exclusion which is, however, scheduled for some future version of the gradle shadow plugin. There a lot of enhancement requests in this regard, e.g. https://github.com/johnrengelman/shadow/issues/159
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194900846
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ //myShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
--- End diff --
yes, this example should be extended with a comment just like in the pom.xml file for Java: `<!-- Add connector dependencies here. They must be in the default scope (compile). -->`
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194902416
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ //myShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ testCompile "junit:junit:4.12"
--- End diff --
yes, this should probably be a commented-out example dependency as well
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r201965094
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -101,23 +111,210 @@ allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/refer
Some Eclipse bundles include that plugin by default, others require you
to install it manually.
-*A note to Mac OS X users*: The default JVM heapsize for Java may be too
+*Please note*: The default JVM heapsize for Java may be too
small for Flink. You have to manually increase it.
-In Eclipse, choose
-`Run Configurations -> Arguments` and write into the `VM Arguments`
-box: `-Xmx800m`.
+In Eclipse, choose `Run Configurations -> Arguments` and write into the `VM Arguments` box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.4'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '{{ site.version }}'
+ scalaBinaryVersion = '{{ site.scala_version }}'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "flinkShadowJar" configuration!
+configurations {
+ flinkShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ flinkShadowJar.exclude group: 'org.slf4j'
+ flinkShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ // Add connector dependencies here.
+ // They must be in the flinkShadowJar configuration in order to be included into the shaded jar.
+ //flinkShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ // Add test dependencies here.
+ // testCompile "junit:junit:4.12"
+}
+
+// make compileOnly dependencies available for tests:
+sourceSets {
+ main.runtimeClasspath += configurations.flinkShadowJar
--- End diff --
Thanks for pointing this out - you are indeed right and I fixed this one with the latest commits.
How about the rest of the PR? Do you think, this is good gradle style / does it work for you?
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194707580
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ //myShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
--- End diff --
What is this line good for? If it should be an example add a line comment.
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by k-mack <gi...@git.apache.org>.
Github user k-mack commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r202062325
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -101,23 +111,210 @@ allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/refer
Some Eclipse bundles include that plugin by default, others require you
to install it manually.
-*A note to Mac OS X users*: The default JVM heapsize for Java may be too
+*Please note*: The default JVM heapsize for Java may be too
small for Flink. You have to manually increase it.
-In Eclipse, choose
-`Run Configurations -> Arguments` and write into the `VM Arguments`
-box: `-Xmx800m`.
+In Eclipse, choose `Run Configurations -> Arguments` and write into the `VM Arguments` box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.4'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '{{ site.version }}'
+ scalaBinaryVersion = '{{ site.scala_version }}'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "flinkShadowJar" configuration!
+configurations {
+ flinkShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ flinkShadowJar.exclude group: 'org.slf4j'
+ flinkShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ // Add connector dependencies here.
+ // They must be in the flinkShadowJar configuration in order to be included into the shaded jar.
+ //flinkShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ // Add test dependencies here.
+ // testCompile "junit:junit:4.12"
+}
+
+// make compileOnly dependencies available for tests:
+sourceSets {
+ main.runtimeClasspath += configurations.flinkShadowJar
--- End diff --
I think the only thing left is to update the javadoc task's classpath to avoid JavaDoc errors, which would happen if authors reference classes associated with the flinkShadowJar configuration in their JavaDocs. I do something similar my in my build scripts for Hadoop applications:
```gradle
// Prevent JavaDoc errors by including the Hadoop dependencies on the javadoc classpath
javadoc.classpath += configurations.hadoopJar
```
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r201310579
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ //myShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ testCompile "junit:junit:4.12"
+}
+
+// make compileOnly dependencies available for tests:
+sourceSets {
+ main.runtimeClasspath += configurations.myShadowJar
+ main.runtimeClasspath += configurations.myShadowJar
+
+ test.compileClasspath += configurations.myShadowJar
+ test.runtimeClasspath += configurations.myShadowJar
+}
+
+jar {
+ manifest {
+ attributes 'Built-By': System.getProperty('user.name'),
+ 'Build-Jdk': System.getProperty('java.version')
+ }
+}
+
+shadowJar {
+ configurations = [project.configurations.myShadowJar]
+}
+ {% endhighlight %}
+ </div>
+ <div class="tab-pane" id="gradle-settings">
+ {% highlight gradle %}
+rootProject.name = 'quickstart'
+ {% endhighlight %}
+ </div>
+ </div>
+ </div>
+
+ <div class="tab-pane" id="gradle-script">
+ {% highlight bash %}
+ $ curl https://flink.apache.org/q/gradle-quickstart.sh | bash
+ {% endhighlight %}
+ This allows you to <strong>name your newly created project</strong>. It will interactively ask
+ you for the project name, organization (also used for the package name), project version,
+ Scala and Flink version.
+ </div>
+</div>
+
+### Inspect Project
+
+There will be a new directory in your working directory based on the
+project name you provided, e.g. for `quickstart`:
+
+{% highlight bash %}
+$ tree quickstart/
+quickstart/
+├── README
+├── build.gradle
+├── settings.gradle
+└── src
+ └── main
+ ├── java
+ │ └── org
+ │ └── myorg
+ │ └── quickstart
+ │ ├── BatchJob.java
+ │ └── StreamingJob.java
+ └── resources
+ └── log4j.properties
+{% endhighlight %}
+
+The sample project is a __Gradle project__, which contains two classes: _StreamingJob_ and _BatchJob_ are the basic skeleton programs for a *DataStream* and *DataSet* program.
+The _main_ method is the entry point of the program, both for in-IDE testing/execution and for proper deployments.
+
+We recommend you __import this project into your IDE__ to develop and
+test it. IntelliJ IDEA supports Gradle projects after installing the `Gradle` plugin.
+Eclipse does so via the [Eclipse Buildship](https://projects.eclipse.org/projects/tools.buildship) plugin.
--- End diff --
just checked - seems to be working (you need to manually select a gradle version >= 3.0 because the default gradle wrapper uses 2.14 which is too old) - I added appropriate comments into the fixup commit
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r201286036
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
--- End diff --
Thanks for this tip, unfortunately, this still leaves us with the inability to exclude transitive dependencies from the shaded jar, i.e.
```
flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
flinkShadowJar.exclude group: 'org.slf4j'
flinkShadowJar.exclude group: 'log4j'
```
Therefore, we still need the workaround with our own configuration for dependencies that go into the shaded jar.
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194708957
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
--- End diff --
Add doc variables here and the following line.
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194711334
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ //myShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ testCompile "junit:junit:4.12"
--- End diff --
This should not be here by default.
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194707286
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
--- End diff --
Call this flinkShadowJar? Or some more meaningful name.
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by twalthr <gi...@git.apache.org>.
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194708090
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ //myShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ testCompile "junit:junit:4.12"
+}
+
+// make compileOnly dependencies available for tests:
+sourceSets {
+ main.runtimeClasspath += configurations.myShadowJar
+ main.runtimeClasspath += configurations.myShadowJar
+
+ test.compileClasspath += configurations.myShadowJar
+ test.runtimeClasspath += configurations.myShadowJar
+}
+
+jar {
+ manifest {
+ attributes 'Built-By': System.getProperty('user.name'),
+ 'Build-Jdk': System.getProperty('java.version')
+ }
+}
+
+shadowJar {
+ configurations = [project.configurations.myShadowJar]
+}
+ {% endhighlight %}
+ </div>
+ <div class="tab-pane" id="gradle-settings">
+ {% highlight gradle %}
+rootProject.name = 'quickstart'
+ {% endhighlight %}
+ </div>
+ </div>
+ </div>
+
+ <div class="tab-pane" id="gradle-script">
+ {% highlight bash %}
+ $ curl https://flink.apache.org/q/gradle-quickstart.sh | bash
+ {% endhighlight %}
+ This allows you to <strong>name your newly created project</strong>. It will interactively ask
+ you for the project name, organization (also used for the package name), project version,
+ Scala and Flink version.
+ </div>
+</div>
+
+### Inspect Project
+
+There will be a new directory in your working directory based on the
+project name you provided, e.g. for `quickstart`:
+
+{% highlight bash %}
+$ tree quickstart/
+quickstart/
+├── README
+├── build.gradle
+├── settings.gradle
+└── src
+ └── main
+ ├── java
+ │ └── org
+ │ └── myorg
+ │ └── quickstart
+ │ ├── BatchJob.java
+ │ └── StreamingJob.java
+ └── resources
+ └── log4j.properties
+{% endhighlight %}
+
+The sample project is a __Gradle project__, which contains two classes: _StreamingJob_ and _BatchJob_ are the basic skeleton programs for a *DataStream* and *DataSet* program.
+The _main_ method is the entry point of the program, both for in-IDE testing/execution and for proper deployments.
+
+We recommend you __import this project into your IDE__ to develop and
+test it. IntelliJ IDEA supports Gradle projects after installing the `Gradle` plugin.
+Eclipse does so via the [Eclipse Buildship](https://projects.eclipse.org/projects/tools.buildship) plugin.
--- End diff --
Did you test the PR in Eclipse?
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by k-mack <gi...@git.apache.org>.
Github user k-mack commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r201746359
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -101,23 +111,210 @@ allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/refer
Some Eclipse bundles include that plugin by default, others require you
to install it manually.
-*A note to Mac OS X users*: The default JVM heapsize for Java may be too
+*Please note*: The default JVM heapsize for Java may be too
small for Flink. You have to manually increase it.
-In Eclipse, choose
-`Run Configurations -> Arguments` and write into the `VM Arguments`
-box: `-Xmx800m`.
+In Eclipse, choose `Run Configurations -> Arguments` and write into the `VM Arguments` box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.4'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '{{ site.version }}'
+ scalaBinaryVersion = '{{ site.scala_version }}'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "flinkShadowJar" configuration!
+configurations {
+ flinkShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ flinkShadowJar.exclude group: 'org.slf4j'
+ flinkShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ // Add connector dependencies here.
+ // They must be in the flinkShadowJar configuration in order to be included into the shaded jar.
+ //flinkShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ // Add test dependencies here.
+ // testCompile "junit:junit:4.12"
+}
+
+// make compileOnly dependencies available for tests:
+sourceSets {
+ main.runtimeClasspath += configurations.flinkShadowJar
--- End diff --
`main.runtimeClasspath` is set twice here in `sourceSets {}`. I think you meant for the first one to be `main.compileClasspath`. Sorry to jump in like this, but I'm interested in this PR :)
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194902217
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
--- End diff --
I'm not sure this is possible...
---
[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/5900#discussion_r194901682
--- Diff: docs/quickstart/java_api_quickstart.md ---
@@ -108,16 +118,201 @@ In Eclipse, choose
box: `-Xmx800m`.
In IntelliJ IDEA recommended way to change JVM options is from the `Help | Edit Custom VM Options` menu. See [this article](https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties) for details.
-## Build Project
+### Build Project
If you want to __build/package your project__, go to your project directory and
run the '`mvn clean package`' command.
You will __find a JAR file__ that contains your application, plus connectors and libraries
that you may have added as dependencies to the application: `target/<artifact-id>-<version>.jar`.
__Note:__ If you use a different class than *StreamingJob* as the application's main class / entry point,
-we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, the Flink
-can run time application from the JAR file without additionally specifying the main class.
+we recommend you change the `mainClass` setting in the `pom.xml` file accordingly. That way, Flink
+can run the application from the JAR file without additionally specifying the main class.
+
+## Gradle
+
+### Requirements
+
+The only requirements are working __Gradle 3.x__ (or higher) and __Java 8.x__ installations.
+
+### Create Project
+
+Use one of the following commands to __create a project__:
+
+<ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-example" data-toggle="tab"><strong>Gradle example</strong></a></li>
+ <li><a href="#gradle-script" data-toggle="tab">Run the <strong>quickstart script</strong></a></li>
+</ul>
+<div class="tab-content">
+ <div class="tab-pane active" id="gradle-example">
+
+ <ul class="nav nav-tabs" style="border-bottom: none;">
+ <li class="active"><a href="#gradle-build" data-toggle="tab"><tt>build.gradle</tt></a></li>
+ <li><a href="#gradle-settings" data-toggle="tab"><tt>settings.gradle</tt></a></li>
+ </ul>
+ <div class="tab-content">
+ <div class="tab-pane active" id="gradle-build">
+ {% highlight gradle %}
+buildscript {
+ repositories {
+ jcenter() // this applies only to the Gradle 'Shadow' plugin
+ }
+ dependencies {
+ classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
+ }
+}
+
+plugins {
+ id 'java'
+ id 'application'
+ // shadow plugin to produce fat JARs
+ id 'com.github.johnrengelman.shadow' version '2.0.3'
+}
+
+
+// artifact properties
+group = 'org.myorg.quickstart'
+version = '0.1-SNAPSHOT'
+mainClassName = 'org.myorg.quickstart.StreamingJob'
+description = """Flink Quickstart Job"""
+
+ext {
+ javaVersion = '1.8'
+ flinkVersion = '1.6-SNAPSHOT'
+ scalaBinaryVersion = '2.11'
+ slf4jVersion = '1.7.7'
+ log4jVersion = '1.2.17'
+}
+
+
+sourceCompatibility = javaVersion
+targetCompatibility = javaVersion
+tasks.withType(JavaCompile) {
+ options.encoding = 'UTF-8'
+}
+
+applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
+
+// declare where to find the dependencies of your project
+repositories {
+ mavenCentral()
+ maven { url "https://repository.apache.org/content/repositories/snapshots/" }
+}
+
+// NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
+// in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
+// shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
+// -> Explicitly define the // libraries we want to be included in the "myShadowJar" configuration!
+configurations {
+ myShadowJar // dependencies which go into the shadowJar
+
+ // always exclude these (also from transitive dependencies) since they are provided by Flink
+ myShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
+ myShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
+ myShadowJar.exclude group: 'org.slf4j'
+ myShadowJar.exclude group: 'log4j'
+}
+
+// declare the dependencies for your production and test code
+dependencies {
+ compile "org.apache.flink:flink-java:${flinkVersion}"
+ compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
+
+ //myShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
+
+ compile "log4j:log4j:${log4jVersion}"
+ compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
+
+ testCompile "junit:junit:4.12"
+}
+
+// make compileOnly dependencies available for tests:
+sourceSets {
+ main.runtimeClasspath += configurations.myShadowJar
+ main.runtimeClasspath += configurations.myShadowJar
+
+ test.compileClasspath += configurations.myShadowJar
+ test.runtimeClasspath += configurations.myShadowJar
+}
+
+jar {
+ manifest {
+ attributes 'Built-By': System.getProperty('user.name'),
+ 'Build-Jdk': System.getProperty('java.version')
+ }
+}
+
+shadowJar {
+ configurations = [project.configurations.myShadowJar]
+}
+ {% endhighlight %}
+ </div>
+ <div class="tab-pane" id="gradle-settings">
+ {% highlight gradle %}
+rootProject.name = 'quickstart'
+ {% endhighlight %}
+ </div>
+ </div>
+ </div>
+
+ <div class="tab-pane" id="gradle-script">
+ {% highlight bash %}
+ $ curl https://flink.apache.org/q/gradle-quickstart.sh | bash
+ {% endhighlight %}
+ This allows you to <strong>name your newly created project</strong>. It will interactively ask
+ you for the project name, organization (also used for the package name), project version,
+ Scala and Flink version.
+ </div>
+</div>
+
+### Inspect Project
+
+There will be a new directory in your working directory based on the
+project name you provided, e.g. for `quickstart`:
+
+{% highlight bash %}
+$ tree quickstart/
+quickstart/
+├── README
+├── build.gradle
+├── settings.gradle
+└── src
+ └── main
+ ├── java
+ │ └── org
+ │ └── myorg
+ │ └── quickstart
+ │ ├── BatchJob.java
+ │ └── StreamingJob.java
+ └── resources
+ └── log4j.properties
+{% endhighlight %}
+
+The sample project is a __Gradle project__, which contains two classes: _StreamingJob_ and _BatchJob_ are the basic skeleton programs for a *DataStream* and *DataSet* program.
+The _main_ method is the entry point of the program, both for in-IDE testing/execution and for proper deployments.
+
+We recommend you __import this project into your IDE__ to develop and
+test it. IntelliJ IDEA supports Gradle projects after installing the `Gradle` plugin.
+Eclipse does so via the [Eclipse Buildship](https://projects.eclipse.org/projects/tools.buildship) plugin.
+You may also use [Gradle's IDE integration](https://docs.gradle.org/current/userguide/userguide.html#ide-integration)
+to create project files from Gradle.
+
+
+*A note to Mac OS X users*: The default JVM heapsize for Java may be too
--- End diff --
I did not `git blame` to find the reason why this was added - since we're using the term "may", we can safely drop the restriction to MacOS X users - but then also from the other parts of this documentation.
---