You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by markreddy <gi...@git.apache.org> on 2016/05/11 09:03:43 UTC
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
GitHub user markreddy opened a pull request:
https://github.com/apache/flink/pull/1982
[FLINK-3852] update the quickstart module to include a streaming skeleton
- Added skeleton StreamingJob
- Moved Job to BatchJob
- Commented out transformers for the mainClass setting with a guide to uncomment
- Updated java and scala docs with a guide on how to run via the cli
- Updated the site docs to include all four sample classes
- Tidied up SocketTextStreamWordCount
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markreddy/flink FLINK-3852-quickstart-streaming
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1982.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1982
----
commit 6cb9a098a8a4dac6a1ac2484aa00efbd41b54ebe
Author: markreddy <ma...@gmail.com>
Date: 2016-05-11T08:55:19Z
[FLINK-3852] added skeleton StreamingJob, moved Job to BatchJob, commented out transformers for the mainClass setting, tidied up SocketTextStreamWordCount and updated docs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/1982#discussion_r62818792
--- Diff: flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/BatchJob.java ---
@@ -32,28 +32,29 @@
* mvn clean package
* in the projects root directory.
* You will find the jar in
- * target/flink-quickstart-0.1-SNAPSHOT-Sample.jar
- *
+ * target/flink-quickstart-0.1-SNAPSHOT.jar
+ * From the cli you can then run
+ * ./bin/flink run -c <BatchJob> <quickstart jar>
*/
-public class Job {
+public class BatchJob {
public static void main(String[] args) throws Exception {
- // set up the execution environment
+ // set up the batch execution environment
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
/**
* Here, you can start creating your execution plan for Flink.
*
* Start with getting some data from the environment, like
- * env.readTextFile(textPath);
+ * env.readTextFile(textPath);
--- End diff --
I think this space was intended formatting, like the below .filter(), .flatMap() etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the pull request:
https://github.com/apache/flink/pull/1982#issuecomment-218413341
Made comments on some rather trivial javadoc / comment related stuff, overall the PR looks good!
I like how you've also made effort to tidy up some details in the original code too :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/1982#discussion_r62820828
--- Diff: flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/BatchJob.java ---
@@ -32,28 +32,29 @@
* mvn clean package
* in the projects root directory.
* You will find the jar in
- * target/flink-quickstart-0.1-SNAPSHOT-Sample.jar
- *
+ * target/flink-quickstart-0.1-SNAPSHOT.jar
+ * From the cli you can then run
+ * ./bin/flink run -c <BatchJob> <quickstart jar>
--- End diff --
Perhaps we can add a link to the cli introduction?
https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/cli.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1982#issuecomment-219815351
No, names are OK.
Just thought `DataSetJob` / `DataStreamJob` would be more consistent with API and documentation.
We can update the names later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/1982#discussion_r62816955
--- Diff: flink-quickstart/flink-quickstart-scala/src/main/resources/archetype-resources/src/main/scala/StreamingJob.scala ---
@@ -0,0 +1,69 @@
+package ${package}
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.flink.api.scala._
+
+/**
+ * Skeleton for a Flink Streaming Job.
+ *
+ * For a full example of a Flink Streaming Job, see the SocketTextStreamWordCount.java
+ * file in the same package/directory or have a look at the website.
+ *
+ * You can also generate a .jar file that you can submit on your Flink
+ * cluster. Just type
+ * {{{
+ * mvn clean package
+ * }}}
+ * in the projects root directory. You will find the jar in
+ * target/flink-quickstart-0.1-SNAPSHOT.jar
+ * From the cli you can then run
+ * {{{
+ * ./bin/flink run -c <StreamingJob> <quickstart jar>
+ * }}}
+ */
+object StreamingJob {
+ def main(args: Array[String]) {
+ // set up the streaming execution environment
+ val env = StreamExecutionEnvironment.getExecutionEnvironment
+
+ /**
+ * Here, you can start creating your execution plan for Flink.
+ *
+ * Start with getting some data from the environment, like
+ * env.readTextFile(textPath);
+ *
+ * then, transform the resulting DataSet[String] using operations
--- End diff --
Mistype here, DataStream[String] ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/1982#discussion_r62817913
--- Diff: flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/BatchJob.java ---
@@ -32,28 +32,29 @@
* mvn clean package
* in the projects root directory.
* You will find the jar in
- * target/flink-quickstart-0.1-SNAPSHOT-Sample.jar
- *
+ * target/flink-quickstart-0.1-SNAPSHOT.jar
--- End diff --
Sorry, I misunderstood. The naming of the jar here depends on the user.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1982#issuecomment-219814154
Merging this...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1982#issuecomment-219814341
Thanks for reworking the quickstart jobs @markreddy. Change look good to me.
I am not sure about the naming of the jobs. Should we use `DataSetJob` instead of `BatchJob` and `DataStreamJob` instead of `StreamingJob`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1982
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by markreddy <gi...@git.apache.org>.
Github user markreddy commented on the pull request:
https://github.com/apache/flink/pull/1982#issuecomment-219230214
Thanks for the review @tzulitai I've pushed fixes for all your comments :thumbsup:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/1982#discussion_r62818171
--- Diff: flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/BatchJob.java ---
@@ -32,28 +32,29 @@
* mvn clean package
* in the projects root directory.
* You will find the jar in
- * target/flink-quickstart-0.1-SNAPSHOT-Sample.jar
- *
+ * target/flink-quickstart-0.1-SNAPSHOT.jar
+ * From the cli you can then run
+ * ./bin/flink run -c <BatchJob> <quickstart jar>
--- End diff --
1 Excessive space after '-c'.
Also, maybe this would be more specific: `./bin/flink run -c ${package}.BatchJob target/flink-quickstart-0.1-SNAPSHOT.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1982#issuecomment-219713912
I like the pull request! (You updated the docs, the PR describes the changes).
+1 to merge
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:
https://github.com/apache/flink/pull/1982#discussion_r62816359
--- Diff: flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/src/main/java/BatchJob.java ---
@@ -32,28 +32,29 @@
* mvn clean package
* in the projects root directory.
* You will find the jar in
- * target/flink-quickstart-0.1-SNAPSHOT-Sample.jar
- *
+ * target/flink-quickstart-0.1-SNAPSHOT.jar
--- End diff --
Should we update the version in this line too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3852] update the quickstart module to i...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1982#issuecomment-219814739
Race condition ;-)
@fhueske I have locally merged this already. To me the names are good, I would keep them unless you feel strongly about the names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---