You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by rmetzger <gi...@git.apache.org> on 2015/06/03 20:10:23 UTC
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/774
[FLINK-2092] Adjust documentation for print()/collect() semantics update
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink flink2092
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/774.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #774
----
commit 665fedd6bb21dc26ad6b5fdb7e8605038707671b
Author: Robert Metzger <rm...@apache.org>
Date: 2015-06-03T07:44:42Z
[FLINK-2092] Adjust documentation for print()/collect() semantics update"
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/774#discussion_r31703115
--- Diff: docs/apis/programming_guide.md ---
@@ -189,6 +187,10 @@ that creates the type information for Flink operations.
</div>
</div>
+
+
+#### Hadoop Dependency Versions
+
If you are using Flink together with Hadoop, the version of the dependency may vary depending on the
version of Hadoop (or more specifically, HDFS) that you want to use Flink with. Please refer to the
[downloads page]({{site.baseurl}}/downloads.html) for a list of available versions, and instructions
--- End diff --
this link is broken... it should refer to the project webpage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/774#discussion_r31703506
--- Diff: docs/apis/programming_guide.md ---
@@ -394,26 +382,66 @@ def write(outputFormat: FileOutputFormat[T],
writeMode: WriteMode = WriteMode.NO_OVERWRITE)
def print()
-{% endhighlight %}
-The last method is only useful for developing/debugging on a local machine,
-it will output the contents of the DataSet to standard output. (Note that in
-a cluster, the result goes to the standard out stream of the cluster nodes and ends
-up in the *.out* files of the workers).
-The first two do as the name suggests, the third one can be used to specify a
-custom data output format. Please refer
-to [Data Sinks](#data-sinks) for more information on writing to files and also
-about custom data output formats.
-
-Once you specified the complete program you need to call `execute` on
-the `ExecutionEnvironment`. This will either execute on your local
-machine or submit your program for execution on a cluster, depending on
-how you created the execution environment.
+def collect()
+{% endhighlight %}
</div>
</div>
+The first two methods (`writeAsText()` and `writeAsCsv()`) do as the name suggests, the third one
+can be used to specify a custom data output format. Please refer to [Data Sinks](#data-sinks) for
+more information on writing to files and also about custom data output formats.
+
+The `print()` method is useful for developing/debugging. It will output the contents of the DataSet
+to standard output (on the JVM starting the Flink execution). **NOTE** The behavior of the `print()`
+method changed with Flink 0.9.x. Before it was printing to the log file of the workers, now its
+sending the DataSet results to the client and printing the results there.
+
+`collect()` allows to retrieve the DataSet from the cluster to the local JVM. The `collect()` method
+will return a `List` containing the elements.
+
+Both `print()` and `collect()` will trigger the execution of the program.
--- End diff --
Let's be explicit. Something along the lines of: ...trigger execution of the program. You don't need a further call to `execute()`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/774#discussion_r31703262
--- Diff: docs/apis/programming_guide.md ---
@@ -283,7 +284,7 @@ This will create a new DataSet by converting every String in the original
set to an Integer. For more information and a list of all the transformations,
please refer to [Transformations](#transformations).
-Once you have a DataSet that needs to be written to disk you call one
+Once you have a DataSet that needs to be processed further you can call one
--- End diff --
I think I understand why you changes this to "processed further", but it does not make sense with print(). ;)
Maybe: once you want to write or read a result?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/774#discussion_r31713906
--- Diff: docs/apis/programming_guide.md ---
@@ -394,26 +382,66 @@ def write(outputFormat: FileOutputFormat[T],
writeMode: WriteMode = WriteMode.NO_OVERWRITE)
def print()
-{% endhighlight %}
-The last method is only useful for developing/debugging on a local machine,
-it will output the contents of the DataSet to standard output. (Note that in
-a cluster, the result goes to the standard out stream of the cluster nodes and ends
-up in the *.out* files of the workers).
-The first two do as the name suggests, the third one can be used to specify a
-custom data output format. Please refer
-to [Data Sinks](#data-sinks) for more information on writing to files and also
-about custom data output formats.
-
-Once you specified the complete program you need to call `execute` on
-the `ExecutionEnvironment`. This will either execute on your local
-machine or submit your program for execution on a cluster, depending on
-how you created the execution environment.
+def collect()
+{% endhighlight %}
</div>
</div>
+The first two methods (`writeAsText()` and `writeAsCsv()`) do as the name suggests, the third one
+can be used to specify a custom data output format. Please refer to [Data Sinks](#data-sinks) for
+more information on writing to files and also about custom data output formats.
+
+The `print()` method is useful for developing/debugging. It will output the contents of the DataSet
--- End diff --
Yes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/774#discussion_r31703697
--- Diff: docs/apis/programming_guide.md ---
@@ -394,26 +382,66 @@ def write(outputFormat: FileOutputFormat[T],
writeMode: WriteMode = WriteMode.NO_OVERWRITE)
def print()
-{% endhighlight %}
-The last method is only useful for developing/debugging on a local machine,
-it will output the contents of the DataSet to standard output. (Note that in
-a cluster, the result goes to the standard out stream of the cluster nodes and ends
-up in the *.out* files of the workers).
-The first two do as the name suggests, the third one can be used to specify a
-custom data output format. Please refer
-to [Data Sinks](#data-sinks) for more information on writing to files and also
-about custom data output formats.
-
-Once you specified the complete program you need to call `execute` on
-the `ExecutionEnvironment`. This will either execute on your local
-machine or submit your program for execution on a cluster, depending on
-how you created the execution environment.
+def collect()
+{% endhighlight %}
</div>
</div>
+The first two methods (`writeAsText()` and `writeAsCsv()`) do as the name suggests, the third one
+can be used to specify a custom data output format. Please refer to [Data Sinks](#data-sinks) for
+more information on writing to files and also about custom data output formats.
+
+The `print()` method is useful for developing/debugging. It will output the contents of the DataSet
+to standard output (on the JVM starting the Flink execution). **NOTE** The behavior of the `print()`
+method changed with Flink 0.9.x. Before it was printing to the log file of the workers, now its
+sending the DataSet results to the client and printing the results there.
+
+`collect()` allows to retrieve the DataSet from the cluster to the local JVM. The `collect()` method
+will return a `List` containing the elements.
+
+Both `print()` and `collect()` will trigger the execution of the program.
+
+
+**NOTE** `print()` and `collect()` retrieve the data from the cluster to the client. Currently,
+the data sizes you can retrieve with `collect()` are limited due to our RPC system. It is not advised
+to collect DataSets larger than 10MBs.
+
+
+Once you specified the complete program you need to **trigger the program execution**. You can call
+`execute()` directly on the `ExecutionEnviroment` or you implicitly trigger the execution with
+`collect()` or `print()`.
+Depending on the type of the `ExecutionEnvironment` the execution will be triggered on your local
+machine or submit your program for execution on a cluster.
+
+Note that you can not call both `print()` (or `collect()`) and `execute()` at the end of program.
+
+The `execute()` method is returning the `JobExecutionResult`, including execution times and
+accumulator results. `print()` and `collect()` are not returning the result, but it can be
+accessed from the `getLastJobExecutionResult()` method.
+
+
+[Back to top](#top)
+
+
+DataSet abstraction
+---------------
+
+The batch processing APIs of Flink are centered around the `DataSet` abstraction. A `DataSet` is only
+an abstract representation of a set of data that can contain duplicates.
+
+Also note that Flink is not always physically creating (materializing) each DataSet at runtime. This
+depends on the used runtime, the configuration and optimizer decisions.
+
+The Flink runtime is usually not materializing the DataSets because it is using a streaming runtime model.
+DataSets are only materialized to avoid distributed deadlocks (usually at points where the data flow
--- End diff --
Can skip the `usally`... and I would say: `at points where the data flow graph branches out and joins again later` instead of `out to join`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/774#discussion_r31703637
--- Diff: docs/apis/programming_guide.md ---
@@ -394,26 +382,66 @@ def write(outputFormat: FileOutputFormat[T],
writeMode: WriteMode = WriteMode.NO_OVERWRITE)
def print()
-{% endhighlight %}
-The last method is only useful for developing/debugging on a local machine,
-it will output the contents of the DataSet to standard output. (Note that in
-a cluster, the result goes to the standard out stream of the cluster nodes and ends
-up in the *.out* files of the workers).
-The first two do as the name suggests, the third one can be used to specify a
-custom data output format. Please refer
-to [Data Sinks](#data-sinks) for more information on writing to files and also
-about custom data output formats.
-
-Once you specified the complete program you need to call `execute` on
-the `ExecutionEnvironment`. This will either execute on your local
-machine or submit your program for execution on a cluster, depending on
-how you created the execution environment.
+def collect()
+{% endhighlight %}
</div>
</div>
+The first two methods (`writeAsText()` and `writeAsCsv()`) do as the name suggests, the third one
+can be used to specify a custom data output format. Please refer to [Data Sinks](#data-sinks) for
+more information on writing to files and also about custom data output formats.
+
+The `print()` method is useful for developing/debugging. It will output the contents of the DataSet
+to standard output (on the JVM starting the Flink execution). **NOTE** The behavior of the `print()`
+method changed with Flink 0.9.x. Before it was printing to the log file of the workers, now its
+sending the DataSet results to the client and printing the results there.
+
+`collect()` allows to retrieve the DataSet from the cluster to the local JVM. The `collect()` method
+will return a `List` containing the elements.
+
+Both `print()` and `collect()` will trigger the execution of the program.
+
+
+**NOTE** `print()` and `collect()` retrieve the data from the cluster to the client. Currently,
+the data sizes you can retrieve with `collect()` are limited due to our RPC system. It is not advised
+to collect DataSets larger than 10MBs.
+
+
+Once you specified the complete program you need to **trigger the program execution**. You can call
+`execute()` directly on the `ExecutionEnviroment` or you implicitly trigger the execution with
+`collect()` or `print()`.
+Depending on the type of the `ExecutionEnvironment` the execution will be triggered on your local
+machine or submit your program for execution on a cluster.
+
+Note that you can not call both `print()` (or `collect()`) and `execute()` at the end of program.
+
+The `execute()` method is returning the `JobExecutionResult`, including execution times and
+accumulator results. `print()` and `collect()` are not returning the result, but it can be
+accessed from the `getLastJobExecutionResult()` method.
+
+
+[Back to top](#top)
+
+
+DataSet abstraction
+---------------
+
+The batch processing APIs of Flink are centered around the `DataSet` abstraction. A `DataSet` is only
+an abstract representation of a set of data that can contain duplicates.
+
+Also note that Flink is not always physically creating (materializing) each DataSet at runtime. This
+depends on the used runtime, the configuration and optimizer decisions.
+
+The Flink runtime is usually not materializing the DataSets because it is using a streaming runtime model.
--- End diff --
We could formulate this more positive: The Flink runtime does not need to always materialize...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/774#discussion_r31703387
--- Diff: docs/apis/programming_guide.md ---
@@ -394,26 +382,66 @@ def write(outputFormat: FileOutputFormat[T],
writeMode: WriteMode = WriteMode.NO_OVERWRITE)
def print()
-{% endhighlight %}
-The last method is only useful for developing/debugging on a local machine,
-it will output the contents of the DataSet to standard output. (Note that in
-a cluster, the result goes to the standard out stream of the cluster nodes and ends
-up in the *.out* files of the workers).
-The first two do as the name suggests, the third one can be used to specify a
-custom data output format. Please refer
-to [Data Sinks](#data-sinks) for more information on writing to files and also
-about custom data output formats.
-
-Once you specified the complete program you need to call `execute` on
-the `ExecutionEnvironment`. This will either execute on your local
-machine or submit your program for execution on a cluster, depending on
-how you created the execution environment.
+def collect()
+{% endhighlight %}
</div>
</div>
+The first two methods (`writeAsText()` and `writeAsCsv()`) do as the name suggests, the third one
+can be used to specify a custom data output format. Please refer to [Data Sinks](#data-sinks) for
+more information on writing to files and also about custom data output formats.
+
+The `print()` method is useful for developing/debugging. It will output the contents of the DataSet
--- End diff --
Stephan added `printOnTaskManager`... can you add this here as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/774#discussion_r31703441
--- Diff: docs/apis/programming_guide.md ---
@@ -394,26 +382,66 @@ def write(outputFormat: FileOutputFormat[T],
writeMode: WriteMode = WriteMode.NO_OVERWRITE)
def print()
-{% endhighlight %}
-The last method is only useful for developing/debugging on a local machine,
-it will output the contents of the DataSet to standard output. (Note that in
-a cluster, the result goes to the standard out stream of the cluster nodes and ends
-up in the *.out* files of the workers).
-The first two do as the name suggests, the third one can be used to specify a
-custom data output format. Please refer
-to [Data Sinks](#data-sinks) for more information on writing to files and also
-about custom data output formats.
-
-Once you specified the complete program you need to call `execute` on
-the `ExecutionEnvironment`. This will either execute on your local
-machine or submit your program for execution on a cluster, depending on
-how you created the execution environment.
+def collect()
+{% endhighlight %}
</div>
</div>
+The first two methods (`writeAsText()` and `writeAsCsv()`) do as the name suggests, the third one
+can be used to specify a custom data output format. Please refer to [Data Sinks](#data-sinks) for
+more information on writing to files and also about custom data output formats.
+
+The `print()` method is useful for developing/debugging. It will output the contents of the DataSet
+to standard output (on the JVM starting the Flink execution). **NOTE** The behavior of the `print()`
+method changed with Flink 0.9.x. Before it was printing to the log file of the workers, now its
+sending the DataSet results to the client and printing the results there.
+
+`collect()` allows to retrieve the DataSet from the cluster to the local JVM. The `collect()` method
--- End diff --
I don't know if this a German thing. I also catch myself doing this alot... but it does not *allow* you to retrieve it... it just retrieves it. ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger closed the pull request at:
https://github.com/apache/flink/pull/774
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/774#issuecomment-108795427
Thanks so much for updating the docs and starting the discussion on the ML. If you don't have time to adress my comments, just give your feedback and I can take care of it. :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/774#issuecomment-108863972
I addressed all your comments. Thank you for the review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2092] Adjust documentation for print()/...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/774#issuecomment-109802799
I've merged the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---