You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by nikste <gi...@git.apache.org> on 2015/05/20 14:01:52 UTC
[GitHub] flink pull request: Eager print
GitHub user nikste opened a pull request:
https://github.com/apache/flink/pull/699
Eager print
Uses collect in print function now.
No need for calling execute() after print().
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nikste/flink eager_print
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/699.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #699
----
commit c7c860c0b3bd4d8d183b61425dde33fbea79fb6a
Author: Aljoscha Krettek <al...@gmail.com>
Date: 2015-04-28T08:52:29Z
[hotfix] Change print() to eager execution
print() now uses collect() internally
commit 5e23864019436380994b0a117b166889ebd960d9
Author: Nikolaas Steenbergen <ni...@googlemail.com>
Date: 2015-05-05T08:49:54Z
Changed tests and examples for not failing with "eager" print method
Added lastJobExecutionResult for getting the result of the last execution, when executing "eager" print
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30712602
--- Diff: flink-staging/flink-tez/src/test/java/org/apache/flink/tez/test/WordCountITCase.java ---
@@ -37,7 +37,7 @@ protected void preSubmit() throws Exception {
@Override
protected void postSubmit() throws Exception {
- compareResultsByLinesInMemory(WordCountData.COUNTS, resultPath);
+ compareResultsByLinesInMemory(WordCountData.COUNTS_AS_TUPLES, resultPath);
--- End diff --
Followup of adjusted WordCount example
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30710708
--- Diff: flink-optimizer/src/test/java/org/apache/flink/optimizer/custompartition/CustomPartitioningGlobalOptimizationTest.java ---
@@ -57,8 +58,8 @@ public void testJoinReduceCombination() {
.withPartitioner(partitioner);
joined.groupBy(1).withPartitioner(partitioner)
- .reduceGroup(new IdentityGroupReducerCombinable<Tuple3<Long,Long,Long>>())
- .print();
+ .reduceGroup(new IdentityGroupReducer<Tuple3<Long,Long,Long>>())
--- End diff --
The semantics of the tests should not be changed. This needs to be changed back to the original `IdentityGroupReducerCombinable`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/699
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:
https://github.com/apache/flink/pull/699#issuecomment-104561157
Congratulations on your first contribution @nikste!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30710903
--- Diff: flink-optimizer/src/test/java/org/apache/flink/optimizer/custompartition/GroupingPojoTranslationTest.java ---
@@ -176,19 +177,19 @@ public void testCustomPartitioningTupleInvalidType() {
fail(e.getMessage());
}
}
-
+
@Test
public void testCustomPartitioningTupleInvalidTypeSorted() {
try {
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
+
DataSet<Pojo3> data = env.fromElements(new Pojo3())
.rebalance().setParallelism(4);
-
--- End diff --
Many whitespace reformattings here. If you use IntelliJ, please set it to not auto-trim whitespaces. It blows up the diffs and distracts from actual changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30711657
--- Diff: flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java ---
@@ -73,13 +73,14 @@ public static void main(String[] args) throws Exception {
// emit result
if(fileOutput) {
- counts.writeAsCsv(outputPath, "\n", " ");
+ counts.writeAsText(outputPath);
--- End diff --
Can you roll this back to "writeAsCsv"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30710432
--- Diff: flink-java/src/main/java/org/apache/flink/api/java/DataSet.java ---
@@ -1357,11 +1362,16 @@ public long count() throws Exception {
/**
* Writes a DataSet to the standard error stream (stderr).<br/>
* For each element of the DataSet the result of {@link Object#toString()} is written.
- *
- * @return The DataSink that writes the DataSet.
*/
- public DataSink<T> printToErr() {
- return output(new PrintingOutputFormat<T>(true));
+ public void printToErr() throws Exception{
+ try {
--- End diff --
See comment above
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/699#issuecomment-103925741
I just noticed this test case:
`flink-tests/src/test/scala/org/apache/flink/api/scala/DeltaIterationSanityCheckTest.scala`
The programs in this test are never meant to execute, so we need to swap the `print()` statements with some other lazy output statement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30710754
--- Diff: flink-optimizer/src/test/java/org/apache/flink/optimizer/custompartition/GroupingKeySelectorTranslationTest.java ---
@@ -84,8 +85,8 @@ public void testCustomPartitioningKeySelectorGroupReduce() {
data.groupBy(new TestKeySelector<Tuple2<Integer,Integer>>())
.withPartitioner(new TestPartitionerInt())
- .reduceGroup(new IdentityGroupReducerCombinable<Tuple2<Integer,Integer>>())
- .print();
+ .reduceGroup(new IdentityGroupReducer<Tuple2<Integer,Integer>>())
--- End diff --
Also here, change back to the original function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30712581
--- Diff: flink-tests/src/test/java/org/apache/flink/test/exampleJavaPrograms/WordCountITCase.java ---
@@ -36,7 +36,7 @@ protected void preSubmit() throws Exception {
@Override
protected void postSubmit() throws Exception {
- compareResultsByLinesInMemory(WordCountData.COUNTS, resultPath);
+ compareResultsByLinesInMemory(WordCountData.COUNTS_AS_TUPLES, resultPath);
--- End diff --
Followup of adjusted WordCount example
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/699#issuecomment-103882937
Okay, the issue with print in test cases seems to be fixed in the second commit, that looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30710944
--- Diff: flink-optimizer/src/test/java/org/apache/flink/optimizer/java/PartitionOperatorTest.java ---
@@ -49,8 +50,8 @@ public void testPartitionOperatorPreservesFields() {
public int partition(Long key, int numPartitions) { return key.intValue(); }
}, 1)
.groupBy(1)
- .reduceGroup(new IdentityGroupReducerCombinable<Tuple2<Long,Long>>())
- .print();
+ .reduceGroup(new IdentityGroupReducer<Tuple2<Long,Long>>())
--- End diff --
Also here, change back to the original function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30710598
--- Diff: flink-optimizer/pom.xml ---
@@ -58,6 +58,12 @@ under the License.
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
+ <dependency>
--- End diff --
This dependency should be removed, it should be unnecessary. If it is actually necessary, then there is an issue somewhere else ;-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30710761
--- Diff: flink-optimizer/src/test/java/org/apache/flink/optimizer/custompartition/GroupingKeySelectorTranslationTest.java ---
@@ -115,8 +116,8 @@ public void testCustomPartitioningKeySelectorGroupReduceSorted() {
data.groupBy(new TestKeySelector<Tuple3<Integer,Integer,Integer>>())
.withPartitioner(new TestPartitionerInt())
.sortGroup(new TestKeySelector<Tuple3<Integer, Integer, Integer>>(), Order.ASCENDING)
- .reduceGroup(new IdentityGroupReducerCombinable<Tuple3<Integer,Integer,Integer>>())
- .print();
+ .reduceGroup(new IdentityGroupReducer<Tuple3<Integer,Integer,Integer>>())
--- End diff --
Also here, change back to the original function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30710411
--- Diff: flink-java/src/main/java/org/apache/flink/api/java/DataSet.java ---
@@ -1336,11 +1335,17 @@ public long count() throws Exception {
/**
* Writes a DataSet to the standard output stream (stdout).<br/>
* For each element of the DataSet the result of {@link Object#toString()} is written.
- *
- * @return The DataSink that writes the DataSet.
+ * This triggers execute() automatically.
*/
- public DataSink<T> print() {
- return output(new PrintingOutputFormat<T>(false));
+ public void print() throws Exception{
+ try {
--- End diff --
How about removing this try/catch block?
We are getting a very deep nesting of exceptions already. Not sure this wrapping exception adds any information. Why not let the "ProgramInvokationFailed" exception from the execution propagate?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/699#issuecomment-104699134
+1!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30712611
--- Diff: flink-staging/flink-tachyon/src/test/java/org/apache/flink/tachyon/TachyonFileSystemWrapperTest.java ---
@@ -141,8 +141,8 @@ public void testTachyon() {
IOUtils.copy(inStream, writer);
String resultString = writer.toString();
- Assert.assertEquals("hello 10\n" +
- "tachyon 10\n", resultString);
+ Assert.assertEquals("(hello,10)\n" +
--- End diff --
Followup of adjusted WordCount example
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/699#discussion_r30712622
--- Diff: flink-staging/flink-tachyon/src/test/java/org/apache/flink/tachyon/HDFSTest.java ---
@@ -110,8 +110,8 @@ public void testHDFS() {
IOUtils.copy(inStream, writer);
String resultString = writer.toString();
- Assert.assertEquals("hdfs 10\n" +
- "hello 10\n", resultString);
+ Assert.assertEquals("(hdfs,10)\n" +
--- End diff --
Followup of adjusted WordCount example
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Eager print
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/699#issuecomment-104240529
Looks good now. Will merge this.
Thank you for the contribution!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---