You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by aljoscha <gi...@git.apache.org> on 2016/01/22 16:22:58 UTC
[GitHub] flink pull request: [FLINK-3178] Don't Emit In-Flight Windows When...
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1542
[FLINK-3178] Don't Emit In-Flight Windows When Closing Window Operator
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink window-no-final-emit
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1542.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1542
----
commit 2f4a15491cc7d307deb1f59d26980df72307b914
Author: Aljoscha Krettek <al...@gmail.com>
Date: 2016-01-18T12:25:03Z
[FLINK-3178] Don't Emit In-Flight Windows When Closing Window Operator
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3178] Don't Emit In-Flight Windows When...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1542
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3178] Don't Emit In-Flight Windows When...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1542#issuecomment-176181849
Merging this...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3178] Don't Emit In-Flight Windows When...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1542#discussion_r50709674
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/complex/ComplexIntegrationTest.java ---
@@ -214,6 +214,7 @@ public void complexIntegrationTest2() throws Exception {
@SuppressWarnings("unchecked")
@Test
+ @Ignore
--- End diff --
Is this an artifact, or does this test now become non-deterministic?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3178] Don't Emit In-Flight Windows When...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1542#issuecomment-174547617
Looks good, with one inline question.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3178] Don't Emit In-Flight Windows When...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/1542#discussion_r50720691
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/complex/ComplexIntegrationTest.java ---
@@ -214,6 +214,7 @@ public void complexIntegrationTest2() throws Exception {
@SuppressWarnings("unchecked")
@Test
+ @Ignore
--- End diff --
It's called `ComplexIntegrationTest` for a reason :smiley: I would either remove it or add a comment why you ignored the test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3178] Don't Emit In-Flight Windows When...
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the pull request:
https://github.com/apache/flink/pull/1542#issuecomment-174439774
This seems to be straightforward :) Looks good :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3178] Don't Emit In-Flight Windows When...
Posted by wenlonglwl <gi...@git.apache.org>.
Github user wenlonglwl commented on the pull request:
https://github.com/apache/flink/pull/1542#issuecomment-190586930
@aljoscha @StephanEwen This would not work for finite stream, data of in-flight window would lost, when the stream finished. Can this clear window operator changed to be configurable. in our production environment, window is used to do some aggregation, and no lost of data is acceptable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3178] Don't Emit In-Flight Windows When...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1542#discussion_r50714080
--- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/complex/ComplexIntegrationTest.java ---
@@ -214,6 +214,7 @@ public void complexIntegrationTest2() throws Exception {
@SuppressWarnings("unchecked")
@Test
+ @Ignore
--- End diff --
The test depended on the behavior that closing emits in-flight windows. Actually, the window fires only once, and the result of this firing is even filtered by the filter. So the count window is useless. I can also remove the test. (Or the complete suite of tests since I mostly don't know what they are doing/testing.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3178] Don't Emit In-Flight Windows When...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1542#issuecomment-190650790
Hi @wenlonglwl,
you are right about this. We are thinking about how we can change the system so that finite sources can emit a final Long.MAX_VALUE watermark when they are finished. When writing a custom finite source you can always emit a Long.MAX_VALUE watermark yourself, once you know that the source is finished.
Sorry for the inconvenience this has caused, we first wanted to make this right for infinite sources but I see that you also have a valid use case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---