You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by aljoscha <gi...@git.apache.org> on 2016/03/16 15:06:55 UTC
[GitHub] flink pull request: [FLINK-3174] Add MergingWindowAssigner and Ses...
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1802
[FLINK-3174] Add MergingWindowAssigner and SessionWindows
This introduces MergingWindowAssigner, an extension of WindowAssigner
that can merge windows. When using a MergingWindowAssigner the
WindowOperator eagerly merges windows when processing elements.
For keeping track of in-flight windows and for merging windows this adds
MergingWindowSet, this keeps track of windows per key.
Only when using a WindowAssigners is the more costly merging logic used
in the WindowOperator.
For triggers there is new method Trigger.onMerge() that notifies the
trigger of the new merged window. This allows the trigger to set a timer
for the newly merged window.
This also adds AbstractStateBackend.mergePartitionedStates for merging
state of several source namespaces into a target namespace. This is only
possible for the newly introduced MergingState which is an extension of
AppendingState. Only ReducingState and ListState are MergingState while
FoldingState is now an AppendingState.
This enables proper support for session windows.
This also adds the SessionWindows window assigner and adapts an existing
session example and adds test cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink window-sessions-eager
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1802.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1802
----
commit bda4e019cc464f411a2b62eaeab41a87f3961a21
Author: Aljoscha Krettek <al...@gmail.com>
Date: 2015-12-15T16:37:48Z
[FLINK-3174] Add MergingWindowAssigner and SessionWindows
This introduces MergingWindowAssigner, an extension of WindowAssigner
that can merge windows. When using a MergingWindowAssigner the
WindowOperator eagerly merges windows when processing elements.
For keeping track of in-flight windows and for merging windows this adds
MergingWindowSet, this keeps track of windows per key.
Only when using a WindowAssigners is the more costly merging logic used
in the WindowOperator.
For triggers there is new method Trigger.onMerge() that notifies the
trigger of the new merged window. This allows the trigger to set a timer
for the newly merged window.
This also adds AbstractStateBackend.mergePartitionedStates for merging
state of several source namespaces into a target namespace. This is only
possible for the newly introduced MergingState which is an extension of
AppendingState. Only ReducingState and ListState are MergingState while
FoldingState is now an AppendingState.
This enables proper support for session windows.
This also adds the SessionWindows window assigner and adapts an existing
session example and adds test cases.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3174] Add MergingWindowAssigner and Ses...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1802#issuecomment-205302445
I did the changes, I introduced `canMerge()` and added a default `onMerge()` that throws a `RuntimeExeption` in `Trigger`. This way we don't break the API for already existing user triggers.
If there are no objections I will also merge the removal of the non-keyed window operator that this PR is based on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3174] Add MergingWindowAssigner and Ses...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1802#issuecomment-205294872
Very nice work.
After an offline chat with @aljoscha , we concluded to slightly adjust the `Trigger` interface to make it simpler for people to develop triggers.
Other than that, this is good to merge...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3174] Add MergingWindowAssigner and Ses...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1802
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3174] Add MergingWindowAssigner and Ses...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1802#issuecomment-205384005
Concerning the removal of the non-keyed window operator: Does that mean that the user functions now see the dummy key?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3174] Add MergingWindowAssigner and Ses...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1802#issuecomment-205394604
No, we introduced the `InternalWindowFunction` before 1.0.0 to decouple the user `WindowFunction` from the window operator implementation. There are now special `InternalWindowFunctions` that don't forward the key to the user function. The API for the user does not change, there an `AllWindowFunction` is used, as before. And that function does not get a key.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---