You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by ggevay <gi...@git.apache.org> on 2015/06/22 14:00:29 UTC

[GitHub] flink pull request: [FLINK-2143] Added ReduceFunctionWithInverse

GitHub user ggevay opened a pull request:

    https://github.com/apache/flink/pull/856

    [FLINK-2143] Added ReduceFunctionWithInverse

    (The first commit is the same as the second commit in PR 684)
    
    I didn't add an overload to reduce, but instead created ReduceFunctionWithInverse as a descendant of ReduceFunction. It has an invReduce method, which should be the inverse of reduce. getWindowBuffer checks if `transformation.getUDF() instanceof ReduceFunctionWithInverse` and creates an InversePreReducer in this case (except when the policy is tumbling or jumping).
    
    I made SumAggregator implement ReduceFunctionWithInverse, so calculating sums of sliding or even more general windows should get a speedup from this. WindowIntegrationTest tests this. (FLINK-2144 will also have a few ReduceFunctionWithInverse implementors.)
    
    Grouped case: This is just creating an instance of the pre-reducer for each new group that appears in the window, which is the same as how the MedianGroupedPreReducer works. Thus, I created a GenericGroupedPreReducer that does this with any non-grouped pre-reducer, and refactored the median code to also use this.
    
    I also added a missing clean and removed a superfluous one.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ggevay/flink invReduce

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/856.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #856
    
----
commit 3424a6294cdda8f8b364aa68747d7890a0a9dbab
Author: Gabor Gevay <gg...@gmail.com>
Date:   2015-05-24T10:43:30Z

    [FLINK-2145] [streaming] Fast calculation of medians of windows

commit b44d2d3323249ce5488cc2389f8fd53d3613b448
Author: Gabor Gevay <gg...@gmail.com>
Date:   2015-06-15T15:35:47Z

    [FLINK-2143] [streaming] Added ReduceFunctionWithInverse, which sometimes speeds up reduceWindow

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2143] Added ReduceFunctionWithInverse

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/856#issuecomment-114184385
  
    Here seems to be something with quite some broader implications. This affects the design of aggregations on windows and how they can be pre-aggregated.
    
    The whole windowing is currently being discussed quite a bit, with regard to semantics, and migration to managed memory, or external state. This change needs to be reviewed in context with that. Simply adding this without having it consistent with the other considerations may backfire later.
    
    @aljoscha Started to drive the windowing discussion. Aljoscha, can you post a pointer to the discussion so far?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2143] Added ReduceFunctionWithInverse

Posted by ggevay <gi...@git.apache.org>.

Github user ggevay closed the pull request at:

    https://github.com/apache/flink/pull/856


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2143] Added ReduceFunctionWithInverse

Posted by ggevay <gi...@git.apache.org>.

Github user ggevay commented on the pull request:

    https://github.com/apache/flink/pull/856#issuecomment-205884988
  
    I think that this unfortunately can't be adapted to the new windowing system, so it can be closed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2143] Added ReduceFunctionWithInverse

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on the pull request:

    https://github.com/apache/flink/pull/856#issuecomment-205883786
  
    @ggevay The windowing system changed a while back. Is this still valid or can it be closed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---