[jira] [Commented] (BEAM-2453) The Java DirectRunner should exercise all parts of a CombineFn

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090755#comment-16090755 ] 

ASF GitHub Bot commented on BEAM-2453:
--------------------------------------

GitHub user tgroh opened a pull request:

    https://github.com/apache/beam/pull/3579

    [BEAM-2453] Perform a Multi-step combine in the DirectRunner

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [ ] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`.
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
    
    ---
    This exercises the entire CombineFn lifecycle for simple combine fns,
    expressed as a collection of DoFns.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tgroh/beam combiner_lifting_direct_runner

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3579.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3579
    
----
commit 522fa542eeeb7a698982ef42b4d3f77ef642f59e
Author: Thomas Groh <tg...@google.com>
Date:   2017-06-15T22:53:46Z

    Perform a Multi-step combine in the DirectRunner
    
    This exercises the entire CombineFn lifecycle for simple combine fns,
    expressed as a collection of DoFns.

----


> The Java DirectRunner should exercise all parts of a CombineFn
> --------------------------------------------------------------
>
>                 Key: BEAM-2453
>                 URL: https://issues.apache.org/jira/browse/BEAM-2453
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-direct
>            Reporter: Thomas Groh
>            Assignee: Thomas Groh
>
> Specifically it should:
> Create some number of accumulators; add elements to these accumulators, merge the created accumulators, and extract the output.
> This can be performed by replacing the {{Combine.perKey}} composite transform with a multi-step combine {{CombineBundles -> GroupByKey -> MergeAccumulators}}
> Where {{CombineBundles}} is a {{ParDo}} which takes input {{KV<K, InputT>}} and produces {{KV<K, AccumT>}}, outputting in {{FinishBundle}} (this can only be performed if the Combine takes no side inputs or does not have merging windows). {{MergeAccumulators}} takes in {{KV<K, Iterable<AccumT>>}} and produces {{KV<K, OutputT>}} by merging all of the accumulators and extracting the output.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)