You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by jkff <gi...@git.apache.org> on 2017/06/21 22:27:33 UTC

[GitHub] beam pull request #3417: [BEAM-1377] Uses KV in SplittableParDo expansion in...

GitHub user jkff opened a pull request:

    https://github.com/apache/beam/pull/3417

    [BEAM-1377] Uses KV in SplittableParDo expansion instead of ElementAndRestriction

    This is a workaround for the following issue.
    
    ElementAndRestriction is in runners-core, which may be shaded by runners
    (and is shaded by Dataflow runner), hence it should be *both* produced
    and consumed by workers - but currently it's produced by (shaded)
    SplittableParDo and consumed by (differently shaded) ProcessFn in the
    runner's worker code.
    
    There are several ways out of this, e.g. moving EAR into the SDK (icky
    because it's an implementation detail of SplittableParDo), or using
    a type that's already in the SDK. There may be other more complicated
    ways too.
    
    (This PR will require building a compatible Dataflow worker, so it will naturally not pass tests initially)
    
    R: @kennknowles 


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jkff/incubator-beam sdf-kv

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3417
    
----
commit c4885833c6edd37fcd3161a124292044ebe820da
Author: Eugene Kirpichov <ki...@google.com>
Date:   2017-06-21T22:21:32Z

    Uses KV in SplittableParDo expansion instead of ElementAndRestriction
    
    This is a workaround for the following issue.
    
    ElementAndRestriction is in runners-core, which may be shaded by runners
    (and is shaded by Dataflow runner), hence it should be *both* produced
    and consumed by workers - but currently it's produced by (shaded)
    SplittableParDo and consumed by (differently shaded) ProcessFn in the
    runner's worker code.
    
    There are several ways out of this, e.g. moving EAR into the SDK (icky
    because it's an implementation detail of SplittableParDo), or using
    a type that's already in the SDK. There may be other more complicated
    ways too.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3417: [BEAM-1377] Uses KV in SplittableParDo expansion in...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/beam/pull/3417


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---