[jira] [Commented] (BEAM-2430) Java FnApiDoFnRunner to share across runners

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138449#comment-16138449 ] 

ASF GitHub Bot commented on BEAM-2430:
--------------------------------------

GitHub user kennknowles opened a pull request:

    https://github.com/apache/beam/pull/3750

    [BEAM-2430] DoFnRunner for processing a bundle over the Fn API

    Follow this checklist to help us incorporate your contribution quickly and easily:
    
     - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
     - [ ] Each commit in the pull request should have a meaningful subject line and body.
     - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
     - [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
     - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
     - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
    
    ---
    
    R: @aljoscha this is not super polished, and much of the complexity I thought I would encapsulate really fits better outside the DoFnRunner. But I thought reviewing this and getting it in might be a good way to proceed and share understanding of the Fn API.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kennknowles/beam FnApiDoFnRunner

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3750.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3750
    
----
commit eb18585969f67089de62b20dc5c3b2722d7e566a
Author: Kenneth Knowles <kl...@google.com>
Date:   2017-08-23T13:20:33Z

    Basic Java wrapper for Fn API data plane

commit f71101685265c53ecee36d3aadec037cc792f0b4
Author: Kenneth Knowles <kl...@google.com>
Date:   2017-07-26T14:42:03Z

    FnApiControlClient, de-inverts Fn API control plane
    
    The Fn API control plane has an inverted client/server relationship in
    order to support firewall rules where the runner is forbidden from
    connecting out to the SDK harness. This Java helper provides an API with
    the more conventional polarity. It also associates streamed gRPC
    requests with responses to support simple future-based programming.

commit 858f219ce3a6df64b8344472022e01141ecf130b
Author: Kenneth Knowles <kl...@google.com>
Date:   2017-07-26T14:44:41Z

    Service for pooling incoming Fn API control plane connections

commit c3c882df800926b46c07bdeb867b5639bcde00f1
Author: Kenneth Knowles <kl...@google.com>
Date:   2017-07-26T14:45:37Z

    Higher-level SdkHarnessClient
    
    This adds a fairly thin wrapper on FnApiSdkHarnessClient, encapsulating
    the fact that all request and response types are injected into a
    disjoint union and sent over the same low-level RPC.

commit a947ccb752f4200cbc8a98f85129115e45577973
Author: Kenneth Knowles <kl...@google.com>
Date:   2017-07-26T14:48:48Z

    Add SdkHarnessDoFnRunner
    
    This encapsulates processing a bundle over the Fn API.
    
    A Beam runner is responsible for:
    
     - Setting up a FnApiControlClientPoolService to listen for incoming
       FnApiSdkHarnessClient connections
     - Wrapping those connections in the higher-level SdkHarnessClient
     - Building the ProcessBundleDescriptor (instruction graph) to be executed
     - Establishing data plane endpoints referenced by the ProcessBundleDescriptor
     - Registering the data plane endpoints and ProcessBundleDescriptor with the
       SdkHarnessClient
    
    This class is responsible for:
    
     - Registering each bundle with the SDK harness
     - Streaming the elements of each bundle to the SDK harness
       over the data plane

----


> Java FnApiDoFnRunner to share across runners
> --------------------------------------------
>
>                 Key: BEAM-2430
>                 URL: https://issues.apache.org/jira/browse/BEAM-2430
>             Project: Beam
>          Issue Type: New Feature
>          Components: runner-core
>            Reporter: Kenneth Knowles
>            Assignee: Kenneth Knowles
>              Labels: beam-python-everywhere
>
> As the portability framework comes into focus, let's fill out the support code for making it easy to onboard a new runner.
> There is some amount of using the Fn API that has to do only with the fact that a runner is implemented in Java, and is not specific to that runner. This should be part of the runners-core library, and designed so that a runner can set it up however it likes, and just pass elements without having to explicitly manage things like requests, responses, protos, and coders.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)