You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by kennknowles <gi...@git.apache.org> on 2017/08/23 14:57:27 UTC
[GitHub] beam pull request #3750: [BEAM-2430] DoFnRunner for processing a bundle over...
GitHub user kennknowles opened a pull request:
https://github.com/apache/beam/pull/3750
[BEAM-2430] DoFnRunner for processing a bundle over the Fn API
Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue.
- [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
---
R: @aljoscha this is not super polished, and much of the complexity I thought I would encapsulate really fits better outside the DoFnRunner. But I thought reviewing this and getting it in might be a good way to proceed and share understanding of the Fn API.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kennknowles/beam FnApiDoFnRunner
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3750.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3750
----
commit eb18585969f67089de62b20dc5c3b2722d7e566a
Author: Kenneth Knowles <kl...@google.com>
Date: 2017-08-23T13:20:33Z
Basic Java wrapper for Fn API data plane
commit f71101685265c53ecee36d3aadec037cc792f0b4
Author: Kenneth Knowles <kl...@google.com>
Date: 2017-07-26T14:42:03Z
FnApiControlClient, de-inverts Fn API control plane
The Fn API control plane has an inverted client/server relationship in
order to support firewall rules where the runner is forbidden from
connecting out to the SDK harness. This Java helper provides an API with
the more conventional polarity. It also associates streamed gRPC
requests with responses to support simple future-based programming.
commit 858f219ce3a6df64b8344472022e01141ecf130b
Author: Kenneth Knowles <kl...@google.com>
Date: 2017-07-26T14:44:41Z
Service for pooling incoming Fn API control plane connections
commit c3c882df800926b46c07bdeb867b5639bcde00f1
Author: Kenneth Knowles <kl...@google.com>
Date: 2017-07-26T14:45:37Z
Higher-level SdkHarnessClient
This adds a fairly thin wrapper on FnApiSdkHarnessClient, encapsulating
the fact that all request and response types are injected into a
disjoint union and sent over the same low-level RPC.
commit a947ccb752f4200cbc8a98f85129115e45577973
Author: Kenneth Knowles <kl...@google.com>
Date: 2017-07-26T14:48:48Z
Add SdkHarnessDoFnRunner
This encapsulates processing a bundle over the Fn API.
A Beam runner is responsible for:
- Setting up a FnApiControlClientPoolService to listen for incoming
FnApiSdkHarnessClient connections
- Wrapping those connections in the higher-level SdkHarnessClient
- Building the ProcessBundleDescriptor (instruction graph) to be executed
- Establishing data plane endpoints referenced by the ProcessBundleDescriptor
- Registering the data plane endpoints and ProcessBundleDescriptor with the
SdkHarnessClient
This class is responsible for:
- Registering each bundle with the SDK harness
- Streaming the elements of each bundle to the SDK harness
over the data plane
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] beam pull request #3750: [BEAM-2430] DoFnRunner for processing a bundle over...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/beam/pull/3750
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---