You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2019/11/09 22:39:21 UTC
[beam] branch asf-site updated: Publishing website 2019/11/09
22:39:14 at commit ede14d4
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c68ab5b Publishing website 2019/11/09 22:39:14 at commit ede14d4
c68ab5b is described below
commit c68ab5bc1864d9902a24aef76e903cb7745bfe2a
Author: jenkins <bu...@apache.org>
AuthorDate: Sat Nov 9 22:39:15 2019 +0000
Publishing website 2019/11/09 22:39:14 at commit ede14d4
---
.../roadmap/connectors-multi-sdk/index.html | 117 ++++++++++++++++++---
1 file changed, 104 insertions(+), 13 deletions(-)
diff --git a/website/generated-content/roadmap/connectors-multi-sdk/index.html b/website/generated-content/roadmap/connectors-multi-sdk/index.html
index 70eac06..4e3df3f 100644
--- a/website/generated-content/roadmap/connectors-multi-sdk/index.html
+++ b/website/generated-content/roadmap/connectors-multi-sdk/index.html
@@ -226,6 +226,20 @@
+<ul class="nav">
+ <li><a href="#completed-and-ongoing-efforts">Completed and Ongoing Efforts</a>
+ <ul>
+ <li><a href="#cross-language-transforms-api-and-expansion-service">Cross-language transforms API and expansion service</a></li>
+ <li><a href="#support-for-flink-runner">Support for Flink runner</a></li>
+ <li><a href="#support-for-dataflow-runner">Support for Dataflow runner</a></li>
+ <li><a href="#support-for-direct-runner">Support for Direct runner</a></li>
+ <li><a href="#connectortransform-support">Connector/transform support</a></li>
+ <li><a href="#portable-beam-schema">Portable Beam schema</a></li>
+ <li><a href="#integrationperformance-testing">Integration/Performance testing</a></li>
+ <li><a href="#documentation">Documentation</a></li>
+ </ul>
+ </li>
+</ul>
</nav>
@@ -255,23 +269,100 @@ efforts. See <a href="https://beam.apache.org/roadmap/portability/">Beam portabi
<h1 id="cross-language-transforms">Cross-language transforms</h1>
-<p>As an added benefit of Beam portability efforts, in the future, we’ll be
-able to utilize Beam transforms across languages. This has many benefits.
-For example.</p>
+<p><em>Last updated on November 2019.</em></p>
+
+<p>As an added benefit of Beam portability effort, we are able to utilize Beam transforms across SDKs. This has many benefits.</p>
+
+<ul>
+ <li>Connector sharing across SDKs. For example,
+ <ul>
+ <li>Beam pipelines written using Python and Go SDKs will be able to utilize the vast selection of connectors that are currently implemented for Java SDK.</li>
+ <li>Java SDK will be able to utilize connectors for systems that only offer a Python API.</li>
+ <li>Go SDK, will be able to utilize connectors currently available for Java and Python SDKs.</li>
+ </ul>
+ </li>
+ <li>Ease of developing and maintaining Beam transforms - in general, with cross-language transforms, Beam transform authors will be able to implement new Beam transforms using a
+language of choice and utilize these transforms from other languages reducing the maintenance and support overheads.</li>
+ <li><a href="https://beam.apache.org/documentation/dsls/sql/overview/">Beam SQL</a>, that is currently only available to Java SDK, will become available to Python and Go SDKs.</li>
+ <li><a href="https://www.tensorflow.org/tfx/transform/get_started">Beam TFX transforms</a>, that are currently only available to Beam Python SDK pipelines will become available to Java and Go SDKs.</li>
+</ul>
+
+<h2 id="completed-and-ongoing-efforts">Completed and Ongoing Efforts</h2>
+
+<p>Many efforts related to cross-language transforms are currently in flux. Some of the completed and ongoing efforts are given below.</p>
+
+<h3 id="cross-language-transforms-api-and-expansion-service">Cross-language transforms API and expansion service</h3>
+
+<p>Work related to developing/updating the cross-language transforms API for Java/Python/Go SDKs and work related to cross-language transform expansion services.</p>
+
+<ul>
+ <li>Basic API for Java SDK - completed</li>
+ <li>Basic API for Python SDK - completed</li>
+ <li>Basic API for Go SDK - Not started</li>
+ <li>Basic cross-language transform expansion service for Java and Python SDKs - completed</li>
+ <li>Artifact staging - In progress - <a href="https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E">email thread</a>, <a href="https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit#heading=h.900gc947qrw8">doc</a></li>
+</ul>
+
+<h3 id="support-for-flink-runner">Support for Flink runner</h3>
+
+<p>Work related to making cross-language transforms available for Flink runner.</p>
+
+<ul>
+ <li>Basic support for executing cross-language transforms on portable Flink runner - completed</li>
+</ul>
+
+<h3 id="support-for-dataflow-runner">Support for Dataflow runner</h3>
+
+<p>Work related to making cross-language transforms available for Dataflow runner.</p>
+
+<ul>
+ <li>Basic support for executing cross-language transforms on Dataflow runner
+ <ul>
+ <li>This work requires updates to Dataflow service’s job submission and job execution logic. This is currently being developed at Google.</li>
+ </ul>
+ </li>
+</ul>
+
+<h3 id="support-for-direct-runner">Support for Direct runner</h3>
+
+<p>Work related to making cross-language transforms available on Direct runner</p>
+
+<ul>
+ <li>Basic support for executing cross-language transforms on portable Direct runner - Not started</li>
+</ul>
+
+<h3 id="connectortransform-support">Connector/transform support</h3>
+
+<p>Ongoing and planned work related to making existing connectors/transforms available to other SDKs through the cross-language transforms framework.</p>
+
+<ul>
+ <li>Java KafkIO - In progress - <a href="https://issues.apache.org/jira/browse/BEAM-7029">BEAM-7029</a></li>
+ <li>Java PubSubIO - In progress - <a href="https://issues.apache.org/jira/browse/BEAM-7738">BEAM-7738</a></li>
+</ul>
+
+<h3 id="portable-beam-schema">Portable Beam schema</h3>
+
+<p>Portable Beam schema support will provide a generalized mechanism for serializing and transferring data across language boundaries which will be extremely useful for pipelines that employ cross-language transforms.</p>
+
+<ul>
+ <li>Make row coder a standard coder and implement in python - In progress - <a href="https://issues.apache.org/jira/browse/BEAM-7886">BEAM-7886</a></li>
+</ul>
+
+<h3 id="integrationperformance-testing">Integration/Performance testing</h3>
+
+<ul>
+ <li>Add an integration test suite for cross-language transforms on Flink runner - In progress - <a href="https://issues.apache.org/jira/browse/BEAM-6683">BEAM-6683</a></li>
+</ul>
+
+<h3 id="documentation">Documentation</h3>
+
+<p>Work related to adding documenting on cross-language transforms to Beam Website.</p>
<ul>
- <li>Beam pipelines written using Python and Go SDKs will be able to utilize
-the vast selection of connectors that are currently available for Java SDK.</li>
- <li>Java SDK will be able to utilize connectors for systems that only offer a
-Python API.</li>
- <li>Go SDK, will be able to utilize connectors currently available for Java and
-Python SDKs.</li>
- <li>Connector authors will be able to implement new Beam connectors using a
-language of choice and utilize these connectors from other languages reducing
-the maintenance and support efforts.</li>
+ <li>Document cross-language transforms API for Java/Python - Not started</li>
+ <li>Document API for making existing transforms available as cross-language transforms for Java/Python - Not started</li>
</ul>
-<p>See <a href="https://beam.apache.org/roadmap/portability/">Beam portability framework roadmap</a> for more details.</p>
</div>
</div>