You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2021/10/07 18:03:29 UTC
[beam] branch asf-site updated: Publishing website 2021/10/07
18:02:54 at commit 87ffd1f
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1a73185 Publishing website 2021/10/07 18:02:54 at commit 87ffd1f
1a73185 is described below
commit 1a73185a2be03a82bb19a08f63d4d09478837458
Author: jenkins <bu...@apache.org>
AuthorDate: Thu Oct 7 18:02:55 2021 +0000
Publishing website 2021/10/07 18:02:54 at commit 87ffd1f
---
website/generated-content/documentation/index.xml | 14 +++++++-------
.../documentation/programming-guide/index.html | 12 ++++++------
website/generated-content/sitemap.xml | 2 +-
3 files changed, 14 insertions(+), 14 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index e3aa7ac..ccdac3b 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -11132,18 +11132,18 @@ $ jar -jar /path/to/expansion_service.jar &lt;PORT_NUMBER&gt;</code>&
</div>
</div>
<p>The expansion service is now ready to serve transforms on the specified port.</p>
-<p>When creating SDK-specific wrappers for your transform, SDKs may provide utilities that are readily available for easily starting up an expansion service. For example, the Python SDK provides the utilities <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py">JavaJarExpansionService and BeamJarExpansionService</a> utility for starting up a Java expansion service using a JAR file.</p>
+<p>When creating SDK-specific wrappers for your transform, SDKs may provide utilities that are readily available for easily starting up an expansion service. For example, the Python SDK provides the utilities <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService"><code>JavaJarExpansionService</code></a> and <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.tr [...]
<p><strong>Including dependencies</strong></p>
<p>If your transform requires external libraries, you can include them by adding them to the classpath of the expansion service. After they are included in the classpath, they will be staged when your transform is expanded by the expansion service.</p>
<p><strong>Writing SDK-specific wrappers</strong></p>
-<p>Your cross-language Java transform can be called through the lower-level <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py">ExternalTransform</a> class in a multi-language pipeline (as described in the next section); however, if possible, you should create a SDK-specific wrapper written in the programming language of the pipeline (such as Python) to access the transform instead. This higher-level abstraction will make it easier [...]
+<p>Your cross-language Java transform can be called through the lower-level <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform"><code>ExternalTransform</code></a> class in a multi-language pipeline (as described in the next section); however, if possible, you should create a SDK-specific wrapper written in the programming language of the pipeline (such as Python) to access the trans [...]
<p>To create an SDK wrapper for use in a Python pipeline, do the following:</p>
<ol>
<li>
<p>Create a Python module for your cross-language transform(s).</p>
</li>
<li>
-<p>In the module, build the payload that should be used to initiate the cross-language transform expansion request using one of the available <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py">PayloadBuilder</a> classes.</p>
+<p>In the module, build the payload that should be used to initiate the cross-language transform expansion request using one of the available <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.PayloadBuilder"><code>PayloadBuilder</code></a> classes.</p>
<p>The parameter names and types of the payload should map to parameter names and types of the configuration POJO provided to the Java <code>ExternalTransformBuilder</code>. Parameter types are mapped across SDKs using a <a href="https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/schema.proto">Beam schema</a>. Parameter names are mapped by simply converting Python underscore-separated variable names to camel-case (Java standard).</p>
<p>In the following example, <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/kafka.py">kafka.py</a> uses <code>NamedTupleBasedPayloadBuilder</code> to build the payload. The parameters map to the Java <a href="https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java">KafkaIO.External.Configuration</a> config object defined previously in the <strong>Implementing the interfac [...]
<div class="snippet">
@@ -11161,7 +11161,7 @@ payload = NamedTupleBasedPayloadBuilder(ReadFromKafkaSchema(...))
</div>
</li>
<li>
-<p>Start an expansion service unless one is specified by the pipeline creator. The Apache Beam Python SDK provides utilities <code>JavaJarExpansionService</code> and <code>BeamJarExpansionService</code> for easily starting up an expansion service using a JAR file.. <code>JavaJarExpansionService</code> can be used to startup an expansion service using path (a local path or a URL) to a given JAR file. <code>BeamJarExpansionService</code> can be used for easily st [...]
+<p>Start an expansion service unless one is specified by the pipeline creator. The Apache Beam Python SDK provides utilities <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService"><code>JavaJarExpansionService</code></a> and <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.BeamJarExpansionService">< [...]
<p>For transforms released with Beam do the following:</p>
<ol>
<li>
@@ -11182,7 +11182,7 @@ payload = NamedTupleBasedPayloadBuilder(ReadFromKafkaSchema(...))
</ol>
</li>
<li>
-<p>Add a Python wrapper transform class that extends <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py">ExternalTransform</a>. Pass the payload and expansion service defined above as parameters to the constructor of the <code>ExternalTransform</code> parent class.</p>
+<p>Add a Python wrapper transform class that extends <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform"><code>ExternalTransform</code></a>. Pass the payload and expansion service defined above as parameters to the constructor of the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform">< [...]
</li>
</ol>
<h4 id="1312-creating-cross-language-python-transforms">13.1.2. Creating cross-language Python transforms</h4>
@@ -11322,7 +11322,7 @@ transforms from other languages; see more at <a href="https://issues.apache.o
</li>
</ol>
<h4 id="1322-using-cross-language-transforms-in-a-python-pipeline">13.2.2. Using cross-language transforms in a Python pipeline</h4>
-<p>If a Python-specific wrapper for a cross-language transform is available, use that; otherwise, you have to use the lower-level <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py">ExternalTransform</a> class to access the transform.</p>
+<p>If a Python-specific wrapper for a cross-language transform is available, use that; otherwise, you have to use the lower-level <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform"><code>ExternalTransform</code></a> class to access the transform.</p>
<p><strong>Using an SDK wrapper</strong></p>
<p>To use a cross-language transform through an SDK wrapper, import the module for the SDK wrapper and call it from your pipeline as shown in the example:</p>
<div class="snippet">
@@ -11355,7 +11355,7 @@ expansion_service=&lt;Address of expansion service&gt;))
<p>Make sure the transform you&rsquo;re trying to use is available and can be used by the expansion service. For Java, make sure the builder and registrar for the transform are available in the classpath of the expansion service.</p>
</li>
<li>
-<p>Include <code>ExternalTransform</code> when instantiating your pipeline. Reference the URN, Payload, and expansion service. You can use one of the available <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py">PayloadBuilder</a> classes to build the payload for <code>ExternalTransform</code>.</p>
+<p>Include <code>ExternalTransform</code> when instantiating your pipeline. Reference the URN, Payload, and expansion service. You can use one of the available <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.PayloadBuilder"><code>PayloadBuilder</code></a> classes to build the payload for <code>ExternalTransform</code>.</p>
<div class="snippet">
<div class="notebook-skip code-snippet without_switcher">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index b86067b..77d0002 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -4168,14 +4168,14 @@ public static class External implements ExternalTransformRegistrar {
</code></pre></div></div></li></ol><p>After you have implemented the <code>ExternalTransformBuilder</code> and <code>ExternalTransformRegistrar</code> interfaces, your transform can be registered and created successfully by the default Java expansion service.</p><p><strong>Starting the expansion service</strong></p><p>An expansion service can be used with multiple transforms in the same pipeline. Java has a default expansion service included and available in the Apache Beam Java SDK [...]
# Start the expansion service at the specified port.
-$ jar -jar /path/to/expansion_service.jar <PORT_NUMBER></code></pre></div></div><p>The expansion service is now ready to serve transforms on the specified port.</p><p>When creating SDK-specific wrappers for your transform, SDKs may provide utilities that are readily available for easily starting up an expansion service. For example, the Python SDK provides the utilities <a href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py>JavaJarExpansio [...]
+$ jar -jar /path/to/expansion_service.jar <PORT_NUMBER></code></pre></div></div><p>The expansion service is now ready to serve transforms on the specified port.</p><p>When creating SDK-specific wrappers for your transform, SDKs may provide utilities that are readily available for easily starting up an expansion service. For example, the Python SDK provides the utilities <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transform [...]
consumer_config: typing.Mapping[str, str]
topics: typing.List[str]
# Other properties omitted for clarity.
payload = NamedTupleBasedPayloadBuilder(ReadFromKafkaSchema(...))
- </code></pre></div></div></li><li><p>Start an expansion service unless one is specified by the pipeline creator. The Apache Beam Python SDK provides utilities <code>JavaJarExpansionService</code> and <code>BeamJarExpansionService</code> for easily starting up an expansion service using a JAR file.. <code>JavaJarExpansionService</code> can be used to startup an expansion service using path (a local path or a URL) to a given JAR file. <code>BeamJarExpansionService</code> can be used fo [...]
- </code></pre></div></div></li></ol></li><li><p>Add a Python wrapper transform class that extends <a href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py>ExternalTransform</a>. Pass the payload and expansion service defined above as parameters to the constructor of the <code>ExternalTransform</code> parent class.</p></li></ol><h4 id=1312-creating-cross-language-python-transforms>13.1.2. Creating cross-language Python transforms</h4><p>To m [...]
+ </code></pre></div></div></li><li><p>Start an expansion service unless one is specified by the pipeline creator. The Apache Beam Python SDK provides utilities <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService><code>JavaJarExpansionService</code></a> and <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.BeamJarExpan [...]
+ </code></pre></div></div></li></ol></li><li><p>Add a Python wrapper transform class that extends <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform><code>ExternalTransform</code></a>. Pass the payload and expansion service defined above as parameters to the constructor of the <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.e [...]
</code></pre></div></div></li><li><p>For an existing Python transform, create a new class to register the URN with the Python expansion service.</p><div class=snippet><div class="notebook-skip code-snippet without_switcher"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><pre><code>@ptransform.PTransform.register_urn(TEST_COMPK_URN, None)
class CombinePerKeyTransform(ptransform.PTransform):
</code></pre></div></div></li><li><p>From within the class, define an expand method that takes an input PCollection, runs the Python transform, and then returns the output PCollection.</p><div class=snippet><div class="notebook-skip code-snippet without_switcher"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><pre><code>def expand(self, pcoll):
@@ -4191,7 +4191,7 @@ def from_runner_api_parameter(
</code></pre></div></div></li></ol><p><strong>Starting the expansion service</strong></p><p>An expansion service can be used with multiple transforms in the same pipeline. Python has a default expansion service included and available in the Apache Beam Python SDK for you to use with your Python transforms. You are free to write your own expansion service, but that is generally not needed, so it is not covered in this section.</p><p>Perform the following steps to start up the default [...]
</code></pre></div></div></li><li><p>Import any modules that contain transforms to be made available using the expansion service.</p><div class=snippet><div class="notebook-skip code-snippet without_switcher"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><pre><code>$ python -m apache_beam.runners.portability.expansion_service_test -p $PORT_FOR_EXPANSION_SERVICE
</code></pre></div></div></li><li><p>This expansion service is not ready to serve up transforms on the address <code>localhost:$PORT_FOR_EXPANSION_SERVICE</code>.</p></li></ol><p><strong>Including dependencies</strong></p><p>Currently Python external transforms are limited to dependencies available in core Beam SDK Harness.</p><h4 id=1313-creating-cross-language-go-transforms>13.1.3. Creating cross-language Go transforms</h4><p>Go currently does not support creating cross-language tr [...]
-transforms from other languages; see more at <a href=https://issues.apache.org/jira/browse/BEAM-9923>BEAM-9923</a>.</p><h3 id=use-x-lang-transforms>13.2. Using cross-language transforms</h3><p>Depending on the SDK language of the pipeline, you can use a high-level SDK-wrapper class, or a low-level transform class to access a cross-language transform.</p><h4 id=1321-using-cross-language-transforms-in-a-java-pipeline>13.2.1. Using cross-language transforms in a Java pipeline</h4><p>Current [...]
+transforms from other languages; see more at <a href=https://issues.apache.org/jira/browse/BEAM-9923>BEAM-9923</a>.</p><h3 id=use-x-lang-transforms>13.2. Using cross-language transforms</h3><p>Depending on the SDK language of the pipeline, you can use a high-level SDK-wrapper class, or a low-level transform class to access a cross-language transform.</p><h4 id=1321-using-cross-language-transforms-in-a-java-pipeline>13.2.1. Using cross-language transforms in a Java pipeline</h4><p>Current [...]
kafka_records = (
pipeline
@@ -4203,7 +4203,7 @@ kafka_records = (
topics=[self.topic],
max_num_records=max_num_records,
expansion_service=<Address of expansion service>))
- </code></pre></div></div><p><strong>Using the ExternalTransform class</strong></p><p>When an SDK-specific wrapper isn’t available, you will have to access the cross-language transform through the <code>ExternalTransform</code> class.</p><ol><li><p>Make sure you have any runtime environment dependencies (like JRE) installed on your local machine. See the expansion service section for more details.</p></li><li><p>Start up the expansion service for the SDK that is in the language of [...]
+ </code></pre></div></div><p><strong>Using the ExternalTransform class</strong></p><p>When an SDK-specific wrapper isn’t available, you will have to access the cross-language transform through the <code>ExternalTransform</code> class.</p><ol><li><p>Make sure you have any runtime environment dependencies (like JRE) installed on your local machine. See the expansion service section for more details.</p></li><li><p>Start up the expansion service for the SDK that is in the language of [...]
res = (
p
| beam.Create(['a', 'b']).with_output_types(unicode)
@@ -4245,7 +4245,7 @@ expansionAddr := "localhost:8097"
outT := beam.UnnamedOutput(typex.New(reflectx.String))
res := beam.CrossLanguage(s, urn, payload, expansionAddr, beam.UnnamedInput(inputPCol), outT)
</code></pre></div></div></li><li><p>After the job has been submitted to the Beam runner, shutdown the expansion service by
-terminating the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the Direct runner can be used with multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><div class=feedback><p class=update>Last updated on 2021/10/04</p><h3>Have you found everything you were looking for?</h3><p class=descr [...]
+terminating the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the Direct runner can be used with multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><div class=feedback><p class=update>Last updated on 2021/10/07</p><h3>Have you found everything you were looking for?</h3><p class=descr [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index 4ad78bf..16d8a39 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.32.0/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-09-17T14:05:48-07:00</lastmod></url><url><loc>/blog/b [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.32.0/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-09-17T14:05:48-07:00</lastmod></url><url><loc>/blog/b [...]
\ No newline at end of file