You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/10/27 22:15:54 UTC
[beam] branch asf-site updated: Publishing website 2022/10/27 22:15:47 at commit a9e43e2
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cc705261738 Publishing website 2022/10/27 22:15:47 at commit a9e43e2
cc705261738 is described below
commit cc7052617386332d7866c7d99c636aad18c583b1
Author: jenkins <bu...@apache.org>
AuthorDate: Thu Oct 27 22:15:48 2022 +0000
Publishing website 2022/10/27 22:15:47 at commit a9e43e2
---
website/generated-content/documentation/index.xml | 27 ++++++++++++++++++----
.../documentation/programming-guide/index.html | 20 ++++++++++++----
website/generated-content/sitemap.xml | 2 +-
3 files changed, 40 insertions(+), 9 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index 44e4a3368a4..65516094577 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -13342,7 +13342,15 @@ $ jar -jar /path/to/expansion_service.jar &lt;PORT_NUMBER&gt;</code>&
</li>
</ol>
<h4 id="1312-creating-cross-language-python-transforms">13.1.2. Creating cross-language Python transforms</h4>
-<p>To make your Python transform usable with different SDK languages, you must create a Python module that registers an existing Python transform as a cross-language transform for use with the Python expansion service and calls into that existing transform to perform its intended operation.</p>
+<p>Any Python transforms defined in the scope of the expansion service should be accessible by specifying their fully qualified names. For example, you could use Python&rsquo;s <code>ReadFromText</code> transform in a Java pipeline with its fully qualified name <code>apache_beam.io.ReadFromText</code>:</p>
+<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;Read&#34;</span><span class="o">,</span>
+<span class="n">PythonExternalTransform</span><span class="o">.&lt;</span><span class="n">PBegin</span><span class="o">,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;&gt;</span><span class="n">from</span><span class="o">(</span><span class="s">&#34;apache_beam.io.ReadFromText&#34;</span><span class="o">)</span>
+<span class="o">.</span><span class="na">withKwarg</span><span class="o">(</span><span class="s">&#34;file_pattern&#34;</span><span class="o">,</span> <span class="n">options</span><span class="o">.</span><span class="na">getInputFile</span><span class="o">())</span>
+<span class="o">.</span><span class="na">withKwarg</span><span class="o">(</span><span class="s">&#34;validate&#34;</span><span class="o">,</span> <span class="kc">false</span><span class="o">))</span>
+</code></pre></div><blockquote>
+<p><strong>Note:</strong> <code>PythonExternalTransform</code> has other useful methods such as <code>withExtraPackages</code> for staging PyPI package dependencies and <code>withOutputCoder</code> for setting an output coder.</p>
+</blockquote>
+<p>Alternatively, you may want to create a Python module that registers an existing Python transform as a cross-language transform for use with the Python expansion service and calls into that existing transform to perform its intended operation. A registered URN can be used later in an expansion request for indicating an expansion target.</p>
<p><strong>Defining the Python module</strong></p>
<ol>
<li>
@@ -13400,7 +13408,7 @@ $ jar -jar /path/to/expansion_service.jar &lt;PORT_NUMBER&gt;</code>&
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
<img src="/images/copy-icon.svg"/>
</a>
-<pre><code>$ python -m apache_beam.runners.portability.expansion_service_test -p $PORT_FOR_EXPANSION_SERVICE
+<pre><code>$ python -m apache_beam.runners.portability.expansion_service_test -p $PORT_FOR_EXPANSION_SERVICE --pickle_library=cloudpickle
</code></pre>
</div>
</div>
@@ -13463,8 +13471,19 @@ version = “v” 1*(DIGIT / “.”) ; For example, ‘v1.2’</code></pr
<h3 id="use-x-lang-transforms">13.2. Using cross-language transforms</h3>
<p>Depending on the SDK language of the pipeline, you can use a high-level SDK-wrapper class, or a low-level transform class to access a cross-language transform.</p>
<h4 id="1321-using-cross-language-transforms-in-a-java-pipeline">13.2.1. Using cross-language transforms in a Java pipeline</h4>
-<p>Currently, to access cross-language transforms from the Java SDK, you have to use the lower-level <a href="https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java">External</a> class.</p>
-<p><strong>Using the External class</strong></p>
+<p>Users have three options to use cross-language transforms in a Java pipeline. At the highest level of abstraction, some popular Python transforms are accessible through dedicated Java wrapper transforms. For example, the Java SDK has the <code>DataframeTransform</code> class, which uses the Python SDK&rsquo;s <code>DataframeTransform</code>, and it has the <code>RunInference</code> class, which uses the Python SDK&rsquo;s <code>RunInference</code>, a [...]
+<p><strong>Using an SDK wrapper</strong></p>
+<p>To use a cross-language transform through an SDK wrapper, import the module for the SDK wrapper and call it from your pipeline, as shown in the example:</p>
+<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.extensions.python.transforms.DataframeTransform</span><span class="o">;</span>
+<span class="n">input</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">DataframeTransform</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;lambda df: df.groupby(&#39;a&#39;).sum()&#34;</span><span class="o">).</span><span class="na">withIndexes</span><span class="o">())</span>
+</code></pre></div><p><strong>Using the PythonExternalTransform class</strong></p>
+<p>When an SDK-specific wrapper is not available, you can access the Python cross-language transform through the <code>PythonExternalTransform</code> class by specifying the fully qualified name and the constructor arguments of the target Python transform.</p>
+<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="n">input</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span>
+<span class="n">PythonExternalTransform</span><span class="o">.&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;&gt;</span><span class="n">from</span><span class="o">(</span>
+<span class="s">&#34;apache_beam.dataframe.transforms.DataframeTransform&#34;</span><span class="o">)</span>
+<span class="o">.</span><span class="na">withKwarg</span><span class="o">(</span><span class="s">&#34;func&#34;</span><span class="o">,</span> <span class="n">PythonCallableSource</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;lambda df: df.groupby(&#39;a&#39;).sum()&#34;</span><span class="o">))</span>
+<span class="o">.</span><span class="na">withKwarg</span><span class="o">(</span><span class="s">&#34;include_indexes&#34;</span><span class="o">,</span> <span class="kc">true</span><span class="o">))</span>
+</code></pre></div><p><strong>Using the External class</strong></p>
<ol>
<li>
<p>Make sure you have any runtime environment dependencies (like the JRE) installed on your local machine (either directly on the local machine or available through a container). See the expansion service section for more details.</p>
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 1945d01c012..0ae040730a0 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -4981,7 +4981,11 @@ $ jar -jar /path/to/expansion_service.jar <PORT_NUMBER></code></pre></div>
<span class=n>payload</span> <span class=o>=</span> <span class=n>NamedTupleBasedPayloadBuilder</span><span class=p>(</span><span class=n>ReadFromKafkaSchema</span><span class=p>(</span><span class=o>...</span><span class=p>))</span>
</code></pre></div></li><li><p>Start an expansion service, unless one is specified by the pipeline creator. The Beam Python SDK provides the utilities <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService><code>JavaJarExpansionService</code></a> and <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.BeamJarExpansionService> [...]
-</code></pre></div></li></ol></li><li><p>Add a Python wrapper transform class that extends <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform><code>ExternalTransform</code></a>. Pass the payload and expansion service defined above as parameters to the constructor of the <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.Extern [...]
+</code></pre></div></li></ol></li><li><p>Add a Python wrapper transform class that extends <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform><code>ExternalTransform</code></a>. Pass the payload and expansion service defined above as parameters to the constructor of the <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.Extern [...]
+ <span class=n>PythonExternalTransform</span><span class=o>.<</span><span class=n>PBegin</span><span class=o>,</span> <span class=n>PCollection</span><span class=o><</span><span class=n>String</span><span class=o>>></span><span class=n>from</span><span class=o>(</span><span class=s>"apache_beam.io.ReadFromText"</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>withKwarg</span><span class=o>(</span><span class=s>"file_pattern"</span><span class=o>,</span> <span class=n>options</span><span class=o>.</span><span class=na>getInputFile</span><span class=o>())</span>
+ <span class=o>.</span><span class=na>withKwarg</span><span class=o>(</span><span class=s>"validate"</span><span class=o>,</span> <span class=kc>false</span><span class=o>))</span>
+</code></pre></div><blockquote><p><strong>Note:</strong> <code>PythonExternalTransform</code> has other useful methods such as <code>withExtraPackages</code> for staging PyPI package dependencies and <code>withOutputCoder</code> for setting an output coder.</p></blockquote><p>Alternatively, you may want to create a Python module that registers an existing Python transform as a cross-language transform for use with the Python expansion service and calls into that existing transform to per [...]
</code></pre></div></li><li><p>For an existing Python transform, create a new class to register the URN with the Python expansion service.</p><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=nd>@ptransform.PTransform.register_urn</span><span class=p>(</span><span class=n>TEST_COMPK_URN</span><span class=p>,</span> <span class=bp>None</span><span class=p>)</span>
<span class=k>class</span> <span class=nc>CombinePerKeyTransform</span><span class=p>(</span><span class=n>ptransform</span><span class=o>.</span><span class=n>PTransform</span><span class=p>):</span>
</code></pre></div></li><li><p>From within the class, define an expand method that takes an input PCollection, runs the Python transform, and then returns the output PCollection.</p><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=k>def</span> <span class=nf>expand</span><span class=p>(</span><span class=bp>self</span><span class=p>,</span> <span class=n>pcoll</span><span class=p>):</span>
@@ -4995,7 +4999,7 @@ $ jar -jar /path/to/expansion_service.jar <PORT_NUMBER></code></pre></div>
<span class=n>unused_ptransform</span><span class=p>,</span> <span class=n>unused_parameter</span><span class=p>,</span> <span class=n>unused_context</span><span class=p>):</span>
<span class=k>return</span> <span class=n>CombinePerKeyTransform</span><span class=p>()</span>
</code></pre></div></li></ol><p><strong>Starting the expansion service</strong></p><p>An expansion service can be used with multiple transforms in the same pipeline. The Beam Python SDK provides a default expansion service for you to use with your Python transforms. You are free to write your own expansion service, but that is generally not needed, so it is not covered in this section.</p><p>Perform the following steps to start up the default Python expansion service directly:</p><ol><li [...]
- </code></pre></div></div></li><li><p>Import any modules that contain transforms to be made available using the expansion service.</p><div class=snippet><div class="notebook-skip code-snippet without_switcher"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><pre><code>$ python -m apache_beam.runners.portability.expansion_service_test -p $PORT_FOR_EXPANSION_SERVICE
+ </code></pre></div></div></li><li><p>Import any modules that contain transforms to be made available using the expansion service.</p><div class=snippet><div class="notebook-skip code-snippet without_switcher"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><pre><code>$ python -m apache_beam.runners.portability.expansion_service_test -p $PORT_FOR_EXPANSION_SERVICE --pickle_library=cloudpickle
</code></pre></div></div></li><li><p>This expansion service is now ready to serve up transforms on the address <code>localhost:$PORT_FOR_EXPANSION_SERVICE</code>.</p></li></ol><p><strong>Including dependencies</strong></p><p>Currently Python external transforms are limited to dependencies available in the core Beam SDK harness.</p><h4 id=1313-creating-cross-language-go-transforms>13.1.3. Creating cross-language Go transforms</h4><p>Go currently does not support creating cross-languag [...]
transforms from other languages; see more at <a href=https://github.com/apache/beam/issues/21767>Issue 21767</a>.</p><h4 id=1314-defining-a-urn>13.1.4. Defining a URN</h4><p>Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section
we provide a convention for defining such URNs. Following this convention is optional but it will ensure that your transform
@@ -5005,7 +5009,15 @@ ns-id = (“beam” / NID) “:” “transform”
id-char = ALPHA / DIGIT / "-" / "." / "_" / "~" ; A subset of characters allowed in a URN
org-identifier = 1*id-char
functionality-identifier = 1*id-char
-version = “v” 1*(DIGIT / “.”) ; For example, ‘v1.2’</code></pre></div></div><h5 id=13142-examples>13.1.4.2. Examples</h5><p>Below we’ve given some example transform classes and corresponding URNs to be used.</p><ul><li>A transform offered with Apache Beam that writes Parquet files.<ul><li><code>beam:transform:org.apache.beam:parquet_write:v1</code></li></ul></li><li>A transform offered with Apache Beam that reads from Kafka with metadata.<ul><li><code>beam:transform:org.apache.beam:kafk [...]
+version = “v” 1*(DIGIT / “.”) ; For example, ‘v1.2’</code></pre></div></div><h5 id=13142-examples>13.1.4.2. Examples</h5><p>Below we’ve given some example transform classes and corresponding URNs to be used.</p><ul><li>A transform offered with Apache Beam that writes Parquet files.<ul><li><code>beam:transform:org.apache.beam:parquet_write:v1</code></li></ul></li><li>A transform offered with Apache Beam that reads from Kafka with metadata.<ul><li><code>beam:transform:org.apache.beam:kafk [...]
+
+<span class=n>input</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>DataframeTransform</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=s>"lambda df: df.groupby('a').sum()"</span><span class=o>).</span><span class=na>withIndexes</span><span class=o>())</span>
+</code></pre></div><p><strong>Using the PythonExternalTransform class</strong></p><p>When an SDK-specific wrapper is not available, you can access the Python cross-language transform through the <code>PythonExternalTransform</code> class by specifying the fully qualified name and the constructor arguments of the target Python transform.</p><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>input</span><span class=o>.</span><span class=na>apply</ [...]
+ <span class=n>PythonExternalTransform</span><span class=o>.<</span><span class=n>PCollection</span><span class=o><</span><span class=n>Row</span><span class=o>>,</span> <span class=n>PCollection</span><span class=o><</span><span class=n>Row</span><span class=o>>></span><span class=n>from</span><span class=o>(</span>
+ <span class=s>"apache_beam.dataframe.transforms.DataframeTransform"</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>withKwarg</span><span class=o>(</span><span class=s>"func"</span><span class=o>,</span> <span class=n>PythonCallableSource</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=s>"lambda df: df.groupby('a').sum()"</span><span class=o>))</span>
+ <span class=o>.</span><span class=na>withKwarg</span><span class=o>(</span><span class=s>"include_indexes"</span><span class=o>,</span> <span class=kc>true</span><span class=o>))</span>
+</code></pre></div><p><strong>Using the External class</strong></p><ol><li><p>Make sure you have any runtime environment dependencies (like the JRE) installed on your local machine (either directly on the local machine or available through a container). See the expansion service section for more details.</p><blockquote><p><strong>Note:</strong> When including Python transforms from within a Java pipeline, all Python dependencies have to be included in the SDK harness container.</p></bloc [...]
<span class=n>kafka_records</span> <span class=o>=</span> <span class=p>(</span>
<span class=n>pipeline</span>
@@ -5144,7 +5156,7 @@ const result: PCollection<string> = await pcoll
)
);
-</code></pre><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the direct runner can be used with multi-language pipelines.</p><p>Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><h3 id=x-lang-transform-tips-troubleshooting>13.4 Tips and Troubleshooting</h3><p>For additional tips and troubleshooting information, see <a href=https://cwiki.apache.org/confluence/display/B [...]
+</code></pre><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the direct runner can be used with multi-language pipelines.</p><p>Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><h3 id=x-lang-transform-tips-troubleshooting>13.4 Tips and Troubleshooting</h3><p>For additional tips and troubleshooting information, see <a href=https://cwiki.apache.org/confluence/display/B [...]
<a href=https://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div><div class="footer__cols__col footer__cols__col__logos"><div class=footer__cols__col--group><div class=footer__cols__col__logo><a href=https://github.com/apache/beam><im [...]
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index 1ff12253687..2dd728732be 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.42.0/</loc><lastmod>2022-10-17T09:50:38-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2022-10-17T09:50:38-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-10-17T09:50:38-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-10-17T09:50:38-07:00</lastmod></url><url><loc>/catego [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.42.0/</loc><lastmod>2022-10-17T09:50:38-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2022-10-17T09:50:38-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-10-17T09:50:38-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-10-17T09:50:38-07:00</lastmod></url><url><loc>/catego [...]
\ No newline at end of file