You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2023/10/13 00:32:15 UTC
[beam] branch asf-site updated: Publishing website 2023/10/13 00:32:05 at commit 0586161
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d33bfc61441 Publishing website 2023/10/13 00:32:05 at commit 0586161
d33bfc61441 is described below
commit d33bfc61441dd20de294e8f18bac0bb545b8940d
Author: runner <ru...@main-runner-7jxcw-j8rtj>
AuthorDate: Fri Oct 13 00:32:05 2023 +0000
Publishing website 2023/10/13 00:32:05 at commit 0586161
---
website/generated-content/documentation/index.xml | 2 +-
.../documentation/ml/multi-language-inference/index.html | 2 +-
.../documentation/sdks/python-pipeline-dependencies/index.html | 9 +++++----
website/generated-content/sitemap.xml | 2 +-
4 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index 04a4369e3d2..43288b91049 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -17783,7 +17783,7 @@ used to predict the last word of a sentence based on the context of the sentence
<p>Finally, we postprocess the model predictions in the <code>Postprocess</code> DoFn. The <code>Postprocess</code> DoFn returns the original text, the last word of the sentence, and the predicted word.</p>
<h3 id="compile-python-code-into-package">Compile Python code into package</h3>
<p>The custom Python code needs to be written in a local package and be compiled as a tarball. This package can then be used by the Java pipeline. The following example shows how to compile the Python package into a tarball:</p>
-<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"> python setup.py sdist
+<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"> pip install --upgrade build <span class="o">&amp;&amp;</span> python -m build --sdist
</span></span></code></pre></div><p>In order to run this, a <code>setup.py</code> is required. The path to the tarball will be used as an argument in the pipeline options of the Java pipeline.</p>
<h3 id="run-the-java-pipeline">Run the Java pipeline</h3>
<p>The Java pipeline is defined in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/multi_language_inference/last_word_prediction/src/main/java/org/apache/beam/examples/MultiLangRunInference.java#L32"><code>MultiLangRunInference</code></a> class. In this pipeline, the data is read from Google Cloud Storage, the cross-language Python transform is applied, and the output is written back to Google Cloud Storage.</p>
diff --git a/website/generated-content/documentation/ml/multi-language-inference/index.html b/website/generated-content/documentation/ml/multi-language-inference/index.html
index 9430730ff8b..f20448e6cff 100644
--- a/website/generated-content/documentation/ml/multi-language-inference/index.html
+++ b/website/generated-content/documentation/ml/multi-language-inference/index.html
@@ -61,7 +61,7 @@ used to predict the last word of a sentence based on the context of the sentence
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>def</span> <span class=nf>batch_elements_kwargs</span><span class=p>(</span><span class=bp>self</span><span class=p>):</span>
</span></span><span class=line><span class=cl> <span class=k>return</span> <span class=p>{</span><span class=s1>'max_batch_size'</span><span class=p>:</span> <span class=mi>1</span><span class=p>}</span>
-</span></span></code></pre></div><p>An alternative aproach is to make all the tensors have the same length. This <a href=https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch_tensorflow_sklearn.ipynb>example</a> shows how to do that.</p><p>The <code>ModelConfig</code> and <code>ModelTokenizer</code> are loaded in the initialization function. The <code>ModelConfig</code> is used to define the model architecture, and the <code>ModelTokenizer</code> is [...]
+</span></span></code></pre></div><p>An alternative aproach is to make all the tensors have the same length. This <a href=https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch_tensorflow_sklearn.ipynb>example</a> shows how to do that.</p><p>The <code>ModelConfig</code> and <code>ModelTokenizer</code> are loaded in the initialization function. The <code>ModelConfig</code> is used to define the model architecture, and the <code>ModelTokenizer</code> is [...]
</span></span></code></pre></div><p>In order to run this, a <code>setup.py</code> is required. The path to the tarball will be used as an argument in the pipeline options of the Java pipeline.</p><h3 id=run-the-java-pipeline>Run the Java pipeline</h3><p>The Java pipeline is defined in the <a href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/multi_language_inference/last_word_prediction/src/main/java/org/apache/beam/examples/MultiLangRunInference.j [...]
</span></span></span><span class=line><span class=cl><span class=se></span> -Dexec.args<span class=o>=</span><span class=s2>"--runner=DataflowRunner \
</span></span></span><span class=line><span class=cl><span class=s2> --project=</span><span class=nv>$GCP_PROJECT</span><span class=s2>\
diff --git a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
index d7aa54de2fd..9bbf61ed886 100644
--- a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
+++ b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
@@ -41,10 +41,11 @@
</code></pre><p>The runner will use the <code>requirements.txt</code> file to install your additional dependencies onto the remote workers.</p></li></ol><blockquote><p><strong>NOTE</strong>: An alternative to <code>pip freeze</code> is to use a library like <a href=https://github.com/jazzband/pip-tools>pip-tools</a> to compile all the dependencies required for the pipeline from a <code>--requirements_file</code>, where only top-level dependencies are mentioned.</p></blockquote><h2 id=cus [...]
COPY <path to requirements.txt> /tmp/requirements.txt
RUN python -m pip install -r /tmp/requirements.txt
-</code></pre></li></ol><h2 id=local-or-nonpypi>Local or non-PyPI Dependencies</h2><p>If your pipeline uses packages that are not available publicly (e.g. packages that you’ve downloaded from a GitHub repo), make these packages available remotely by performing the following steps:</p><ol><li><p>Identify which packages are installed on your machine and are not public. Run the following command:</p><p>pip freeze</p><p>This command lists all packages that are installed on your machine, [...]
-</code></pre><p>where package-name is the package’s tarball. If you have the <code>setup.py</code> for that
-package then you can build the tarball with the following command:</p><pre><code> python setup.py sdist
-</code></pre><p>See the <a href=https://docs.python.org/3/distutils/sourcedist.html>sdist documentation</a> for more details on this command.</p></li></ol><h2 id=multiple-file-dependencies>Multiple File Dependencies</h2><p>Often, your pipeline code spans multiple files. To run your project remotely, you must group these files as a Python package and specify the package when you run your pipeline. When the remote workers start, they will install your package. To group your files as a Pyth [...]
+</code></pre></li></ol><h2 id=local-or-nonpypi>Local or non-PyPI Dependencies</h2><p>If your pipeline uses packages that are not available publicly (e.g. packages that you’ve downloaded from a GitHub repo), make these packages available remotely by performing the following steps:</p><ol><li><p>Identify which packages are installed on your machine and are not public. Run the following command:</p><p>pip freeze</p><p>This command lists all packages that are installed on your machine, [...]
+</code></pre><p>where package-name is the package’s tarball. You can build the package tarball using a command line tool called <a href=https://setuptools.pypa.io/en/latest/userguide/quickstart.html#install-build>build</a>.</p><pre><code> # Install build using pip
+ pip install --upgrade build
+ python -m build --sdist
+</code></pre><p>See the <a href=https://pypa-build.readthedocs.io/en/latest/index.html>build documentation</a> for more details on this command.</p></li></ol></li></ol><h2 id=multiple-file-dependencies>Multiple File Dependencies</h2><p>Often, your pipeline code spans multiple files. To run your project remotely, you must group these files as a Python package and specify the package when you run your pipeline. When the remote workers start, they will install your package. To group your fi [...]
setuptools.setup(
name='PACKAGE-NAME',
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index 586f2a808e9..c9a70ac9180 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.51.0/</loc><lastmod>2023-10-12T18:57:35+04:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2023-10-12T18:57:35+04:00</lastmod></url><url><loc>/blog/</loc><lastmod>2023-10-12T18:57:35+04:00</lastmod></url><url><loc>/categories/</loc><lastmod>2023-10-12T18:57:35+04:00</lastmod></url><url><loc>/catego [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.51.0/</loc><lastmod>2023-10-12T13:20:39-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2023-10-12T13:20:39-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2023-10-12T13:20:39-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2023-10-12T13:20:39-07:00</lastmod></url><url><loc>/catego [...]
\ No newline at end of file