You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/03/29 16:09:39 UTC
[beam] branch asf-site updated: Publishing website 2022/03/29 16:07:08 at commit c84818d
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 88c4ee3 Publishing website 2022/03/29 16:07:08 at commit c84818d
88c4ee3 is described below
commit 88c4ee3900275eac815bd10b503ad502fc3efa16
Author: jenkins <bu...@apache.org>
AuthorDate: Tue Mar 29 16:07:08 2022 +0000
Publishing website 2022/03/29 16:07:08 at commit c84818d
---
website/generated-content/documentation/index.xml | 38 ++++++++++++++++++++--
.../documentation/runtime/environments/index.html | 31 ++++++++++++++++--
.../sdks/python-pipeline-dependencies/index.html | 19 ++++++++---
website/generated-content/sitemap.xml | 2 +-
4 files changed, 80 insertions(+), 10 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index b233966..9c1774b 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -14313,10 +14313,11 @@ limitations under the License.
</blockquote>
<p>For optimal user experience, we also recommend you use the latest released version of Beam.</p>
<h3 id="building-and-pushing-custom-containers">Building and pushing custom containers</h3>
-<p>Beam <a href="https://hub.docker.com/search?q=apache%2Fbeam&amp;type=image">SDK container images</a> are built from Dockerfiles checked into the <a href="https://github.com/apache/beam">Github</a> repository and published to Docker Hub for every release. You can build customized containers in one of two ways:</p>
+<p>Beam <a href="https://hub.docker.com/search?q=apache%2Fbeam&amp;type=image">SDK container images</a> are built from Dockerfiles checked into the <a href="https://github.com/apache/beam">Github</a> repository and published to Docker Hub for every release. You can build customized containers in one of three ways:</p>
<ol>
<li><strong><a href="#writing-new-dockerfiles">Writing a new</a> Dockerfile based on a released container image</strong>. This is sufficient for simple additions to the image, such as adding artifacts or environment variables.</li>
<li><strong><a href="#modifying-dockerfiles">Modifying</a> a source Dockerfile in <a href="https://github.com/apache/beam">Beam</a></strong>. This method requires building from Beam source but allows for greater customization of the container (including replacement of artifacts or base OS/language versions).</li>
+<li><strong><a href="#modify-existing-base-image">Modifying</a> an existing container image to make it compatible with Apache Beam Runners</strong>. This method is used when users start from an existing image, and configure the image to be compatible with Apache Beam Runners.</li>
</ol>
<h4 id="writing-new-dockerfiles">Writing a new Dockerfile based on an existing published container image</h4>
<ol>
@@ -14410,7 +14411,40 @@ docker push &quot;${IMAGE_NAME}:${TAG}&quot;
<pre><code>./gradlew :sdks:java:container:java8:docker -Pdocker-pull-licenses
</code></pre><p>creates a Java 8 SDK image with appropriate licenses in <code>/opt/apache/beam/third_party_licenses/</code>.</p>
<p>By default, no licenses/notices are added to the docker images.</p>
-<h2 id="running-pipelines">Running pipelines with custom container images</h2>
+<h4 id="modify-existing-base-image">Modifying an existing container image to make it compatible with Apache Beam Runners</h4>
+<p>Beam offers a way to provide your own custom container image. The easiest way to build a new custom image that is compatible with Apache Beam Runners is to use a <a href="https://docs.docker.com/develop/develop-images/multistage-build/">multi-stage build</a> process. This copies over the necessary artifacts from a default Apache Beam base image to build your custom container image.</p>
+<ol>
+<li>Copy necessary artifacts from Apache Beam base image to your image.</li>
+</ol>
+<pre><code># This can be any container image,
+FROM python:3.7-bullseye
+# Install SDK. (needed for Python SDK)
+RUN pip install --no-cache-dir apache-beam[gcp]==2.35.0
+# Copy files from official SDK image, including script/dependencies.
+COPY --from=apache/beam_python3.7_sdk:2.35.0 /opt/apache/beam /opt/apache/beam
+# Perform any additional customizations if desired
+# Set the entrypoint to Apache Beam SDK launcher.
+ENTRYPOINT [&quot;/opt/apache/beam/boot&quot;]
+</code></pre><blockquote>
+<p><strong>NOTE</strong>: This example assumes necessary dependencies (in this case, Python 3.7 and pip) have been installed on the existing base image. Installing the Apache Beam SDK into the image will ensure that the image has the necessary SDK dependencies and reduce the worker startup time.
+The version specified in the <code>RUN</code> instruction must match the version used to launch the pipeline.<br>
+<strong>Make sure that the Python or Java runtime version specified in the base image is the same as the version used to run the pipeline.</strong></p>
+</blockquote>
+<ol start="2">
+<li><a href="https://docs.docker.com/engine/reference/commandline/build/">Build</a> and <a href="https://docs.docker.com/engine/reference/commandline/push/">push</a> the image using Docker.</li>
+</ol>
+<pre><code> export BASE_IMAGE=&quot;apache/beam_python3.7_sdk:2.25.0&quot;
+export IMAGE_NAME=&quot;myremoterepo/mybeamsdk&quot;
+export TAG=&quot;latest&quot;
+# Optional - pull the base image into your local Docker daemon to ensure
+# you have the most up-to-date version of the base image locally.
+docker pull &quot;${BASE_IMAGE}&quot;
+docker build -f Dockerfile -t &quot;${IMAGE_NAME}:${TAG}&quot; .
+</code></pre><ol start="3">
+<li>If your runner is running remotely, retag the image and <a href="https://docs.docker.com/engine/reference/commandline/push/">push</a> the image to your repository.</li>
+</ol>
+<pre><code>docker push &quot;${IMAGE_NAME}:${TAG}&quot;
+</code></pre><h2 id="running-pipelines">Running pipelines with custom container images</h2>
<p>The common method for providing a container image requires using the
PortableRunner flag <code>--environment_config</code> as supported by the Portable
Runner or by runners supported PortableRunner flags.
diff --git a/website/generated-content/documentation/runtime/environments/index.html b/website/generated-content/documentation/runtime/environments/index.html
index 6b3e3b7..79b680c 100644
--- a/website/generated-content/documentation/runtime/environments/index.html
+++ b/website/generated-content/documentation/runtime/environments/index.html
@@ -18,7 +18,7 @@
function addPlaceholder(){$('input:text').attr('placeholder',"What are you looking for?");}
function endSearch(){var search=document.querySelector(".searchBar");search.classList.add("disappear");var icons=document.querySelector("#iconsBar");icons.classList.remove("disappear");}
function blockScroll(){$("body").toggleClass("fixedPosition");}
-function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Documentation</span></li><li><a href=/documentation>Using the Documentation</a></li><li class=section-nav-item--collapsible><span class=section-nav-lis [...]
+function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Documentation</span></li><li><a href=/documentation>Using the Documentation</a></li><li class=section-nav-item--collapsible><span class=section-nav-lis [...]
ENV FOO=bar
COPY /src/path/to/file /dest/path/to/file/
@@ -72,7 +72,32 @@ docker tag apache/beam_python3.7_sdk "${IMAGE_NAME}:${TAG}"
docker push "${IMAGE_NAME}:${TAG}"
</code></pre><ol start=6><li>After pushing a container image, verify the remote image ID and digest matches the local image ID and digest output from <code>docker_images --digests</code>.</li></ol><h4 id=additional-build-parameters>Additional build parameters</h4><p>The docker Gradle task defines a default image repository and <a href=https://docs.docker.com/engine/reference/commandline/tag/>tag</a> is the SDK version defined at <a href=https://github.com/apache/beam/blob/master/gradle.p [...]
</code></pre><p>builds the Python 3.6 container and tags it as <code>example-repo/beam_python3.6_sdk:2.26.0-custom</code>.</p><p>From Beam 2.21.0 and later, a <code>docker-pull-licenses</code> flag was introduced to add licenses/notices for third party dependencies to the docker images. For example:</p><pre><code>./gradlew :sdks:java:container:java8:docker -Pdocker-pull-licenses
-</code></pre><p>creates a Java 8 SDK image with appropriate licenses in <code>/opt/apache/beam/third_party_licenses/</code>.</p><p>By default, no licenses/notices are added to the docker images.</p><h2 id=running-pipelines>Running pipelines with custom container images</h2><p>The common method for providing a container image requires using the
+</code></pre><p>creates a Java 8 SDK image with appropriate licenses in <code>/opt/apache/beam/third_party_licenses/</code>.</p><p>By default, no licenses/notices are added to the docker images.</p><h4 id=modify-existing-base-image>Modifying an existing container image to make it compatible with Apache Beam Runners</h4><p>Beam offers a way to provide your own custom container image. The easiest way to build a new custom image that is compatible with Apache Beam Runners is to use a <a hre [...]
+FROM python:3.7-bullseye
+
+# Install SDK. (needed for Python SDK)
+RUN pip install --no-cache-dir apache-beam[gcp]==2.35.0
+
+# Copy files from official SDK image, including script/dependencies.
+COPY --from=apache/beam_python3.7_sdk:2.35.0 /opt/apache/beam /opt/apache/beam
+
+# Perform any additional customizations if desired
+
+# Set the entrypoint to Apache Beam SDK launcher.
+ENTRYPOINT ["/opt/apache/beam/boot"]
+
+</code></pre><blockquote><p><strong>NOTE</strong>: This example assumes necessary dependencies (in this case, Python 3.7 and pip) have been installed on the existing base image. Installing the Apache Beam SDK into the image will ensure that the image has the necessary SDK dependencies and reduce the worker startup time.
+The version specified in the <code>RUN</code> instruction must match the version used to launch the pipeline.<br><strong>Make sure that the Python or Java runtime version specified in the base image is the same as the version used to run the pipeline.</strong></p></blockquote><ol start=2><li><a href=https://docs.docker.com/engine/reference/commandline/build/>Build</a> and <a href=https://docs.docker.com/engine/reference/commandline/push/>push</a> the image using Docker.</li></ol><pre><co [...]
+ export IMAGE_NAME="myremoterepo/mybeamsdk"
+ export TAG="latest"
+
+ # Optional - pull the base image into your local Docker daemon to ensure
+ # you have the most up-to-date version of the base image locally.
+ docker pull "${BASE_IMAGE}"
+
+ docker build -f Dockerfile -t "${IMAGE_NAME}:${TAG}" .
+</code></pre><ol start=3><li>If your runner is running remotely, retag the image and <a href=https://docs.docker.com/engine/reference/commandline/push/>push</a> the image to your repository.</li></ol><pre><code>docker push "${IMAGE_NAME}:${TAG}"
+</code></pre><h2 id=running-pipelines>Running pipelines with custom container images</h2><p>The common method for providing a container image requires using the
PortableRunner flag <code>--environment_config</code> as supported by the Portable
Runner or by runners supported PortableRunner flags.
Other runners, such as Dataflow, support specifying containers with different flags.</p><div class="runner-direct snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><pre><code class=language-direct data-lang=direct>export IMAGE="my-repo/beam_python_sdk_custom"
@@ -142,7 +167,7 @@ accessed by any third-party service, if needed.</li><li>Local runners attempt to
images. If an image cannot be pulled locally (by the docker daemon),
you may see an log message like:<pre><code>Error response from daemon: manifest for remote.repo/beam_python3.7_sdk:2.25.0-custom not found: manifest unknown: ...
INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Unable to pull image...
-</code></pre></li></ul><div class=feedback><p class=update>Last updated on 2022/03/03</p><h3>Have you found everything you were looking for?</h3><p class=description>Was it all useful and clear? Is there anything that you would like to change? Let us know!</p><button class=load-button><a href="mailto:dev@beam.apache.org?subject=Beam Website Feedback">SEND FEEDBACK</a></button></div></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class="footer__c [...]
+</code></pre></li></ul><div class=feedback><p class=update>Last updated on 2022/03/29</p><h3>Have you found everything you were looking for?</h3><p class=description>Was it all useful and clear? Is there anything that you would like to change? Let us know!</p><button class=load-button><a href="mailto:dev@beam.apache.org?subject=Beam Website Feedback">SEND FEEDBACK</a></button></div></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class="footer__c [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
index ee1a2fc..a54d102 100644
--- a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
+++ b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
@@ -18,10 +18,12 @@
function addPlaceholder(){$('input:text').attr('placeholder',"What are you looking for?");}
function endSearch(){var search=document.querySelector(".searchBar");search.classList.add("disappear");var icons=document.querySelector("#iconsBar");icons.classList.remove("disappear");}
function blockScroll(){$("body").toggleClass("fixedPosition");}
-function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Languages</span></li><li><span class=section-nav-list-title>Java</span><ul class=section-nav-list><li><a href=/documentation/sdks/java/>Java SDK overvi [...]
+function openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfix container-main-content"><div class="section-nav closed" data-offset-top=90 data-offset-bottom=500><span class="section-nav-back glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list data-section-nav><li><span class=section-nav-list-main-title>Languages</span></li><li><span class=section-nav-list-title>Java</span><ul class=section-nav-list><li><a href=/documentation/sdks/java/>Java SDK overvi [...]
</code></pre><p>This command creates a <code>requirements.txt</code> file that lists all packages that are installed on your machine, regardless of where they were installed from.</p></li><li><p>Edit the <code>requirements.txt</code> file and leave only the packages that were installed from PyPI and are used in the workflow source. Delete all packages that are not relevant to your code.</p></li><li><p>Run your pipeline with the following command-line option:</p><pre><code> --requirements [...]
-</code></pre><p>The runner will use the <code>requirements.txt</code> file to install your additional dependencies onto the remote workers.</p></li></ol><p><strong>Important:</strong> Remote workers will install all packages listed in the <code>requirements.txt</code> file. Because of this, it’s very important that you delete non-PyPI packages from the <code>requirements.txt</code> file, as stated in step 2. If you don’t remove non-PyPI packages, the remote workers will fail [...]
-</code></pre><p>This command lists all packages that are installed on your machine, regardless of where they were installed from.</p></li><li><p>Run your pipeline with the following command-line option:</p><pre><code> --extra_package /path/to/package/package-name
+</code></pre><p>The runner will use the <code>requirements.txt</code> file to install your additional dependencies onto the remote workers.</p></li></ol><p><strong>Important:</strong> Remote workers will install all packages listed in the <code>requirements.txt</code> file. Because of this, it’s very important that you delete non-PyPI packages from the <code>requirements.txt</code> file, as stated in step 2. If you don’t remove non-PyPI packages, the remote workers will fail [...]
+COPY <path to requirements.txt> /tmp/requirements.txt
+RUN python -m pip install -r /tmp/requirements.txt
+</code></pre></li></ol><h2 id=local-or-nonpypi>Local or non-PyPI Dependencies</h2><p>If your pipeline uses packages that are not available publicly (e.g. packages that you’ve downloaded from a GitHub repo), make these packages available remotely by performing the following steps:</p><ol><li><p>Identify which packages are installed on your machine and are not public. Run the following command:</p><p>pip freeze</p><p>This command lists all packages that are installed on your machine, [...]
</code></pre><p>where package-name is the package’s tarball. If you have the <code>setup.py</code> for that
package then you can build the tarball with the following command:</p><pre><code> python setup.py sdist
</code></pre><p>See the <a href=https://docs.python.org/2/distutils/sourcedist.html>sdist documentation</a> for more details on this command.</p></li></ol><h2 id=multiple-file-dependencies>Multiple File Dependencies</h2><p>Often, your pipeline code spans multiple files. To run your project remotely, you must group these files as a Python package and specify the package when you run your pipeline. When the remote workers start, they will install your package. To group your files as a Pyth [...]
@@ -42,7 +44,16 @@ package then you can build the tarball with the following command:</p><pre><code
main.py
other_files_dir/
</code></pre><p>See the <a href=https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/juliaset>Juliaset</a> project for an example that follows this required project structure.</p></li><li><p>Run your pipeline with the following command-line option:</p><pre><code> --setup_file /path/to/setup.py
-</code></pre></li></ol><p><strong>Note:</strong> Because custom commands execute after the dependencies for your workflow are installed (by <code>pip</code>), you should omit the PyPI package dependency from the pipeline’s <code>requirements.txt</code> file and from the <code>install_requires</code> parameter in the <code>setuptools.setup()</code> call of your <code>setup.py</code> file.</p></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div c [...]
+</code></pre></li></ol><p><strong>Note:</strong> Because custom commands execute after the dependencies for your workflow are installed (by <code>pip</code>), you should omit the PyPI package dependency from the pipeline’s <code>requirements.txt</code> file and from the <code>install_requires</code> parameter in the <code>setuptools.setup()</code> call of your <code>setup.py</code> file.</p><h2 id=pre-building-sdk-container-image>Pre-building SDK container image</h2><p>In pipeline [...]
+However, it may be possible to pre-build the SDK containers and perform the dependency installation once before the workers start. To pre-build the container image before pipeline submission, provide the pipeline options mentioned below.</p><ol><li><p>Provide the container engine. Beam supports <code>local_docker</code>(requires local installation of Docker) and <code>cloud_build</code>(requires a GCP project with Cloud Build API enabled).</p><pre><code>--prebuild_sdk_container_engine=&l [...]
+</code></pre></li><li><p>If using <code>local_docker</code> engine, provide a URL for the remote registry to which the image will be pushed by passing</p><pre><code>--docker_registry_push_url=<remote_registry_url>
+# Example: --docker_registry_push_url=<registry_name>/beam
+# pre-built image will be pushed to the <registry_name>/beam/beam_python_prebuilt_sdk:<unique_image_tag>
+# <unique_image_tag> tag is generated by Beam SDK.
+</code></pre><p><strong>NOTE:</strong> <code>docker_registry_push_url</code> must be a remote registry.</p></li></ol><blockquote><p>The pre-building feature requires the Apache Beam SDK for Python, version 2.25.0 or later.
+The container images created during prebuilding will persist beyond the pipeline runtime.
+Once your job is finished or stopped, you can remove the pre-built image from the container registry.</p></blockquote><blockquote><p>If your pipeline is using a custom container image, most likely you will not benefit from pre-building step as extra dependencies can be preinstalled in the custom image at build time. If you still would like to use pre-building with custom images, use Apache Beam SDK 2.38.0 or newer and
+supply your custom image via <code>--sdk_container_image</code> pipeline option.</p></blockquote><p><strong>NOTE</strong>: This feature is available only for the <code>Dataflow Runner v2</code>.</p></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class="footer__cols__col footer__cols__col__logos"><div class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg class=footer__logo alt="Beam logo"></div><div class=footer__cols__col__logo><im [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index 90dc234..5eb00de 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.37.0/</loc><lastmod>2022-03-04T10:14:02-08:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/blog/u [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.37.0/</loc><lastmod>2022-03-04T10:14:02-08:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2022-03-28T08:41:34-07:00</lastmod></url><url><loc>/blog/u [...]
\ No newline at end of file