You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by da...@apache.org on 2022/10/04 13:18:50 UTC
[beam] branch master updated: AI/ML pipelines master page documentation (#23443)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 53377832323 AI/ML pipelines master page documentation (#23443)
53377832323 is described below
commit 533778323237a4e30665f765b20671729dfa5b41
Author: Juta Staes <ju...@gmail.com>
AuthorDate: Tue Oct 4 15:18:38 2022 +0200
AI/ML pipelines master page documentation (#23443)
* add AI/ML pipelines master page to documentation
* Update website/www/site/content/en/documentation/ml/overview.md
Co-authored-by: Danny McCormick <da...@google.com>
Co-authored-by: Danny McCormick <da...@google.com>
---
.../site/content/en/documentation/ml/overview.md | 61 ++++++++++++++++++++++
.../partials/section-menu/en/documentation.html | 1 +
website/www/site/static/images/ml-workflows.svg | 17 ++++++
3 files changed, 79 insertions(+)
diff --git a/website/www/site/content/en/documentation/ml/overview.md b/website/www/site/content/en/documentation/ml/overview.md
new file mode 100755
index 00000000000..1c6e4e03fd0
--- /dev/null
+++ b/website/www/site/content/en/documentation/ml/overview.md
@@ -0,0 +1,61 @@
+---
+title: "Overview"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# AI/ML pipelines
+
+Beam <3 Machine Learning. Being productive and successful as a machine learning practitioner is often dependent on your ability to efficiently leverage large volumes of data in a way that is uniquely tailored to your resources, requirements and budget. When starting your next AI/ML project or upscaling an existing one, a vital tool you should consider adding to your project is Beam.
+
+Beam enables you to process large volumes of data, both for preprocessing and for inference. It allows you to experiment with your data during the exploration phase of your project, while providing a seamless transition to upscaling your data pipelines as part of your MLOps ecosystem in a production environment. It allows you to run your model in production on a varying data load, both in batch and streaming.
+
+## AI/ML workloads
+
+Let’s take a look at the different building blocks that we need to build an end-to-end AI/ML use case, and where Beam will come in handy in building those blocks.
+
+
+
+1. Data ingestion: incoming new data will be stored in your filesystem, database or published on a messaging queue.
+2. **Data validation**: once you have received your data you need to check the quality of your data such as detecting outliers and reporting on standard deviations and class distributions.
+3. **Data preprocessing**: after validating your data, you need to transform it so that it is ready to be used for training your model.
+4. Model training: once your data is ready, you can start training your AI/ML model. This step will typically be repeated multiple times depending on the quality of your trained model.
+5. Model validation: before deploying your new model you need to validate its performance and accuracy.
+6. **Model deployment**: finally you can deploy your model, meaning it can run inference on any new or existing data.
+
+All of these steps can be executed multiple times, as your data might grow and evolve over time and you want your model to stay up to date and guarantee its best performance. This is why it is very important to apply MLOps to your project, meaning that you aim to automate the AI/ML workflows throughout the model and data lifecycle. This can be achieved by using orchestrators that automate this flow and handle the transition between the different building blocks in your project.
+
+Beam can be used for data validation, data preprocessing and model deployment/inference. We will now take a look at these different building blocks in more detail and at how they can be orchestrated. Finally, you can also find full examples of AI/ML pipelines in Beam.
+
+## Data processing
+
+Data validation and preprocessing can be done in Beam by setting up data pipelines that transform your data and output metrics computed from your data. Beam has a rich set of [IO connectors](https://beam.apache.org/documentation/io/built-in/) for ingesting and writing data, which means you can easily integrate it with your existing filesystem, database or messaging queue. When developing your ML model, you can also first explore your data with the [Beam DataFrame API](https://beam.apache [...]
+
+## Inference
+
+There are several ways to use and deploy your model:
+1. Making it available for online predictions via an API
+2. Running it in real-time as new data becomes available in a pipeline
+3. Running it in batch on an existing dataset
+
+Beam is ideally suitable for the last 2 use cases. In this case your data will run through a pipeline (streaming or batch), and you can obtain predictions by running inference in one of the steps of your pipeline. Beam provides the [RunInference API](https://beam.apache.org/documentation/sdks/python-machine-learning/) to facilitate the integration of your model into a pipeline step. When running your model, a common requirement is to enable GPU execution. Beam also provides support for this.
+
+## Orchestrators
+
+In order to automate and track the AI/ML workflows throughout your project, you can use orchestrators such as [Kubeflow pipelines](https://www.kubeflow.org/docs/components/pipelines/introduction/) (KFP) or [TensorFlow Extended](https://www.tensorflow.org/tfx) (TFX) that automate your different building blocks and handle the transitions between them. When you use Beam as one of the building blocks in your project, these orchestrators will then be able to launch your Beam job and keep trac [...]
+
+## Examples
+
+You can find examples of end-to-end AI/ML pipelines for several use cases:
+* [Online Clustering in Beam](/documentation/ml/online-clustering)
diff --git a/website/www/site/layouts/partials/section-menu/en/documentation.html b/website/www/site/layouts/partials/section-menu/en/documentation.html
index 6a6ff67e032..f55e30f6c4d 100644
--- a/website/www/site/layouts/partials/section-menu/en/documentation.html
+++ b/website/www/site/layouts/partials/section-menu/en/documentation.html
@@ -212,6 +212,7 @@
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">AI/ML pipelines</span>
<ul class="section-nav-list">
+ <li><a href="/documentation/ml/overview/">Overview</a></li>
<li><a href="/documentation/ml/online-clustering/">Online Clustering</a></li>
</ul>
</li>
diff --git a/website/www/site/static/images/ml-workflows.svg b/website/www/site/static/images/ml-workflows.svg
new file mode 100755
index 00000000000..2a9cb3c1f27
--- /dev/null
+++ b/website/www/site/static/images/ml-workflows.svg
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<!-- Do not edit this file with editors other than diagrams.net -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="890px" height="290px" viewBox="-0.5 -0.5 890 290" content="<mxfile host="app.diagrams.net" modified="2022-09-30T09:13:42.911Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" etag="jwziFvvco7NdpbSyV7Nh" version="20.3.7" type="google"><diagram id="8C2 [...]
\ No newline at end of file