[GitHub] [beam] damccorm commented on a diff in pull request #22587: WIP: Dataframe API ML preprocessing notebook

[GitBox <gi...@apache.org>]

damccorm commented on code in PR #22587:
URL: https://github.com/apache/beam/pull/22587#discussion_r1013020478


##########
examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb:
##########
@@ -0,0 +1,3496 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "source": [
+        "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+        "\n",
+        "# Licensed to the Apache Software Foundation (ASF) under one\n",
+        "# or more contributor license agreements. See the NOTICE file\n",
+        "# distributed with this work for additional information\n",
+        "# regarding copyright ownership. The ASF licenses this file\n",
+        "# to you under the Apache License, Version 2.0 (the\n",
+        "# \"License\"); you may not use this file except in compliance\n",
+        "# with the License. You may obtain a copy of the License at\n",
+        "#\n",
+        "#   http://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing,\n",
+        "# software distributed under the License is distributed on an\n",
+        "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+        "# KIND, either express or implied. See the License for the\n",
+        "# specific language governing permissions and limitations\n",
+        "# under the License."
+      ],
+      "metadata": {
+        "id": "sARMhsXz8yR1",
+        "cellView": "form"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Overview\n",
+        "\n",
+        "One of the most common tools used for data exploration and pre-processing is [pandas DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html). Pandas has become very popular for its ease of use. It has very intuitive methods to perform common analytical tasks and data pre-processing. \n",
+        "\n",
+        "Pandas loads all of the data into memory on a single machine (one node) for rapid execution. This works well when dealing with small-scale datasets. However, many projects involve datasets that can grow too big to fit in memory. These use cases generally require the usage of parallel data processing frameworks such as Apache Beam.\n",
+        "\n",
+        "\n",
+        "## Beam DataFrames\n",
+        "\n",
+        "\n",
+        "Beam DataFrames provide a pandas-like\n",
+        "API to declare and define Beam processing pipelines. It provides a familiar interface for machine learning practioners to build complex data-processing pipelines by only invoking standard pandas commands.\n",
+        "\n",
+        "> ℹ️ To learn more about Beam DataFrames, take a look at the\n",
+        "[Beam DataFrames overview](https://beam.apache.org/documentation/dsls/dataframes/overview) page.\n",
+        "\n",
+        "## Goal\n",
+        "The goal of this notebook is to explore a dataset preprocessed it for machine learning model training using the Beam DataFrames API.\n",
+        "\n",
+        "\n",
+        "## Tutorial outline\n",
+        "\n",
+        "In this notebook, we walk through the use of the Beam DataFrames API to perform common data exploration as well as pre-processing steps that are necessary to prepare your dataset for machine learning model training and inference, such as:  \n",
+        "\n",
+        "*   Removing unwanted columns.\n",
+        "*   One-hot encoding categorical columns.\n",
+        "*   Normalizing numerical columns.\n",
+        "\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "iFZC1inKuUCy"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Installation\n",
+        "\n",
+        "As we want to explore the elements within a `PCollection`, we can make use of the the Interactive runner by installing Apache Beam with the `interactive` component. The latest implemented DataFrames API methods invoked in this notebook are available in Beam <b>2.43</b> or later.\n"
+      ],
+      "metadata": {
+        "id": "A0f2HJ22D4lt"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "pCjwrwNWnuqI"
+      },
+      "source": [
+        "Install latest version"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "-OJC0Xn5Um-C",
+        "beam:comment": "TODO(https://github.com/apache/beam/XXXX): Just install 2.43.0 once it's released, [`issue 23276`](https://github.com/apache/beam/issues/23276)  is currently not implemented for Beam 2.42 (required fix for implementing `str.get_dummies()`"

Review Comment:
   ```suggestion
           "beam:comment": "TODO(https://github.com/apache/beam/issues/23961): Just install 2.43.0 once it's released, [`issue 23276`](https://github.com/apache/beam/issues/23276)  is currently not implemented for Beam 2.42 (required fix for implementing `str.get_dummies()`"
   ```
   
   I filed an issue for this, could we reference it directly?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org