You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/08/09 23:55:19 UTC
[GitHub] [beam] TheNeuralBit commented on a diff in pull request #22587: WIP: Dataframe API ML preprocessing notebook
TheNeuralBit commented on code in PR #22587:
URL: https://github.com/apache/beam/pull/22587#discussion_r941893216
##########
examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb:
##########
@@ -0,0 +1,1907 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Overview\n",
+ "\n",
+ "One of the most common tools used for data exploration and pre-processing is [pandas DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html). Pandas has become very popular for its ease of use. It has very intuitive methods to perform common analytical tasks and data pre-processing. \n",
+ "\n",
+ "Pandas loads all of the data into memory on a single machine (one node) for rapid execution. This works well when dealing with small-scale datasets. However, many projects involve datasets that can grow too big to fit in memory. These use cases generally require the usage of parallel data processing frameworks such as Apache Beam.\n",
+ "\n",
+ "\n",
+ "## Beam DataFrames\n",
+ "\n",
+ "\n",
+ "Beam DataFrames provide a pandas-like DataFrame\n",
+ "API to declare and define Beam processing pipelines. It provides a familiar interface for machine learning practioners to build complex data-processing pipelines by only invoking standard pandas commands.\n",
+ "\n",
+ "> ℹ️ To learn more about Beam DataFrames, take a look at the\n",
+ "[Beam DataFrames overview](https://beam.apache.org/documentation/dsls/dataframes/overview) page.\n",
+ "\n",
+ "## Tutorial outline\n",
+ "\n",
+ "In this notebook, we walk through the use of the Beam DataFrames API to perform common data exploration as well as pre-processing steps that are necessary to prepare your dataset for machine learning model training and inference, such as: \n",
+ "\n",
+ "* Removing unwanted columns.\n",
+ "* One-hot encoding categorical columns.\n",
+ "* Normalizing numerical columns.\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "iFZC1inKuUCy"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Installation\n",
+ "\n",
+ "First, we need to install Apache Beam with the `interactive` component to be able to use the Interactive runner. The latest implemented DataFrames API methods invoked in this notebook are available in Beam <b>2.41</b> or later.\n"
+ ],
+ "metadata": {
+ "id": "A0f2HJ22D4lt"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pCjwrwNWnuqI"
+ },
+ "source": [
+ "**Option 1:** Install latest version with implemented df.mean()\n",
+ "\n",
+ "TODO: Remove this text later"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "-OJC0Xn5Um-C"
+ },
+ "outputs": [],
+ "source": [
+ "!git clone https://github.com/apache/beam.git\n",
+ "\n",
+ "!cd beam/sdks/python && pip3 install -r build-requirements.txt \n",
+ "\n",
+ "%pip install -e beam/sdks/python/.[interactive,gcp]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xfXzNzA1n3ZP"
+ },
+ "source": [
+ "**Option 2:** Install latest release version \n",
+ "\n",
+ "**[12/07/2022]:** df.mean() is currently not supported for this version (beam 2.40)\n",
+ "\n",
+ "TODO: Remove this text later"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "4xY7ECJZOuJj"
+ },
+ "outputs": [],
+ "source": [
+ "! pip install apache-beam[interactive,gcp]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Part I : Local exploration with the Interactive Beam runner\n",
+ "We first use the [Interactive Beam](https://beam.apache.org/releases/pydoc/2.20.0/apache_beam.runners.interactive.interactive_beam.html) to explore and develop our pipeline.\n",
+ "This allows us to quickly test our pipeline locally before running it on a distributed runner. \n",
+ "\n",
+ "\n",
+ "> ℹ️ In this section, we will only be working with a subset of the original dataset since we're only using the the compute resources of the notebook instance.\n"
+ ],
+ "metadata": {
+ "id": "3NO6RgB7GkkE"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5I3G094hoB1P"
+ },
+ "source": [
+ "# Loading the data\n",
+ "\n",
+ "Pandas has the\n",
+ "[`pandas.read_csv`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)\n",
+ "function to easily read CSV files into DataFrames.\n",
+ "We're using the beam\n",
+ "[`beam.dataframe.io.read_csv`](https://beam.apache.org/releases/pydoc/current/apache_beam.dataframe.io.html#apache_beam.dataframe.io.read_csv)\n",
+ "function that emulates `pandas.read_csv`. The main difference between them is that the beam method returns a deferred Beam DataFrame while pandas return a standard DataFrame.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "X3_OB9cAULav"
+ },
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "\n",
+ "import numpy as np\n",
+ "import pandas as pd \n",
+ "import apache_beam as beam\n",
+ "import apache_beam.runners.interactive.interactive_beam as ib\n",
+ "from apache_beam.runners.interactive.interactive_runner import InteractiveRunner\n",
+ "from apache_beam.runners.dataflow import DataflowRunner\n",
+ "\n",
+ "# Available options: [sample_1000, sample_10000, sample_100000, sample] where\n",
+ "# sample contains all of the dataset (around 1000000 samples)\n",
+ "file_location = 'gs://apache-beam-samples/nasa_jpl_asteroid/sample_10000.csv'\n",
+ "\n",
+ "# Initialize pipline\n",
+ "p = beam.Pipeline(InteractiveRunner())\n",
+ "\n",
+ "# Create a deferred Beam DataFrame with the contents of our csv file.\n",
+ "beam_df = p | beam.dataframe.io.read_csv(file_location, splittable=True)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "paf7yf3YpCh8"
+ },
+ "source": [
+ "# Data pre-processing\n",
+ "\n",
+ "## Dataset description \n",
+ "\n",
+ "### [NASA - Nearest Earth Objects dataset](https://cneos.jpl.nasa.gov/ca/)\n",
+ "There are an innumerable number of objects in the outer space. Some of them are closer than we think. Even though we might think that a distance of 70,000 Km can not potentially harm us, but at an astronomical scale, this is a very small distance and can disrupt many natural phenomena. \n",
+ "\n",
+ "These objects/asteroids can thus prove to be harmful. Hence, it is wise to know what is surrounding us and what can harm us amongst those. Thus, this dataset compiles the list of NASA certified asteroids that are classified as the nearest earth object."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "\n",
+ "Let's first inspect the columns of our dataset and their types"
+ ],
+ "metadata": {
+ "id": "cvAu5T0ENjuQ"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "LwW77ixE-pjR",
+ "outputId": "c24ff83d-3a13-47a6-c9c2-3978729fde82"
+ },
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "spk_id int64\n",
+ "full_name object\n",
+ "near_earth_object object\n",
+ "absolute_magnitude float64\n",
+ "diameter float64\n",
+ "albedo float64\n",
+ "diameter_sigma float64\n",
+ "eccentricity float64\n",
+ "inclination float64\n",
+ "moid_ld float64\n",
+ "object_class object\n",
+ "semi_major_axis_au_unit float64\n",
+ "hazardous_flag object\n",
+ "dtype: object"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 4
+ }
+ ],
+ "source": [
+ "beam_df.dtypes"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "When using Interactive Beam, we can use `ib.collect()` to bring a Beam DataFrame into local memory as a Pandas DataFrame."
+ ],
+ "metadata": {
+ "id": "1Wa6fpbyQige"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 746
+ },
+ "id": "DPxkAmkpq4Xv",
+ "outputId": "14fa80de-2dee-4963-99d8-3e321f949ff8"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n",
+ " <div id=\"progress_indicator_a986c6cc61ed5a5b622e163d92f73775\">\n",
+ " <div class=\"spinner-border text-info\" role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing... collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery == 'undefined') {\n",
+ " var jqueryScript = document.createElement('script');\n",
+ " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript = document.createElement('script');\n",
+ " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery = jQuery.noConflict(true);\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_a986c6cc61ed5a5b622e163d92f73775\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_a986c6cc61ed5a5b622e163d92f73775\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " spk_id full_name near_earth_object \\\n",
+ "0 2000001 1 Ceres N \n",
+ "1 2000002 2 Pallas N \n",
+ "2 2000003 3 Juno N \n",
+ "3 2000004 4 Vesta N \n",
+ "4 2000005 5 Astraea N \n",
+ "... ... ... ... \n",
+ "9994 2009995 9995 Alouette (4805 P-L) N \n",
+ "9995 2009996 9996 ANS (9070 P-L) N \n",
+ "9996 2009997 9997 COBE (1217 T-1) N \n",
+ "9997 2009998 9998 ISO (1293 T-1) N \n",
+ "9998 2009999 9999 Wiles (4196 T-2) N \n",
+ "\n",
+ " absolute_magnitude diameter albedo diameter_sigma eccentricity \\\n",
+ "0 3.40 939.400 0.0900 0.200 0.076009 \n",
+ "1 4.20 545.000 0.1010 18.000 0.229972 \n",
+ "2 5.33 246.596 0.2140 10.594 0.256936 \n",
+ "3 3.00 525.400 0.4228 0.200 0.088721 \n",
+ "4 6.90 106.699 0.2740 3.140 0.190913 \n",
+ "... ... ... ... ... ... \n",
+ "9994 15.10 2.564 0.2450 0.550 0.160610 \n",
+ "9995 13.60 8.978 0.1130 0.376 0.235174 \n",
+ "9996 14.30 NaN NaN NaN 0.113059 \n",
+ "9997 15.10 2.235 0.3880 0.373 0.093852 \n",
+ "9998 13.00 7.148 0.2620 0.065 0.071351 \n",
+ "\n",
+ " inclination moid_ld object_class semi_major_axis_au_unit \\\n",
+ "0 10.594067 620.640533 MBA 2.769165 \n",
+ "1 34.832932 480.348639 MBA 2.773841 \n",
+ "2 12.991043 402.514639 MBA 2.668285 \n",
+ "3 7.141771 443.451432 MBA 2.361418 \n",
+ "4 5.367427 426.433027 MBA 2.574037 \n",
+ "... ... ... ... ... \n",
+ "9994 2.311731 388.723233 MBA 2.390249 \n",
+ "9995 7.657713 444.194746 MBA 2.796605 \n",
+ "9996 2.459643 495.460110 MBA 2.545674 \n",
+ "9997 3.912263 373.848377 MBA 2.160961 \n",
+ "9998 3.198839 632.144398 MBA 2.839917 \n",
+ "\n",
+ " hazardous_flag \n",
+ "0 N \n",
+ "1 N \n",
+ "2 N \n",
+ "3 N \n",
+ "4 N \n",
+ "... ... \n",
+ "9994 N \n",
+ "9995 N \n",
+ "9996 N \n",
+ "9997 N \n",
+ "9998 N \n",
+ "\n",
+ "[9999 rows x 13 columns]"
+ ],
+ "text/html": [
+ "\n",
+ " <div id=\"df-0161aa69-d50f-4d6f-84c1-10dacb278880\">\n",
+ " <div class=\"colab-df-container\">\n",
+ " <div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>spk_id</th>\n",
+ " <th>full_name</th>\n",
+ " <th>near_earth_object</th>\n",
+ " <th>absolute_magnitude</th>\n",
+ " <th>diameter</th>\n",
+ " <th>albedo</th>\n",
+ " <th>diameter_sigma</th>\n",
+ " <th>eccentricity</th>\n",
+ " <th>inclination</th>\n",
+ " <th>moid_ld</th>\n",
+ " <th>object_class</th>\n",
+ " <th>semi_major_axis_au_unit</th>\n",
+ " <th>hazardous_flag</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>0</th>\n",
+ " <td>2000001</td>\n",
+ " <td>1 Ceres</td>\n",
+ " <td>N</td>\n",
+ " <td>3.40</td>\n",
+ " <td>939.400</td>\n",
+ " <td>0.0900</td>\n",
+ " <td>0.200</td>\n",
+ " <td>0.076009</td>\n",
+ " <td>10.594067</td>\n",
+ " <td>620.640533</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.769165</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>1</th>\n",
+ " <td>2000002</td>\n",
+ " <td>2 Pallas</td>\n",
+ " <td>N</td>\n",
+ " <td>4.20</td>\n",
+ " <td>545.000</td>\n",
+ " <td>0.1010</td>\n",
+ " <td>18.000</td>\n",
+ " <td>0.229972</td>\n",
+ " <td>34.832932</td>\n",
+ " <td>480.348639</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.773841</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>2</th>\n",
+ " <td>2000003</td>\n",
+ " <td>3 Juno</td>\n",
+ " <td>N</td>\n",
+ " <td>5.33</td>\n",
+ " <td>246.596</td>\n",
+ " <td>0.2140</td>\n",
+ " <td>10.594</td>\n",
+ " <td>0.256936</td>\n",
+ " <td>12.991043</td>\n",
+ " <td>402.514639</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.668285</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>3</th>\n",
+ " <td>2000004</td>\n",
+ " <td>4 Vesta</td>\n",
+ " <td>N</td>\n",
+ " <td>3.00</td>\n",
+ " <td>525.400</td>\n",
+ " <td>0.4228</td>\n",
+ " <td>0.200</td>\n",
+ " <td>0.088721</td>\n",
+ " <td>7.141771</td>\n",
+ " <td>443.451432</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.361418</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>4</th>\n",
+ " <td>2000005</td>\n",
+ " <td>5 Astraea</td>\n",
+ " <td>N</td>\n",
+ " <td>6.90</td>\n",
+ " <td>106.699</td>\n",
+ " <td>0.2740</td>\n",
+ " <td>3.140</td>\n",
+ " <td>0.190913</td>\n",
+ " <td>5.367427</td>\n",
+ " <td>426.433027</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.574037</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>...</th>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9994</th>\n",
+ " <td>2009995</td>\n",
+ " <td>9995 Alouette (4805 P-L)</td>\n",
+ " <td>N</td>\n",
+ " <td>15.10</td>\n",
+ " <td>2.564</td>\n",
+ " <td>0.2450</td>\n",
+ " <td>0.550</td>\n",
+ " <td>0.160610</td>\n",
+ " <td>2.311731</td>\n",
+ " <td>388.723233</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.390249</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9995</th>\n",
+ " <td>2009996</td>\n",
+ " <td>9996 ANS (9070 P-L)</td>\n",
+ " <td>N</td>\n",
+ " <td>13.60</td>\n",
+ " <td>8.978</td>\n",
+ " <td>0.1130</td>\n",
+ " <td>0.376</td>\n",
+ " <td>0.235174</td>\n",
+ " <td>7.657713</td>\n",
+ " <td>444.194746</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.796605</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9996</th>\n",
+ " <td>2009997</td>\n",
+ " <td>9997 COBE (1217 T-1)</td>\n",
+ " <td>N</td>\n",
+ " <td>14.30</td>\n",
+ " <td>NaN</td>\n",
+ " <td>NaN</td>\n",
+ " <td>NaN</td>\n",
+ " <td>0.113059</td>\n",
+ " <td>2.459643</td>\n",
+ " <td>495.460110</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.545674</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9997</th>\n",
+ " <td>2009998</td>\n",
+ " <td>9998 ISO (1293 T-1)</td>\n",
+ " <td>N</td>\n",
+ " <td>15.10</td>\n",
+ " <td>2.235</td>\n",
+ " <td>0.3880</td>\n",
+ " <td>0.373</td>\n",
+ " <td>0.093852</td>\n",
+ " <td>3.912263</td>\n",
+ " <td>373.848377</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.160961</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9998</th>\n",
+ " <td>2009999</td>\n",
+ " <td>9999 Wiles (4196 T-2)</td>\n",
+ " <td>N</td>\n",
+ " <td>13.00</td>\n",
+ " <td>7.148</td>\n",
+ " <td>0.2620</td>\n",
+ " <td>0.065</td>\n",
+ " <td>0.071351</td>\n",
+ " <td>3.198839</td>\n",
+ " <td>632.144398</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.839917</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "<p>9999 rows × 13 columns</p>\n",
+ "</div>\n",
+ " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0161aa69-d50f-4d6f-84c1-10dacb278880')\"\n",
+ " title=\"Convert this dataframe to an interactive table.\"\n",
+ " style=\"display:none;\">\n",
+ " \n",
+ " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
+ " width=\"24px\">\n",
+ " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+ " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
+ " </svg>\n",
+ " </button>\n",
+ " \n",
+ " <style>\n",
+ " .colab-df-container {\n",
+ " display:flex;\n",
+ " flex-wrap:wrap;\n",
+ " gap: 12px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert {\n",
+ " background-color: #E8F0FE;\n",
+ " border: none;\n",
+ " border-radius: 50%;\n",
+ " cursor: pointer;\n",
+ " display: none;\n",
+ " fill: #1967D2;\n",
+ " height: 32px;\n",
+ " padding: 0 0 0 0;\n",
+ " width: 32px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert:hover {\n",
+ " background-color: #E2EBFA;\n",
+ " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
+ " fill: #174EA6;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert {\n",
+ " background-color: #3B4455;\n",
+ " fill: #D2E3FC;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert:hover {\n",
+ " background-color: #434B5C;\n",
+ " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+ " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+ " fill: #FFFFFF;\n",
+ " }\n",
+ " </style>\n",
+ "\n",
+ " <script>\n",
+ " const buttonEl =\n",
+ " document.querySelector('#df-0161aa69-d50f-4d6f-84c1-10dacb278880 button.colab-df-convert');\n",
+ " buttonEl.style.display =\n",
+ " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
+ "\n",
+ " async function convertToInteractive(key) {\n",
+ " const element = document.querySelector('#df-0161aa69-d50f-4d6f-84c1-10dacb278880');\n",
+ " const dataTable =\n",
+ " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
+ " [key], {});\n",
+ " if (!dataTable) return;\n",
+ "\n",
+ " const docLinkHtml = 'Like what you see? Visit the ' +\n",
+ " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
+ " + ' to learn more about interactive tables.';\n",
+ " element.innerHTML = '';\n",
+ " dataTable['output_type'] = 'display_data';\n",
+ " await google.colab.output.renderOutput(dataTable, element);\n",
+ " const docLink = document.createElement('div');\n",
+ " docLink.innerHTML = docLinkHtml;\n",
+ " element.appendChild(docLink);\n",
+ " }\n",
+ " </script>\n",
+ " </div>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {},
+ "execution_count": 5
+ }
+ ],
+ "source": [
+ "ib.collect(beam_df)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can see that our datasets consists of both:\n",
+ "\n",
+ "* **Numerical columns:** These columns need to be transformed through [normalization](https://en.wikipedia.org/wiki/Normalization_(statistics)) before they can be used for training a machine learning model.\n",
+ "\n",
+ "* **Categorical columns:** We need to transform those columns with [one-hot encoding](https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/) to use them during training. \n"
+ ],
+ "metadata": {
+ "id": "8jV9odKhNyF2"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "D9uJtHLSSAMC"
+ },
+ "source": [
+ "Before executing any transformations, we need to check if all the columns can be used for model training. Let's first have a look at the column description as provided by the [JPL website](https://ssd.jpl.nasa.gov/sbdb_query.cgi):\n",
+ "\n",
+ "* **spk_id:** Object primary SPK-ID\n",
+ "* **full_name:** Asteroid name\n",
+ "* **near_earth_object:** Near-earth object flag\n",
+ "* **absolute_magnitude:** the apparent magnitude an object would have if it were located at a distance of 10 parsecs.\n",
+ "* **diameter:** object diameter (from equivalent sphere) km Unit\n",
+ "* **albedo:** a measure of the diffuse reflection of solar radiation out of the total solar radiation and measured on a scale from 0 to 1.\n",
+ "* **diameter_sigma:** 1-sigma uncertainty in object diameter km Unit.\n",
+ "* **eccentricity:** value between 0 and 1 that referes to how flat or round the shape of the asteroid is \n",
+ "* **inclination:** angle with respect to x-y ecliptic plane\n",
+ "* **moid_ld:** Earth Minimum Orbit Intersection Distance au Unit\n",
+ "* **object_class:** the classification of the asteroid. Checkout this [link](https://pdssbn.astro.umd.edu/data_other/objclass.shtml) for a more detailed description.\n",
+ "* **Semi-major axis au Unit:** the length of half of the long axis in AU unit\n",
+ "* **hazardous_flag:** Hazardous Asteroid Flag"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "DzYVKbwTp72d"
+ },
+ "source": [
+ "Columns **'spkid'** and **'full_name'** are unique for each row. These columns can be removed since they are not needed for model training."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "piRPwH2aqT06"
+ },
+ "outputs": [],
+ "source": [
+ "beam_df = beam_df.drop(['spk_id', 'full_name'], axis='columns', inplace=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "fRvNyahSuX_y"
+ },
+ "source": [
+ "Let's have a look at the number of missing values"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 353
+ },
+ "id": "A2PLchW8vXvt",
+ "outputId": "c08d7f23-3a48-4282-a252-66f73cc7fd86"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "/content/beam/sdks/python/apache_beam/dataframe/frame_base.py:145: RuntimeWarning: invalid value encountered in long_scalars\n",
+ " lambda left, right: getattr(left, op)(right), name=op, args=[other])\n"
+ ]
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n",
+ " <div id=\"progress_indicator_02cba1067de8e024374a26297834d233\">\n",
+ " <div class=\"spinner-border text-info\" role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing... collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery == 'undefined') {\n",
+ " var jqueryScript = document.createElement('script');\n",
+ " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript = document.createElement('script');\n",
+ " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery = jQuery.noConflict(true);\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_02cba1067de8e024374a26297834d233\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_02cba1067de8e024374a26297834d233\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "near_earth_object 0.000000\n",
+ "absolute_magnitude 0.000000\n",
+ "diameter 13.111311\n",
+ "albedo 13.271327\n",
+ "diameter_sigma 14.081408\n",
+ "eccentricity 0.000000\n",
+ "inclination 0.000000\n",
+ "moid_ld 0.000000\n",
+ "object_class 0.000000\n",
+ "semi_major_axis_au_unit 0.000000\n",
+ "hazardous_flag 0.000000\n",
+ "dtype: float64"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 7
+ }
+ ],
+ "source": [
+ "ib.collect(beam_df.isnull().mean() * 100)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "00MRdFGLwQiD"
+ },
+ "source": [
+ "Most columns have no missing values. Columns **'diameter'**, **'albedo'** and **'diameter_sigma'** have many missing values. Since these values cannot be measured or derived, we can remove since they will not be required for machine learning model training."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "tHYeCHREwvyB",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 538
+ },
+ "outputId": "5b1b2767-6ae9-4920-f96e-fd1f18e697bb"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n",
+ " <div id=\"progress_indicator_0a93a40e87f23e4a235dfd56cd10188b\">\n",
+ " <div class=\"spinner-border text-info\" role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing... collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery == 'undefined') {\n",
+ " var jqueryScript = document.createElement('script');\n",
+ " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript = document.createElement('script');\n",
+ " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery = jQuery.noConflict(true);\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_0a93a40e87f23e4a235dfd56cd10188b\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_0a93a40e87f23e4a235dfd56cd10188b\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " near_earth_object absolute_magnitude eccentricity inclination \\\n",
+ "0 N 3.40 0.076009 10.594067 \n",
+ "1 N 4.20 0.229972 34.832932 \n",
+ "2 N 5.33 0.256936 12.991043 \n",
+ "3 N 3.00 0.088721 7.141771 \n",
+ "4 N 6.90 0.190913 5.367427 \n",
+ "... ... ... ... ... \n",
+ "9994 N 15.10 0.160610 2.311731 \n",
+ "9995 N 13.60 0.235174 7.657713 \n",
+ "9996 N 14.30 0.113059 2.459643 \n",
+ "9997 N 15.10 0.093852 3.912263 \n",
+ "9998 N 13.00 0.071351 3.198839 \n",
+ "\n",
+ " moid_ld object_class semi_major_axis_au_unit hazardous_flag \n",
+ "0 620.640533 MBA 2.769165 N \n",
+ "1 480.348639 MBA 2.773841 N \n",
+ "2 402.514639 MBA 2.668285 N \n",
+ "3 443.451432 MBA 2.361418 N \n",
+ "4 426.433027 MBA 2.574037 N \n",
+ "... ... ... ... ... \n",
+ "9994 388.723233 MBA 2.390249 N \n",
+ "9995 444.194746 MBA 2.796605 N \n",
+ "9996 495.460110 MBA 2.545674 N \n",
+ "9997 373.848377 MBA 2.160961 N \n",
+ "9998 632.144398 MBA 2.839917 N \n",
+ "\n",
+ "[9999 rows x 8 columns]"
+ ],
+ "text/html": [
+ "\n",
+ " <div id=\"df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398\">\n",
+ " <div class=\"colab-df-container\">\n",
+ " <div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>near_earth_object</th>\n",
+ " <th>absolute_magnitude</th>\n",
+ " <th>eccentricity</th>\n",
+ " <th>inclination</th>\n",
+ " <th>moid_ld</th>\n",
+ " <th>object_class</th>\n",
+ " <th>semi_major_axis_au_unit</th>\n",
+ " <th>hazardous_flag</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>0</th>\n",
+ " <td>N</td>\n",
+ " <td>3.40</td>\n",
+ " <td>0.076009</td>\n",
+ " <td>10.594067</td>\n",
+ " <td>620.640533</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.769165</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>1</th>\n",
+ " <td>N</td>\n",
+ " <td>4.20</td>\n",
+ " <td>0.229972</td>\n",
+ " <td>34.832932</td>\n",
+ " <td>480.348639</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.773841</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>2</th>\n",
+ " <td>N</td>\n",
+ " <td>5.33</td>\n",
+ " <td>0.256936</td>\n",
+ " <td>12.991043</td>\n",
+ " <td>402.514639</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.668285</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>3</th>\n",
+ " <td>N</td>\n",
+ " <td>3.00</td>\n",
+ " <td>0.088721</td>\n",
+ " <td>7.141771</td>\n",
+ " <td>443.451432</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.361418</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>4</th>\n",
+ " <td>N</td>\n",
+ " <td>6.90</td>\n",
+ " <td>0.190913</td>\n",
+ " <td>5.367427</td>\n",
+ " <td>426.433027</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.574037</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>...</th>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9994</th>\n",
+ " <td>N</td>\n",
+ " <td>15.10</td>\n",
+ " <td>0.160610</td>\n",
+ " <td>2.311731</td>\n",
+ " <td>388.723233</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.390249</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9995</th>\n",
+ " <td>N</td>\n",
+ " <td>13.60</td>\n",
+ " <td>0.235174</td>\n",
+ " <td>7.657713</td>\n",
+ " <td>444.194746</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.796605</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9996</th>\n",
+ " <td>N</td>\n",
+ " <td>14.30</td>\n",
+ " <td>0.113059</td>\n",
+ " <td>2.459643</td>\n",
+ " <td>495.460110</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.545674</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9997</th>\n",
+ " <td>N</td>\n",
+ " <td>15.10</td>\n",
+ " <td>0.093852</td>\n",
+ " <td>3.912263</td>\n",
+ " <td>373.848377</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.160961</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9998</th>\n",
+ " <td>N</td>\n",
+ " <td>13.00</td>\n",
+ " <td>0.071351</td>\n",
+ " <td>3.198839</td>\n",
+ " <td>632.144398</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.839917</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "<p>9999 rows × 8 columns</p>\n",
+ "</div>\n",
+ " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398')\"\n",
+ " title=\"Convert this dataframe to an interactive table.\"\n",
+ " style=\"display:none;\">\n",
+ " \n",
+ " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
+ " width=\"24px\">\n",
+ " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+ " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
+ " </svg>\n",
+ " </button>\n",
+ " \n",
+ " <style>\n",
+ " .colab-df-container {\n",
+ " display:flex;\n",
+ " flex-wrap:wrap;\n",
+ " gap: 12px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert {\n",
+ " background-color: #E8F0FE;\n",
+ " border: none;\n",
+ " border-radius: 50%;\n",
+ " cursor: pointer;\n",
+ " display: none;\n",
+ " fill: #1967D2;\n",
+ " height: 32px;\n",
+ " padding: 0 0 0 0;\n",
+ " width: 32px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert:hover {\n",
+ " background-color: #E2EBFA;\n",
+ " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
+ " fill: #174EA6;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert {\n",
+ " background-color: #3B4455;\n",
+ " fill: #D2E3FC;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert:hover {\n",
+ " background-color: #434B5C;\n",
+ " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+ " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+ " fill: #FFFFFF;\n",
+ " }\n",
+ " </style>\n",
+ "\n",
+ " <script>\n",
+ " const buttonEl =\n",
+ " document.querySelector('#df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398 button.colab-df-convert');\n",
+ " buttonEl.style.display =\n",
+ " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
+ "\n",
+ " async function convertToInteractive(key) {\n",
+ " const element = document.querySelector('#df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398');\n",
+ " const dataTable =\n",
+ " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
+ " [key], {});\n",
+ " if (!dataTable) return;\n",
+ "\n",
+ " const docLinkHtml = 'Like what you see? Visit the ' +\n",
+ " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
+ " + ' to learn more about interactive tables.';\n",
+ " element.innerHTML = '';\n",
+ " dataTable['output_type'] = 'display_data';\n",
+ " await google.colab.output.renderOutput(dataTable, element);\n",
+ " const docLink = document.createElement('div');\n",
+ " docLink.innerHTML = docLinkHtml;\n",
+ " element.appendChild(docLink);\n",
+ " }\n",
+ " </script>\n",
+ " </div>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {},
+ "execution_count": 8
+ }
+ ],
+ "source": [
+ "beam_df = beam_df.drop(['diameter', 'albedo', 'diameter_sigma'], axis='columns', inplace=False)\n",
+ "ib.collect(beam_df)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "a3PojL3WBqgE"
+ },
+ "source": [
+ "The numerical columns need to be normalized before using them to train a model. A common method of standarization is to subtract the mean and divide by standard deviation. This ensures that all the data have the same scale and are weighted equally during training. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "sZ2_gB8wENF1"
+ },
+ "source": [
+ "Let's first get both the the numerical columns and categorical columns"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "vsWY8xW5d_Wn"
+ },
+ "outputs": [],
+ "source": [
+ "numerical_cols = beam_df.select_dtypes(include=np.number).columns.tolist()\n",
+ "categorical_cols = list(set(beam_df.columns) - set(numerical_cols))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 356
+ },
+ "id": "Gjc0UlDD-xUn",
+ "outputId": "cadc4402-7edc-43f6-bc6a-bc4f1fd3314a"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "/content/beam/sdks/python/apache_beam/dataframe/frame_base.py:145: RuntimeWarning: invalid value encountered in double_scalars\n",
+ " lambda left, right: getattr(left, op)(right), name=op, args=[other])\n"
+ ]
+ },
+ {
+ "output_type": "error",
+ "ename": "NotImplementedError",
+ "evalue": "ignored",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mNotImplementedError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/tmp/ipykernel_325/987840581.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;31m# Normalizing method_1: Can work but relies on ticket\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mbeam_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mnumerical_cols\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbeam_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumerical_cols\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mbeam_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumerical_cols\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001
b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/dataframe/frame_base.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 426\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 427\u001b[0m raise NotImplementedError(\n\u001b[0;32m--> 428\u001b[0;31m \u001b[0;34mf\"{op!r} is not implemented yet. \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 429\u001b[0m \u001b[0;34mf\"If support for {op!r} is important to you, please let the Beam \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 430\u001b[0m \u001b[0;34m\"community know by writing to user@beam.apache.org \"\u001b[0m\u001b[0;34m\u
001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mNotImplementedError\u001b[0m: 'loc.setitem' is not implemented yet. If support for 'loc.setitem' is important to you, please let the Beam community know by writing to user@beam.apache.org (see https://beam.apache.org/community/contact-us/) or commenting on https://github.com/apache/beam/issues/20318"
+ ]
+ }
+ ],
+ "source": [
+ "# Normalizing method_1: Can work but relies on ticket #22267\n",
+ "beam_df.loc[:,numerical_cols] = (beam_df.loc[:, numerical_cols] - beam_df.loc[:, numerical_cols].mean())/beam_numerical_cols.std()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "v03ABuXJKEmv"
+ },
+ "source": [
+ "Normalizing the data"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 538
+ },
+ "id": "PD_DTxPCP4hs",
+ "outputId": "be40308f-d27c-46fa-e365-43a635addf6b"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n",
+ " <div id=\"progress_indicator_d18c3f74760fd465e55dc784f6b3cf87\">\n",
+ " <div class=\"spinner-border text-info\" role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing... collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery == 'undefined') {\n",
+ " var jqueryScript = document.createElement('script');\n",
+ " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript = document.createElement('script');\n",
+ " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery = jQuery.noConflict(true);\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_d18c3f74760fd465e55dc784f6b3cf87\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_d18c3f74760fd465e55dc784f6b3cf87\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " absolute_magnitude eccentricity inclination moid_ld \\\n",
+ "306 -1.570727 -0.062543 -0.278518 0.373194 \n",
+ "310 -1.631718 -1.724526 -0.736389 1.087833 \n",
+ "546 -1.753698 1.028793 1.415303 -0.339489 \n",
+ "635 -1.875678 0.244869 0.005905 0.214107 \n",
+ "701 -3.278451 -1.570523 2.006145 1.542754 \n",
+ "... ... ... ... ... \n",
+ "9697 0.807888 -1.151809 -0.082944 -0.129556 \n",
+ "9813 1.722740 0.844551 -0.583247 -1.006447 \n",
+ "9868 0.807888 -0.207399 -0.784665 -0.462136 \n",
+ "9903 0.868878 0.460086 0.092258 -0.107597 \n",
+ "9956 0.746898 -0.234132 -0.161116 -0.601379 \n",
+ "\n",
+ " semi_major_axis_au_unit \n",
+ "306 0.357201 \n",
+ "310 0.344233 \n",
+ "546 0.139080 \n",
+ "635 0.367559 \n",
+ "701 0.829337 \n",
+ "... ... \n",
+ "9697 -0.533538 \n",
+ "9813 -0.677961 \n",
+ "9868 -0.539794 \n",
+ "9903 0.071794 \n",
+ "9956 -0.664887 \n",
+ "\n",
+ "[9999 rows x 5 columns]"
+ ],
+ "text/html": [
+ "\n",
+ " <div id=\"df-5bcfe283-1b7d-4af1-af32-05eea5ddacbc\">\n",
+ " <div class=\"colab-df-container\">\n",
+ " <div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>absolute_magnitude</th>\n",
+ " <th>eccentricity</th>\n",
+ " <th>inclination</th>\n",
+ " <th>moid_ld</th>\n",
+ " <th>semi_major_axis_au_unit</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>306</th>\n",
+ " <td>-1.570727</td>\n",
+ " <td>-0.062543</td>\n",
+ " <td>-0.278518</td>\n",
+ " <td>0.373194</td>\n",
+ " <td>0.357201</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>310</th>\n",
+ " <td>-1.631718</td>\n",
+ " <td>-1.724526</td>\n",
+ " <td>-0.736389</td>\n",
+ " <td>1.087833</td>\n",
+ " <td>0.344233</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>546</th>\n",
+ " <td>-1.753698</td>\n",
+ " <td>1.028793</td>\n",
+ " <td>1.415303</td>\n",
+ " <td>-0.339489</td>\n",
+ " <td>0.139080</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>635</th>\n",
+ " <td>-1.875678</td>\n",
+ " <td>0.244869</td>\n",
+ " <td>0.005905</td>\n",
+ " <td>0.214107</td>\n",
+ " <td>0.367559</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>701</th>\n",
+ " <td>-3.278451</td>\n",
+ " <td>-1.570523</td>\n",
+ " <td>2.006145</td>\n",
+ " <td>1.542754</td>\n",
+ " <td>0.829337</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>...</th>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9697</th>\n",
+ " <td>0.807888</td>\n",
+ " <td>-1.151809</td>\n",
+ " <td>-0.082944</td>\n",
+ " <td>-0.129556</td>\n",
+ " <td>-0.533538</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9813</th>\n",
+ " <td>1.722740</td>\n",
+ " <td>0.844551</td>\n",
+ " <td>-0.583247</td>\n",
+ " <td>-1.006447</td>\n",
+ " <td>-0.677961</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9868</th>\n",
+ " <td>0.807888</td>\n",
+ " <td>-0.207399</td>\n",
+ " <td>-0.784665</td>\n",
+ " <td>-0.462136</td>\n",
+ " <td>-0.539794</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9903</th>\n",
+ " <td>0.868878</td>\n",
+ " <td>0.460086</td>\n",
+ " <td>0.092258</td>\n",
+ " <td>-0.107597</td>\n",
+ " <td>0.071794</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9956</th>\n",
+ " <td>0.746898</td>\n",
+ " <td>-0.234132</td>\n",
+ " <td>-0.161116</td>\n",
+ " <td>-0.601379</td>\n",
+ " <td>-0.664887</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "<p>9999 rows × 5 columns</p>\n",
+ "</div>\n",
+ " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5bcfe283-1b7d-4af1-af32-05eea5ddacbc')\"\n",
+ " title=\"Convert this dataframe to an interactive table.\"\n",
+ " style=\"display:none;\">\n",
+ " \n",
+ " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
+ " width=\"24px\">\n",
+ " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+ " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
+ " </svg>\n",
+ " </button>\n",
+ " \n",
+ " <style>\n",
+ " .colab-df-container {\n",
+ " display:flex;\n",
+ " flex-wrap:wrap;\n",
+ " gap: 12px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert {\n",
+ " background-color: #E8F0FE;\n",
+ " border: none;\n",
+ " border-radius: 50%;\n",
+ " cursor: pointer;\n",
+ " display: none;\n",
+ " fill: #1967D2;\n",
+ " height: 32px;\n",
+ " padding: 0 0 0 0;\n",
+ " width: 32px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert:hover {\n",
+ " background-color: #E2EBFA;\n",
+ " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
+ " fill: #174EA6;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert {\n",
+ " background-color: #3B4455;\n",
+ " fill: #D2E3FC;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert:hover {\n",
+ " background-color: #434B5C;\n",
+ " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+ " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+ " fill: #FFFFFF;\n",
+ " }\n",
+ " </style>\n",
+ "\n",
+ " <script>\n",
+ " const buttonEl =\n",
+ " document.querySelector('#df-5bcfe283-1b7d-4af1-af32-05eea5ddacbc button.colab-df-convert');\n",
+ " buttonEl.style.display =\n",
+ " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
+ "\n",
+ " async function convertToInteractive(key) {\n",
+ " const element = document.querySelector('#df-5bcfe283-1b7d-4af1-af32-05eea5ddacbc');\n",
+ " const dataTable =\n",
+ " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
+ " [key], {});\n",
+ " if (!dataTable) return;\n",
+ "\n",
+ " const docLinkHtml = 'Like what you see? Visit the ' +\n",
+ " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
+ " + ' to learn more about interactive tables.';\n",
+ " element.innerHTML = '';\n",
+ " dataTable['output_type'] = 'display_data';\n",
+ " await google.colab.output.renderOutput(dataTable, element);\n",
+ " const docLink = document.createElement('div');\n",
+ " docLink.innerHTML = docLinkHtml;\n",
+ " element.appendChild(docLink);\n",
+ " }\n",
+ " </script>\n",
+ " </div>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {},
+ "execution_count": 11
+ }
+ ],
+ "source": [
+ "# Get numerical columns\n",
+ "beam_numerical_cols = beam_df.filter(items=numerical_cols)\n",
+ "\n",
+ "# Standarize dataframes only with numerical columns\n",
+ "beam_numerical_cols = (beam_numerical_cols - beam_numerical_cols.mean())/beam_numerical_cols.std()\n",
+ "\n",
+ "ib.collect(beam_numerical_cols)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "qdNILsajFvex"
+ },
+ "source": [
+ "Next, we need to convert the categorical columns into one-hot encoding variables to use them during training. \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 321
+ },
+ "id": "Ngoxg0rSywVd",
+ "outputId": "d81bb29a-f8f8-4186-b186-5ff85667dbec"
+ },
+ "outputs": [
+ {
+ "output_type": "error",
+ "ename": "AttributeError",
+ "evalue": "ignored",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/tmp/ipykernel_325/1671644751.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mobject_class_col\u001b[0m\u001b[0;34m=\u001b[0m \u001b[0mbeam_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfilter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'object_class'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mobject_class_col\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_dummies\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/dataframe/frames.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m 2482\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2483\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2484\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getattribute__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2485\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2486\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[
0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mAttributeError\u001b[0m: 'DeferredDataFrame' object has no attribute 'get_dummies'"
+ ]
+ }
+ ],
+ "source": [
+ "object_class_col= beam_df.filter(items=['object_class'])\n",
+ "object_class_col.get_dummies()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 321
+ },
+ "id": "hz8s7z8caTq-",
+ "outputId": "5c543a6b-0ea1-41f8-afec-9691bbbd1f5b"
+ },
+ "outputs": [
+ {
+ "output_type": "error",
+ "ename": "AttributeError",
+ "evalue": "ignored",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/tmp/ipykernel_325/1927971370.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;31m# df['categories_concat'].str.get_dummies('-')\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mobject_class_col\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_dummies\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/dataframe/frames.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m 2482\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2483\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2484\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getattribute__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2485\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2486\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[
0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mAttributeError\u001b[0m: 'DeferredDataFrame' object has no attribute 'str'"
+ ]
+ }
+ ],
+ "source": [
+ "object_class_col.str.get_dummies()"
Review Comment:
`str` is an attribute on Series, you need to access just a Series (e.g. `object_class_col = beam_df.object_class`) in order to use `str.get_dummies`. Selecting a single column with `filter` produces a single column DataFrame.
You will also need to configure that column to have a `CategoricalDtype`. This is where you can use `unique` and `ib.collect`, something like:
```
unique_classes = ib.collect(object_class_col.unique())
object_class_col.astype(pd.CategoricalDtype(unique_classes)).str.get_dummies()
```
##########
examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb:
##########
@@ -0,0 +1,1907 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Overview\n",
+ "\n",
+ "One of the most common tools used for data exploration and pre-processing is [pandas DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html). Pandas has become very popular for its ease of use. It has very intuitive methods to perform common analytical tasks and data pre-processing. \n",
+ "\n",
+ "Pandas loads all of the data into memory on a single machine (one node) for rapid execution. This works well when dealing with small-scale datasets. However, many projects involve datasets that can grow too big to fit in memory. These use cases generally require the usage of parallel data processing frameworks such as Apache Beam.\n",
+ "\n",
+ "\n",
+ "## Beam DataFrames\n",
+ "\n",
+ "\n",
+ "Beam DataFrames provide a pandas-like DataFrame\n",
+ "API to declare and define Beam processing pipelines. It provides a familiar interface for machine learning practioners to build complex data-processing pipelines by only invoking standard pandas commands.\n",
+ "\n",
+ "> ℹ️ To learn more about Beam DataFrames, take a look at the\n",
+ "[Beam DataFrames overview](https://beam.apache.org/documentation/dsls/dataframes/overview) page.\n",
+ "\n",
+ "## Tutorial outline\n",
+ "\n",
+ "In this notebook, we walk through the use of the Beam DataFrames API to perform common data exploration as well as pre-processing steps that are necessary to prepare your dataset for machine learning model training and inference, such as: \n",
+ "\n",
+ "* Removing unwanted columns.\n",
+ "* One-hot encoding categorical columns.\n",
+ "* Normalizing numerical columns.\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "iFZC1inKuUCy"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Installation\n",
+ "\n",
+ "First, we need to install Apache Beam with the `interactive` component to be able to use the Interactive runner. The latest implemented DataFrames API methods invoked in this notebook are available in Beam <b>2.41</b> or later.\n"
+ ],
+ "metadata": {
+ "id": "A0f2HJ22D4lt"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pCjwrwNWnuqI"
+ },
+ "source": [
+ "**Option 1:** Install latest version with implemented df.mean()\n",
+ "\n",
+ "TODO: Remove this text later"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "-OJC0Xn5Um-C"
+ },
+ "outputs": [],
+ "source": [
+ "!git clone https://github.com/apache/beam.git\n",
+ "\n",
+ "!cd beam/sdks/python && pip3 install -r build-requirements.txt \n",
+ "\n",
+ "%pip install -e beam/sdks/python/.[interactive,gcp]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xfXzNzA1n3ZP"
+ },
+ "source": [
+ "**Option 2:** Install latest release version \n",
+ "\n",
+ "**[12/07/2022]:** df.mean() is currently not supported for this version (beam 2.40)\n",
+ "\n",
+ "TODO: Remove this text later"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "4xY7ECJZOuJj"
+ },
+ "outputs": [],
+ "source": [
+ "! pip install apache-beam[interactive,gcp]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Part I : Local exploration with the Interactive Beam runner\n",
+ "We first use the [Interactive Beam](https://beam.apache.org/releases/pydoc/2.20.0/apache_beam.runners.interactive.interactive_beam.html) to explore and develop our pipeline.\n",
+ "This allows us to quickly test our pipeline locally before running it on a distributed runner. \n",
+ "\n",
+ "\n",
+ "> ℹ️ In this section, we will only be working with a subset of the original dataset since we're only using the the compute resources of the notebook instance.\n"
+ ],
+ "metadata": {
+ "id": "3NO6RgB7GkkE"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5I3G094hoB1P"
+ },
+ "source": [
+ "# Loading the data\n",
+ "\n",
+ "Pandas has the\n",
+ "[`pandas.read_csv`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)\n",
+ "function to easily read CSV files into DataFrames.\n",
+ "We're using the beam\n",
+ "[`beam.dataframe.io.read_csv`](https://beam.apache.org/releases/pydoc/current/apache_beam.dataframe.io.html#apache_beam.dataframe.io.read_csv)\n",
+ "function that emulates `pandas.read_csv`. The main difference between them is that the beam method returns a deferred Beam DataFrame while pandas return a standard DataFrame.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "X3_OB9cAULav"
+ },
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "\n",
+ "import numpy as np\n",
+ "import pandas as pd \n",
+ "import apache_beam as beam\n",
+ "import apache_beam.runners.interactive.interactive_beam as ib\n",
+ "from apache_beam.runners.interactive.interactive_runner import InteractiveRunner\n",
+ "from apache_beam.runners.dataflow import DataflowRunner\n",
+ "\n",
+ "# Available options: [sample_1000, sample_10000, sample_100000, sample] where\n",
+ "# sample contains all of the dataset (around 1000000 samples)\n",
+ "file_location = 'gs://apache-beam-samples/nasa_jpl_asteroid/sample_10000.csv'\n",
+ "\n",
+ "# Initialize pipline\n",
+ "p = beam.Pipeline(InteractiveRunner())\n",
+ "\n",
+ "# Create a deferred Beam DataFrame with the contents of our csv file.\n",
+ "beam_df = p | beam.dataframe.io.read_csv(file_location, splittable=True)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "paf7yf3YpCh8"
+ },
+ "source": [
+ "# Data pre-processing\n",
+ "\n",
+ "## Dataset description \n",
+ "\n",
+ "### [NASA - Nearest Earth Objects dataset](https://cneos.jpl.nasa.gov/ca/)\n",
+ "There are an innumerable number of objects in the outer space. Some of them are closer than we think. Even though we might think that a distance of 70,000 Km can not potentially harm us, but at an astronomical scale, this is a very small distance and can disrupt many natural phenomena. \n",
+ "\n",
+ "These objects/asteroids can thus prove to be harmful. Hence, it is wise to know what is surrounding us and what can harm us amongst those. Thus, this dataset compiles the list of NASA certified asteroids that are classified as the nearest earth object."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "\n",
+ "Let's first inspect the columns of our dataset and their types"
+ ],
+ "metadata": {
+ "id": "cvAu5T0ENjuQ"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "LwW77ixE-pjR",
+ "outputId": "c24ff83d-3a13-47a6-c9c2-3978729fde82"
+ },
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "spk_id int64\n",
+ "full_name object\n",
+ "near_earth_object object\n",
+ "absolute_magnitude float64\n",
+ "diameter float64\n",
+ "albedo float64\n",
+ "diameter_sigma float64\n",
+ "eccentricity float64\n",
+ "inclination float64\n",
+ "moid_ld float64\n",
+ "object_class object\n",
+ "semi_major_axis_au_unit float64\n",
+ "hazardous_flag object\n",
+ "dtype: object"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 4
+ }
+ ],
+ "source": [
+ "beam_df.dtypes"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "When using Interactive Beam, we can use `ib.collect()` to bring a Beam DataFrame into local memory as a Pandas DataFrame."
+ ],
+ "metadata": {
+ "id": "1Wa6fpbyQige"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 746
+ },
+ "id": "DPxkAmkpq4Xv",
+ "outputId": "14fa80de-2dee-4963-99d8-3e321f949ff8"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n",
+ " <div id=\"progress_indicator_a986c6cc61ed5a5b622e163d92f73775\">\n",
+ " <div class=\"spinner-border text-info\" role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing... collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery == 'undefined') {\n",
+ " var jqueryScript = document.createElement('script');\n",
+ " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript = document.createElement('script');\n",
+ " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery = jQuery.noConflict(true);\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_a986c6cc61ed5a5b622e163d92f73775\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_a986c6cc61ed5a5b622e163d92f73775\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " spk_id full_name near_earth_object \\\n",
+ "0 2000001 1 Ceres N \n",
+ "1 2000002 2 Pallas N \n",
+ "2 2000003 3 Juno N \n",
+ "3 2000004 4 Vesta N \n",
+ "4 2000005 5 Astraea N \n",
+ "... ... ... ... \n",
+ "9994 2009995 9995 Alouette (4805 P-L) N \n",
+ "9995 2009996 9996 ANS (9070 P-L) N \n",
+ "9996 2009997 9997 COBE (1217 T-1) N \n",
+ "9997 2009998 9998 ISO (1293 T-1) N \n",
+ "9998 2009999 9999 Wiles (4196 T-2) N \n",
+ "\n",
+ " absolute_magnitude diameter albedo diameter_sigma eccentricity \\\n",
+ "0 3.40 939.400 0.0900 0.200 0.076009 \n",
+ "1 4.20 545.000 0.1010 18.000 0.229972 \n",
+ "2 5.33 246.596 0.2140 10.594 0.256936 \n",
+ "3 3.00 525.400 0.4228 0.200 0.088721 \n",
+ "4 6.90 106.699 0.2740 3.140 0.190913 \n",
+ "... ... ... ... ... ... \n",
+ "9994 15.10 2.564 0.2450 0.550 0.160610 \n",
+ "9995 13.60 8.978 0.1130 0.376 0.235174 \n",
+ "9996 14.30 NaN NaN NaN 0.113059 \n",
+ "9997 15.10 2.235 0.3880 0.373 0.093852 \n",
+ "9998 13.00 7.148 0.2620 0.065 0.071351 \n",
+ "\n",
+ " inclination moid_ld object_class semi_major_axis_au_unit \\\n",
+ "0 10.594067 620.640533 MBA 2.769165 \n",
+ "1 34.832932 480.348639 MBA 2.773841 \n",
+ "2 12.991043 402.514639 MBA 2.668285 \n",
+ "3 7.141771 443.451432 MBA 2.361418 \n",
+ "4 5.367427 426.433027 MBA 2.574037 \n",
+ "... ... ... ... ... \n",
+ "9994 2.311731 388.723233 MBA 2.390249 \n",
+ "9995 7.657713 444.194746 MBA 2.796605 \n",
+ "9996 2.459643 495.460110 MBA 2.545674 \n",
+ "9997 3.912263 373.848377 MBA 2.160961 \n",
+ "9998 3.198839 632.144398 MBA 2.839917 \n",
+ "\n",
+ " hazardous_flag \n",
+ "0 N \n",
+ "1 N \n",
+ "2 N \n",
+ "3 N \n",
+ "4 N \n",
+ "... ... \n",
+ "9994 N \n",
+ "9995 N \n",
+ "9996 N \n",
+ "9997 N \n",
+ "9998 N \n",
+ "\n",
+ "[9999 rows x 13 columns]"
+ ],
+ "text/html": [
+ "\n",
+ " <div id=\"df-0161aa69-d50f-4d6f-84c1-10dacb278880\">\n",
+ " <div class=\"colab-df-container\">\n",
+ " <div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>spk_id</th>\n",
+ " <th>full_name</th>\n",
+ " <th>near_earth_object</th>\n",
+ " <th>absolute_magnitude</th>\n",
+ " <th>diameter</th>\n",
+ " <th>albedo</th>\n",
+ " <th>diameter_sigma</th>\n",
+ " <th>eccentricity</th>\n",
+ " <th>inclination</th>\n",
+ " <th>moid_ld</th>\n",
+ " <th>object_class</th>\n",
+ " <th>semi_major_axis_au_unit</th>\n",
+ " <th>hazardous_flag</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>0</th>\n",
+ " <td>2000001</td>\n",
+ " <td>1 Ceres</td>\n",
+ " <td>N</td>\n",
+ " <td>3.40</td>\n",
+ " <td>939.400</td>\n",
+ " <td>0.0900</td>\n",
+ " <td>0.200</td>\n",
+ " <td>0.076009</td>\n",
+ " <td>10.594067</td>\n",
+ " <td>620.640533</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.769165</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>1</th>\n",
+ " <td>2000002</td>\n",
+ " <td>2 Pallas</td>\n",
+ " <td>N</td>\n",
+ " <td>4.20</td>\n",
+ " <td>545.000</td>\n",
+ " <td>0.1010</td>\n",
+ " <td>18.000</td>\n",
+ " <td>0.229972</td>\n",
+ " <td>34.832932</td>\n",
+ " <td>480.348639</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.773841</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>2</th>\n",
+ " <td>2000003</td>\n",
+ " <td>3 Juno</td>\n",
+ " <td>N</td>\n",
+ " <td>5.33</td>\n",
+ " <td>246.596</td>\n",
+ " <td>0.2140</td>\n",
+ " <td>10.594</td>\n",
+ " <td>0.256936</td>\n",
+ " <td>12.991043</td>\n",
+ " <td>402.514639</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.668285</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>3</th>\n",
+ " <td>2000004</td>\n",
+ " <td>4 Vesta</td>\n",
+ " <td>N</td>\n",
+ " <td>3.00</td>\n",
+ " <td>525.400</td>\n",
+ " <td>0.4228</td>\n",
+ " <td>0.200</td>\n",
+ " <td>0.088721</td>\n",
+ " <td>7.141771</td>\n",
+ " <td>443.451432</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.361418</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>4</th>\n",
+ " <td>2000005</td>\n",
+ " <td>5 Astraea</td>\n",
+ " <td>N</td>\n",
+ " <td>6.90</td>\n",
+ " <td>106.699</td>\n",
+ " <td>0.2740</td>\n",
+ " <td>3.140</td>\n",
+ " <td>0.190913</td>\n",
+ " <td>5.367427</td>\n",
+ " <td>426.433027</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.574037</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>...</th>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9994</th>\n",
+ " <td>2009995</td>\n",
+ " <td>9995 Alouette (4805 P-L)</td>\n",
+ " <td>N</td>\n",
+ " <td>15.10</td>\n",
+ " <td>2.564</td>\n",
+ " <td>0.2450</td>\n",
+ " <td>0.550</td>\n",
+ " <td>0.160610</td>\n",
+ " <td>2.311731</td>\n",
+ " <td>388.723233</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.390249</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9995</th>\n",
+ " <td>2009996</td>\n",
+ " <td>9996 ANS (9070 P-L)</td>\n",
+ " <td>N</td>\n",
+ " <td>13.60</td>\n",
+ " <td>8.978</td>\n",
+ " <td>0.1130</td>\n",
+ " <td>0.376</td>\n",
+ " <td>0.235174</td>\n",
+ " <td>7.657713</td>\n",
+ " <td>444.194746</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.796605</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9996</th>\n",
+ " <td>2009997</td>\n",
+ " <td>9997 COBE (1217 T-1)</td>\n",
+ " <td>N</td>\n",
+ " <td>14.30</td>\n",
+ " <td>NaN</td>\n",
+ " <td>NaN</td>\n",
+ " <td>NaN</td>\n",
+ " <td>0.113059</td>\n",
+ " <td>2.459643</td>\n",
+ " <td>495.460110</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.545674</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9997</th>\n",
+ " <td>2009998</td>\n",
+ " <td>9998 ISO (1293 T-1)</td>\n",
+ " <td>N</td>\n",
+ " <td>15.10</td>\n",
+ " <td>2.235</td>\n",
+ " <td>0.3880</td>\n",
+ " <td>0.373</td>\n",
+ " <td>0.093852</td>\n",
+ " <td>3.912263</td>\n",
+ " <td>373.848377</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.160961</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9998</th>\n",
+ " <td>2009999</td>\n",
+ " <td>9999 Wiles (4196 T-2)</td>\n",
+ " <td>N</td>\n",
+ " <td>13.00</td>\n",
+ " <td>7.148</td>\n",
+ " <td>0.2620</td>\n",
+ " <td>0.065</td>\n",
+ " <td>0.071351</td>\n",
+ " <td>3.198839</td>\n",
+ " <td>632.144398</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.839917</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "<p>9999 rows × 13 columns</p>\n",
+ "</div>\n",
+ " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0161aa69-d50f-4d6f-84c1-10dacb278880')\"\n",
+ " title=\"Convert this dataframe to an interactive table.\"\n",
+ " style=\"display:none;\">\n",
+ " \n",
+ " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
+ " width=\"24px\">\n",
+ " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+ " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
+ " </svg>\n",
+ " </button>\n",
+ " \n",
+ " <style>\n",
+ " .colab-df-container {\n",
+ " display:flex;\n",
+ " flex-wrap:wrap;\n",
+ " gap: 12px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert {\n",
+ " background-color: #E8F0FE;\n",
+ " border: none;\n",
+ " border-radius: 50%;\n",
+ " cursor: pointer;\n",
+ " display: none;\n",
+ " fill: #1967D2;\n",
+ " height: 32px;\n",
+ " padding: 0 0 0 0;\n",
+ " width: 32px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert:hover {\n",
+ " background-color: #E2EBFA;\n",
+ " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
+ " fill: #174EA6;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert {\n",
+ " background-color: #3B4455;\n",
+ " fill: #D2E3FC;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert:hover {\n",
+ " background-color: #434B5C;\n",
+ " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+ " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+ " fill: #FFFFFF;\n",
+ " }\n",
+ " </style>\n",
+ "\n",
+ " <script>\n",
+ " const buttonEl =\n",
+ " document.querySelector('#df-0161aa69-d50f-4d6f-84c1-10dacb278880 button.colab-df-convert');\n",
+ " buttonEl.style.display =\n",
+ " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
+ "\n",
+ " async function convertToInteractive(key) {\n",
+ " const element = document.querySelector('#df-0161aa69-d50f-4d6f-84c1-10dacb278880');\n",
+ " const dataTable =\n",
+ " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
+ " [key], {});\n",
+ " if (!dataTable) return;\n",
+ "\n",
+ " const docLinkHtml = 'Like what you see? Visit the ' +\n",
+ " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
+ " + ' to learn more about interactive tables.';\n",
+ " element.innerHTML = '';\n",
+ " dataTable['output_type'] = 'display_data';\n",
+ " await google.colab.output.renderOutput(dataTable, element);\n",
+ " const docLink = document.createElement('div');\n",
+ " docLink.innerHTML = docLinkHtml;\n",
+ " element.appendChild(docLink);\n",
+ " }\n",
+ " </script>\n",
+ " </div>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {},
+ "execution_count": 5
+ }
+ ],
+ "source": [
+ "ib.collect(beam_df)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can see that our datasets consists of both:\n",
+ "\n",
+ "* **Numerical columns:** These columns need to be transformed through [normalization](https://en.wikipedia.org/wiki/Normalization_(statistics)) before they can be used for training a machine learning model.\n",
+ "\n",
+ "* **Categorical columns:** We need to transform those columns with [one-hot encoding](https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/) to use them during training. \n"
+ ],
+ "metadata": {
+ "id": "8jV9odKhNyF2"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "D9uJtHLSSAMC"
+ },
+ "source": [
+ "Before executing any transformations, we need to check if all the columns can be used for model training. Let's first have a look at the column description as provided by the [JPL website](https://ssd.jpl.nasa.gov/sbdb_query.cgi):\n",
+ "\n",
+ "* **spk_id:** Object primary SPK-ID\n",
+ "* **full_name:** Asteroid name\n",
+ "* **near_earth_object:** Near-earth object flag\n",
+ "* **absolute_magnitude:** the apparent magnitude an object would have if it were located at a distance of 10 parsecs.\n",
+ "* **diameter:** object diameter (from equivalent sphere) km Unit\n",
+ "* **albedo:** a measure of the diffuse reflection of solar radiation out of the total solar radiation and measured on a scale from 0 to 1.\n",
+ "* **diameter_sigma:** 1-sigma uncertainty in object diameter km Unit.\n",
+ "* **eccentricity:** value between 0 and 1 that referes to how flat or round the shape of the asteroid is \n",
+ "* **inclination:** angle with respect to x-y ecliptic plane\n",
+ "* **moid_ld:** Earth Minimum Orbit Intersection Distance au Unit\n",
+ "* **object_class:** the classification of the asteroid. Checkout this [link](https://pdssbn.astro.umd.edu/data_other/objclass.shtml) for a more detailed description.\n",
+ "* **Semi-major axis au Unit:** the length of half of the long axis in AU unit\n",
+ "* **hazardous_flag:** Hazardous Asteroid Flag"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "DzYVKbwTp72d"
+ },
+ "source": [
+ "Columns **'spkid'** and **'full_name'** are unique for each row. These columns can be removed since they are not needed for model training."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "piRPwH2aqT06"
+ },
+ "outputs": [],
+ "source": [
+ "beam_df = beam_df.drop(['spk_id', 'full_name'], axis='columns', inplace=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "fRvNyahSuX_y"
+ },
+ "source": [
+ "Let's have a look at the number of missing values"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 353
+ },
+ "id": "A2PLchW8vXvt",
+ "outputId": "c08d7f23-3a48-4282-a252-66f73cc7fd86"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "/content/beam/sdks/python/apache_beam/dataframe/frame_base.py:145: RuntimeWarning: invalid value encountered in long_scalars\n",
+ " lambda left, right: getattr(left, op)(right), name=op, args=[other])\n"
+ ]
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n",
+ " <div id=\"progress_indicator_02cba1067de8e024374a26297834d233\">\n",
+ " <div class=\"spinner-border text-info\" role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing... collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery == 'undefined') {\n",
+ " var jqueryScript = document.createElement('script');\n",
+ " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript = document.createElement('script');\n",
+ " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery = jQuery.noConflict(true);\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_02cba1067de8e024374a26297834d233\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_02cba1067de8e024374a26297834d233\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "near_earth_object 0.000000\n",
+ "absolute_magnitude 0.000000\n",
+ "diameter 13.111311\n",
+ "albedo 13.271327\n",
+ "diameter_sigma 14.081408\n",
+ "eccentricity 0.000000\n",
+ "inclination 0.000000\n",
+ "moid_ld 0.000000\n",
+ "object_class 0.000000\n",
+ "semi_major_axis_au_unit 0.000000\n",
+ "hazardous_flag 0.000000\n",
+ "dtype: float64"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 7
+ }
+ ],
+ "source": [
+ "ib.collect(beam_df.isnull().mean() * 100)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "00MRdFGLwQiD"
+ },
+ "source": [
+ "Most columns have no missing values. Columns **'diameter'**, **'albedo'** and **'diameter_sigma'** have many missing values. Since these values cannot be measured or derived, we can remove since they will not be required for machine learning model training."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "tHYeCHREwvyB",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 538
+ },
+ "outputId": "5b1b2767-6ae9-4920-f96e-fd1f18e697bb"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n",
+ " <div id=\"progress_indicator_0a93a40e87f23e4a235dfd56cd10188b\">\n",
+ " <div class=\"spinner-border text-info\" role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing... collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery == 'undefined') {\n",
+ " var jqueryScript = document.createElement('script');\n",
+ " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript = document.createElement('script');\n",
+ " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery = jQuery.noConflict(true);\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_0a93a40e87f23e4a235dfd56cd10188b\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_0a93a40e87f23e4a235dfd56cd10188b\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " near_earth_object absolute_magnitude eccentricity inclination \\\n",
+ "0 N 3.40 0.076009 10.594067 \n",
+ "1 N 4.20 0.229972 34.832932 \n",
+ "2 N 5.33 0.256936 12.991043 \n",
+ "3 N 3.00 0.088721 7.141771 \n",
+ "4 N 6.90 0.190913 5.367427 \n",
+ "... ... ... ... ... \n",
+ "9994 N 15.10 0.160610 2.311731 \n",
+ "9995 N 13.60 0.235174 7.657713 \n",
+ "9996 N 14.30 0.113059 2.459643 \n",
+ "9997 N 15.10 0.093852 3.912263 \n",
+ "9998 N 13.00 0.071351 3.198839 \n",
+ "\n",
+ " moid_ld object_class semi_major_axis_au_unit hazardous_flag \n",
+ "0 620.640533 MBA 2.769165 N \n",
+ "1 480.348639 MBA 2.773841 N \n",
+ "2 402.514639 MBA 2.668285 N \n",
+ "3 443.451432 MBA 2.361418 N \n",
+ "4 426.433027 MBA 2.574037 N \n",
+ "... ... ... ... ... \n",
+ "9994 388.723233 MBA 2.390249 N \n",
+ "9995 444.194746 MBA 2.796605 N \n",
+ "9996 495.460110 MBA 2.545674 N \n",
+ "9997 373.848377 MBA 2.160961 N \n",
+ "9998 632.144398 MBA 2.839917 N \n",
+ "\n",
+ "[9999 rows x 8 columns]"
+ ],
+ "text/html": [
+ "\n",
+ " <div id=\"df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398\">\n",
+ " <div class=\"colab-df-container\">\n",
+ " <div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>near_earth_object</th>\n",
+ " <th>absolute_magnitude</th>\n",
+ " <th>eccentricity</th>\n",
+ " <th>inclination</th>\n",
+ " <th>moid_ld</th>\n",
+ " <th>object_class</th>\n",
+ " <th>semi_major_axis_au_unit</th>\n",
+ " <th>hazardous_flag</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>0</th>\n",
+ " <td>N</td>\n",
+ " <td>3.40</td>\n",
+ " <td>0.076009</td>\n",
+ " <td>10.594067</td>\n",
+ " <td>620.640533</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.769165</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>1</th>\n",
+ " <td>N</td>\n",
+ " <td>4.20</td>\n",
+ " <td>0.229972</td>\n",
+ " <td>34.832932</td>\n",
+ " <td>480.348639</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.773841</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>2</th>\n",
+ " <td>N</td>\n",
+ " <td>5.33</td>\n",
+ " <td>0.256936</td>\n",
+ " <td>12.991043</td>\n",
+ " <td>402.514639</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.668285</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>3</th>\n",
+ " <td>N</td>\n",
+ " <td>3.00</td>\n",
+ " <td>0.088721</td>\n",
+ " <td>7.141771</td>\n",
+ " <td>443.451432</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.361418</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>4</th>\n",
+ " <td>N</td>\n",
+ " <td>6.90</td>\n",
+ " <td>0.190913</td>\n",
+ " <td>5.367427</td>\n",
+ " <td>426.433027</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.574037</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>...</th>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9994</th>\n",
+ " <td>N</td>\n",
+ " <td>15.10</td>\n",
+ " <td>0.160610</td>\n",
+ " <td>2.311731</td>\n",
+ " <td>388.723233</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.390249</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9995</th>\n",
+ " <td>N</td>\n",
+ " <td>13.60</td>\n",
+ " <td>0.235174</td>\n",
+ " <td>7.657713</td>\n",
+ " <td>444.194746</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.796605</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9996</th>\n",
+ " <td>N</td>\n",
+ " <td>14.30</td>\n",
+ " <td>0.113059</td>\n",
+ " <td>2.459643</td>\n",
+ " <td>495.460110</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.545674</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9997</th>\n",
+ " <td>N</td>\n",
+ " <td>15.10</td>\n",
+ " <td>0.093852</td>\n",
+ " <td>3.912263</td>\n",
+ " <td>373.848377</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.160961</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9998</th>\n",
+ " <td>N</td>\n",
+ " <td>13.00</td>\n",
+ " <td>0.071351</td>\n",
+ " <td>3.198839</td>\n",
+ " <td>632.144398</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.839917</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "<p>9999 rows × 8 columns</p>\n",
+ "</div>\n",
+ " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398')\"\n",
+ " title=\"Convert this dataframe to an interactive table.\"\n",
+ " style=\"display:none;\">\n",
+ " \n",
+ " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
+ " width=\"24px\">\n",
+ " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+ " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
+ " </svg>\n",
+ " </button>\n",
+ " \n",
+ " <style>\n",
+ " .colab-df-container {\n",
+ " display:flex;\n",
+ " flex-wrap:wrap;\n",
+ " gap: 12px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert {\n",
+ " background-color: #E8F0FE;\n",
+ " border: none;\n",
+ " border-radius: 50%;\n",
+ " cursor: pointer;\n",
+ " display: none;\n",
+ " fill: #1967D2;\n",
+ " height: 32px;\n",
+ " padding: 0 0 0 0;\n",
+ " width: 32px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert:hover {\n",
+ " background-color: #E2EBFA;\n",
+ " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
+ " fill: #174EA6;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert {\n",
+ " background-color: #3B4455;\n",
+ " fill: #D2E3FC;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert:hover {\n",
+ " background-color: #434B5C;\n",
+ " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+ " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+ " fill: #FFFFFF;\n",
+ " }\n",
+ " </style>\n",
+ "\n",
+ " <script>\n",
+ " const buttonEl =\n",
+ " document.querySelector('#df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398 button.colab-df-convert');\n",
+ " buttonEl.style.display =\n",
+ " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
+ "\n",
+ " async function convertToInteractive(key) {\n",
+ " const element = document.querySelector('#df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398');\n",
+ " const dataTable =\n",
+ " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
+ " [key], {});\n",
+ " if (!dataTable) return;\n",
+ "\n",
+ " const docLinkHtml = 'Like what you see? Visit the ' +\n",
+ " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
+ " + ' to learn more about interactive tables.';\n",
+ " element.innerHTML = '';\n",
+ " dataTable['output_type'] = 'display_data';\n",
+ " await google.colab.output.renderOutput(dataTable, element);\n",
+ " const docLink = document.createElement('div');\n",
+ " docLink.innerHTML = docLinkHtml;\n",
+ " element.appendChild(docLink);\n",
+ " }\n",
+ " </script>\n",
+ " </div>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {},
+ "execution_count": 8
+ }
+ ],
+ "source": [
+ "beam_df = beam_df.drop(['diameter', 'albedo', 'diameter_sigma'], axis='columns', inplace=False)\n",
+ "ib.collect(beam_df)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "a3PojL3WBqgE"
+ },
+ "source": [
+ "The numerical columns need to be normalized before using them to train a model. A common method of standarization is to subtract the mean and divide by standard deviation. This ensures that all the data have the same scale and are weighted equally during training. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "sZ2_gB8wENF1"
+ },
+ "source": [
+ "Let's first get both the the numerical columns and categorical columns"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "vsWY8xW5d_Wn"
+ },
+ "outputs": [],
+ "source": [
+ "numerical_cols = beam_df.select_dtypes(include=np.number).columns.tolist()\n",
+ "categorical_cols = list(set(beam_df.columns) - set(numerical_cols))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 356
+ },
+ "id": "Gjc0UlDD-xUn",
+ "outputId": "cadc4402-7edc-43f6-bc6a-bc4f1fd3314a"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "/content/beam/sdks/python/apache_beam/dataframe/frame_base.py:145: RuntimeWarning: invalid value encountered in double_scalars\n",
+ " lambda left, right: getattr(left, op)(right), name=op, args=[other])\n"
+ ]
+ },
+ {
+ "output_type": "error",
+ "ename": "NotImplementedError",
+ "evalue": "ignored",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mNotImplementedError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/tmp/ipykernel_325/987840581.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;31m# Normalizing method_1: Can work but relies on ticket\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mbeam_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mnumerical_cols\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbeam_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumerical_cols\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mbeam_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumerical_cols\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001
b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/dataframe/frame_base.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 426\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 427\u001b[0m raise NotImplementedError(\n\u001b[0;32m--> 428\u001b[0;31m \u001b[0;34mf\"{op!r} is not implemented yet. \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 429\u001b[0m \u001b[0;34mf\"If support for {op!r} is important to you, please let the Beam \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 430\u001b[0m \u001b[0;34m\"community know by writing to user@beam.apache.org \"\u001b[0m\u001b[0;34m\u
001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mNotImplementedError\u001b[0m: 'loc.setitem' is not implemented yet. If support for 'loc.setitem' is important to you, please let the Beam community know by writing to user@beam.apache.org (see https://beam.apache.org/community/contact-us/) or commenting on https://github.com/apache/beam/issues/20318"
+ ]
+ }
+ ],
+ "source": [
+ "# Normalizing method_1: Can work but relies on ticket #22267\n",
+ "beam_df.loc[:,numerical_cols] = (beam_df.loc[:, numerical_cols] - beam_df.loc[:, numerical_cols].mean())/beam_numerical_cols.std()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "v03ABuXJKEmv"
+ },
+ "source": [
+ "Normalizing the data"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 538
+ },
+ "id": "PD_DTxPCP4hs",
+ "outputId": "be40308f-d27c-46fa-e365-43a635addf6b"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n",
+ " <div id=\"progress_indicator_d18c3f74760fd465e55dc784f6b3cf87\">\n",
+ " <div class=\"spinner-border text-info\" role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing... collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery == 'undefined') {\n",
+ " var jqueryScript = document.createElement('script');\n",
+ " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript = document.createElement('script');\n",
+ " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery = jQuery.noConflict(true);\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_d18c3f74760fd465e55dc784f6b3cf87\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_d18c3f74760fd465e55dc784f6b3cf87\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " absolute_magnitude eccentricity inclination moid_ld \\\n",
+ "306 -1.570727 -0.062543 -0.278518 0.373194 \n",
+ "310 -1.631718 -1.724526 -0.736389 1.087833 \n",
+ "546 -1.753698 1.028793 1.415303 -0.339489 \n",
+ "635 -1.875678 0.244869 0.005905 0.214107 \n",
+ "701 -3.278451 -1.570523 2.006145 1.542754 \n",
+ "... ... ... ... ... \n",
+ "9697 0.807888 -1.151809 -0.082944 -0.129556 \n",
+ "9813 1.722740 0.844551 -0.583247 -1.006447 \n",
+ "9868 0.807888 -0.207399 -0.784665 -0.462136 \n",
+ "9903 0.868878 0.460086 0.092258 -0.107597 \n",
+ "9956 0.746898 -0.234132 -0.161116 -0.601379 \n",
+ "\n",
+ " semi_major_axis_au_unit \n",
+ "306 0.357201 \n",
+ "310 0.344233 \n",
+ "546 0.139080 \n",
+ "635 0.367559 \n",
+ "701 0.829337 \n",
+ "... ... \n",
+ "9697 -0.533538 \n",
+ "9813 -0.677961 \n",
+ "9868 -0.539794 \n",
+ "9903 0.071794 \n",
+ "9956 -0.664887 \n",
+ "\n",
+ "[9999 rows x 5 columns]"
+ ],
+ "text/html": [
+ "\n",
+ " <div id=\"df-5bcfe283-1b7d-4af1-af32-05eea5ddacbc\">\n",
+ " <div class=\"colab-df-container\">\n",
+ " <div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>absolute_magnitude</th>\n",
+ " <th>eccentricity</th>\n",
+ " <th>inclination</th>\n",
+ " <th>moid_ld</th>\n",
+ " <th>semi_major_axis_au_unit</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>306</th>\n",
+ " <td>-1.570727</td>\n",
+ " <td>-0.062543</td>\n",
+ " <td>-0.278518</td>\n",
+ " <td>0.373194</td>\n",
+ " <td>0.357201</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>310</th>\n",
+ " <td>-1.631718</td>\n",
+ " <td>-1.724526</td>\n",
+ " <td>-0.736389</td>\n",
+ " <td>1.087833</td>\n",
+ " <td>0.344233</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>546</th>\n",
+ " <td>-1.753698</td>\n",
+ " <td>1.028793</td>\n",
+ " <td>1.415303</td>\n",
+ " <td>-0.339489</td>\n",
+ " <td>0.139080</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>635</th>\n",
+ " <td>-1.875678</td>\n",
+ " <td>0.244869</td>\n",
+ " <td>0.005905</td>\n",
+ " <td>0.214107</td>\n",
+ " <td>0.367559</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>701</th>\n",
+ " <td>-3.278451</td>\n",
+ " <td>-1.570523</td>\n",
+ " <td>2.006145</td>\n",
+ " <td>1.542754</td>\n",
+ " <td>0.829337</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>...</th>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9697</th>\n",
+ " <td>0.807888</td>\n",
+ " <td>-1.151809</td>\n",
+ " <td>-0.082944</td>\n",
+ " <td>-0.129556</td>\n",
+ " <td>-0.533538</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9813</th>\n",
+ " <td>1.722740</td>\n",
+ " <td>0.844551</td>\n",
+ " <td>-0.583247</td>\n",
+ " <td>-1.006447</td>\n",
+ " <td>-0.677961</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9868</th>\n",
+ " <td>0.807888</td>\n",
+ " <td>-0.207399</td>\n",
+ " <td>-0.784665</td>\n",
+ " <td>-0.462136</td>\n",
+ " <td>-0.539794</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9903</th>\n",
+ " <td>0.868878</td>\n",
+ " <td>0.460086</td>\n",
+ " <td>0.092258</td>\n",
+ " <td>-0.107597</td>\n",
+ " <td>0.071794</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9956</th>\n",
+ " <td>0.746898</td>\n",
+ " <td>-0.234132</td>\n",
+ " <td>-0.161116</td>\n",
+ " <td>-0.601379</td>\n",
+ " <td>-0.664887</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "<p>9999 rows × 5 columns</p>\n",
+ "</div>\n",
+ " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5bcfe283-1b7d-4af1-af32-05eea5ddacbc')\"\n",
+ " title=\"Convert this dataframe to an interactive table.\"\n",
+ " style=\"display:none;\">\n",
+ " \n",
+ " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
+ " width=\"24px\">\n",
+ " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+ " <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
+ " </svg>\n",
+ " </button>\n",
+ " \n",
+ " <style>\n",
+ " .colab-df-container {\n",
+ " display:flex;\n",
+ " flex-wrap:wrap;\n",
+ " gap: 12px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert {\n",
+ " background-color: #E8F0FE;\n",
+ " border: none;\n",
+ " border-radius: 50%;\n",
+ " cursor: pointer;\n",
+ " display: none;\n",
+ " fill: #1967D2;\n",
+ " height: 32px;\n",
+ " padding: 0 0 0 0;\n",
+ " width: 32px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert:hover {\n",
+ " background-color: #E2EBFA;\n",
+ " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
+ " fill: #174EA6;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert {\n",
+ " background-color: #3B4455;\n",
+ " fill: #D2E3FC;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert:hover {\n",
+ " background-color: #434B5C;\n",
+ " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+ " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+ " fill: #FFFFFF;\n",
+ " }\n",
+ " </style>\n",
+ "\n",
+ " <script>\n",
+ " const buttonEl =\n",
+ " document.querySelector('#df-5bcfe283-1b7d-4af1-af32-05eea5ddacbc button.colab-df-convert');\n",
+ " buttonEl.style.display =\n",
+ " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
+ "\n",
+ " async function convertToInteractive(key) {\n",
+ " const element = document.querySelector('#df-5bcfe283-1b7d-4af1-af32-05eea5ddacbc');\n",
+ " const dataTable =\n",
+ " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
+ " [key], {});\n",
+ " if (!dataTable) return;\n",
+ "\n",
+ " const docLinkHtml = 'Like what you see? Visit the ' +\n",
+ " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
+ " + ' to learn more about interactive tables.';\n",
+ " element.innerHTML = '';\n",
+ " dataTable['output_type'] = 'display_data';\n",
+ " await google.colab.output.renderOutput(dataTable, element);\n",
+ " const docLink = document.createElement('div');\n",
+ " docLink.innerHTML = docLinkHtml;\n",
+ " element.appendChild(docLink);\n",
+ " }\n",
+ " </script>\n",
+ " </div>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {},
+ "execution_count": 11
+ }
+ ],
+ "source": [
+ "# Get numerical columns\n",
+ "beam_numerical_cols = beam_df.filter(items=numerical_cols)\n",
+ "\n",
+ "# Standarize dataframes only with numerical columns\n",
+ "beam_numerical_cols = (beam_numerical_cols - beam_numerical_cols.mean())/beam_numerical_cols.std()\n",
+ "\n",
+ "ib.collect(beam_numerical_cols)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "qdNILsajFvex"
+ },
+ "source": [
+ "Next, we need to convert the categorical columns into one-hot encoding variables to use them during training. \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 321
+ },
+ "id": "Ngoxg0rSywVd",
+ "outputId": "d81bb29a-f8f8-4186-b186-5ff85667dbec"
+ },
+ "outputs": [
+ {
+ "output_type": "error",
+ "ename": "AttributeError",
+ "evalue": "ignored",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/tmp/ipykernel_325/1671644751.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mobject_class_col\u001b[0m\u001b[0;34m=\u001b[0m \u001b[0mbeam_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfilter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'object_class'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mobject_class_col\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_dummies\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/dataframe/frames.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m 2482\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2483\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2484\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getattribute__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2485\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2486\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[
0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mAttributeError\u001b[0m: 'DeferredDataFrame' object has no attribute 'get_dummies'"
+ ]
+ }
+ ],
+ "source": [
+ "object_class_col= beam_df.filter(items=['object_class'])\n",
+ "object_class_col.get_dummies()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 321
+ },
+ "id": "hz8s7z8caTq-",
+ "outputId": "5c543a6b-0ea1-41f8-afec-9691bbbd1f5b"
+ },
+ "outputs": [
+ {
+ "output_type": "error",
+ "ename": "AttributeError",
+ "evalue": "ignored",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/tmp/ipykernel_325/1927971370.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;31m# df['categories_concat'].str.get_dummies('-')\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mobject_class_col\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_dummies\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/dataframe/frames.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m 2482\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2483\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2484\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getattribute__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2485\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2486\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[
0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mAttributeError\u001b[0m: 'DeferredDataFrame' object has no attribute 'str'"
+ ]
+ }
+ ],
+ "source": [
+ "object_class_col.str.get_dummies()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "rVdSIyCB0spw"
+ },
+ "source": [
+ "# Putting it all together\n",
+ "\n",
+ "Let's now try to summarize all the steps that we've executed above into a full pipeline implementation and visualize our pre-processed data.\n",
+ "\n",
+ "> ℹ️ Note that the only standard Beam method invoked here is the `pipeline` instance. The rest of the pre-processing commands are all based on native pandas methods that have been integrated with the Beam DataFrame API. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ndaSNond0v8Q",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 498
+ },
+ "outputId": "0155d359-45c9-4345-e1b6-b1881408f049"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "/content/beam/sdks/python/apache_beam/dataframe/frame_base.py:145: RuntimeWarning: invalid value encountered in double_scalars\n",
+ " lambda left, right: getattr(left, op)(right), name=op, args=[other])\n"
+ ]
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n",
+ " <div id=\"progress_indicator_fc5349f6a626d7566f941a2f2a1fccfe\">\n",
+ " <div class=\"spinner-border text-info\" role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing... collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery == 'undefined') {\n",
+ " var jqueryScript = document.createElement('script');\n",
+ " jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript = document.createElement('script');\n",
+ " datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery = jQuery.noConflict(true);\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_fc5349f6a626d7566f941a2f2a1fccfe\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ " window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ " $(\"#progress_indicator_fc5349f6a626d7566f941a2f2a1fccfe\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "error",
+ "ename": "ValueError",
+ "evalue": "ignored",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
+ "\u001b[0;32m/tmp/ipykernel_325/1408061827.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 25\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 26\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 27\u001b[0;31m \u001b[0mib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcollect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpreprocessed_dataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/runners/interactive/utils.py\u001b[0m in \u001b[0;36mrun_within_progress_indicator\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 275\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mrun_within_progress_indicator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 276\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mProgressIndicator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf'Processing... {func.__name__}'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Done.'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 277\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001
b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 278\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 279\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mrun_within_progress_indicator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/runners/interactive/interactive_beam.py\u001b[0m in \u001b[0;36mcollect\u001b[0;34m(pcoll, n, duration, include_window_info)\u001b[0m\n\u001b[1;32m 945\u001b[0m element_type=element_type)\n\u001b[1;32m 946\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 947\u001b[0;31m \u001b[0mrecording\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrecording_manager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecord\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpcoll\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_n\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_duration\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mduration\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 948\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 949\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0
;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/runners/interactive/recording_manager.py\u001b[0m in \u001b[0;36mrecord\u001b[0;34m(self, pcolls, max_n, max_duration)\u001b[0m\n\u001b[1;32m 459\u001b[0m pf.PipelineFragment(\n\u001b[1;32m 460\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muncomputed_pcolls\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 461\u001b[0;31m self.user_pipeline.options).run(blocking=is_remote_run)\n\u001b[0m\u001b[1;32m 462\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mie\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcurrent_env\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpipeline_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muser_pipeline\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n
\u001b[1;32m 463\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/runners/interactive/pipeline_fragment.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, display_pipeline_graph, use_cache, blocking)\u001b[0m\n\u001b[1;32m 111\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_runner_pipeline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrunner\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_force_compute\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0muse_cache\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 112\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_runner_pipeline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrunner\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_blocking\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mblocking\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 113\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdeduce_fragment\u001
b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 114\u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 115\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_runner_pipeline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrunner\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_skip_display\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpreserved_skip_display\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/runners/interactive/pipeline_fragment.py\u001b[0m in \u001b[0;36mdeduce_fragment\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 98\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_runner_pipeline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_runner_api\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 99\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_runner_pipeline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrunner\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 100\u001b[0;31m self._options)\n\u001b[0m\u001b[1;32m 101\u001b[0m \u001b[0mie\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcurrent_env\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_derived_pipeline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[
0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_runner_pipeline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfragment\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 102\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfragment\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/content/beam/sdks/python/apache_beam/pipeline.py\u001b[0m in \u001b[0;36mfrom_runner_api\u001b[0;34m(proto, runner, options, return_context)\u001b[0m\n\u001b[1;32m 990\u001b[0m \u001b[0mpcollection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpipeline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 991\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mpcollection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mproducer\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 992\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'No producer for %s'\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mid\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 993\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 994\u001b[0m
\u001b[0;31m# Inject PBegin input where necessary.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mValueError\u001b[0m: No producer for ref_PCollection_PCollection_265"
+ ]
+ }
+ ],
+ "source": [
+ "# Initialize pipline\n",
+ "p = beam.Pipeline(InteractiveRunner())\n",
+ "\n",
+ "# Create a deferred Beam DataFrame with the contents of our csv file.\n",
+ "beam_df = p | beam.dataframe.io.read_csv('/content/drive/MyDrive/apache beam/dataset/nasa/sample_10000.csv', splittable=True)\n",
+ "\n",
+ "# Drop irrelavant columns/columns with missing values\n",
+ "beam_df = beam_df.drop(['spk_id', 'full_name','diameter', 'albedo', 'diameter_sigma'], axis='columns', inplace=False)\n",
+ "\n",
+ "# Get numerical columns/columns with categorical variables\n",
+ "numerical_cols = beam_df.select_dtypes(include=np.number).columns.tolist()\n",
+ "categorical_cols = list(set(beam_df.columns) - set(numerical_cols))\n",
+ "\n",
+ "# Normalize the numerical variables \n",
+ "beam_df_numerical = beam_df.filter(items=numerical_cols)\n",
+ "beam_df_numerical = (beam_df_numerical - beam_df_numerical.mean())/beam_df_numerical.std()\n",
+ "\n",
+ "\n",
+ "# One-hot encode the categorical variables \n",
+ "beam_df_categorical = beam_df.filter(items=categorical_cols)\n",
+ "# ToDo: one hot-encoding step\n",
+ "\n",
+ "# Merge the normalized variables with the one-hot encoded variables\n",
+ "preprocessed_dataset = beam_df_categorical.merge(beam_df_numerical, left_index = True, right_index = True)\n",
+ "\n",
+ "ib.collect(preprocessed_dataset)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xZvJTqa3XKI_"
+ },
+ "source": [
+ "# Part II : Process the full dataset with the Distributed Runner\n",
+ "Now that we've showcased how to build and execute the pipeline locally using the Interactive Runner. It's time to execute our pipeline on our full dataset by switching to a distributed runner. For this example, we will exectue our pipeline on [Dataflow](https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "PROJECT_ID = \"<my-gcp-project>\"\n",
+ "REGION = \"us-west1\"\n",
+ "TEMP_DIR = \"gs://<my-bucket>/tmp\"\n",
+ "OUTPUT_DIR = \"gs://<my-bucket>/dataframe-result\""
+ ],
+ "metadata": {
+ "id": "dDBYbMEWbL4t"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "> ℹ️ Note that we are now processing the full dataset `sample.csv` that containts approximately 1 million rows. We're also writing the results to a `csv` file instead of using `ib.collect()` to materialize the deferred dataframe.\n",
Review Comment:
Should `sample.csv` be renamed `full.csv`? The name sample makes me think it's still a subset of the full dataset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org