You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tvalentyn (via GitHub)" <gi...@apache.org> on 2023/03/09 01:11:01 UTC
[GitHub] [beam] tvalentyn commented on a diff in pull request #25719: Add one example to learn beam by doing
tvalentyn commented on code in PR #25719:
URL: https://github.com/apache/beam/pull/25719#discussion_r1130218407
##########
examples/notebooks/get-started/learn_beam_basics_by_doing.ipynb:
##########
@@ -0,0 +1,1070 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "V913rQcmLS72"
+ },
+ "source": [
+ "# Welcome to Apache Beam!\n",
+ "\n",
+ "This notebook will be your introductory guide to Beam's main concepts and its uses. This tutorial **does not** assume any prior Apache Beam knowledge.\n",
+ "\n",
+ "We'll cover what Beam is, what it does, and a few basic transforms!\n",
+ "\n",
+ "We aim to give you familiarity with:\n",
+ "- Creating a `Pipeline`\n",
+ "- Creating a `PCollection`\n",
+ "- Performing basic `PTransforms`\n",
+ " - Map\n",
+ " - Filter\n",
+ " - FlatMap\n",
+ " - Combine\n",
+ "- Applications\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Lduh-9oXt3P_"
+ },
+ "source": [
+ "## How To Approach This Tutorial\n",
+ "\n",
+ "This tutorial was designed for someone who likes to **learn by doing**. As such, there will be opportunities for you to practice writing your own code in these notebooks with the answer hidden in a cell below.\n",
+ "\n",
+ "Codes that require editing will be with an `...` and each cell title will say `Edit This Code`. However, you are free to play around with the other cells if you would like to add something beyond our tutorial.\n",
+ "\n",
+ "It may be tempting to just copy and paste solutions, but even if you do look at the Answer cells, try typing out the solutions manually. The muscle memory will be very helpful.\n",
+ "\n",
+ "> Tip: For those who would like to learn concepts more from the ground up, check out these [notebooks](https://beam.apache.org/get-started/tour-of-beam/)!"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "42B-64Lvef3K"
+ },
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "We'll assume you have familiarity with Python or Pandas, but you should be able to follow along even if you’re coming from a different programming language. We'll also assume you understand programming concepts like functions, objects, arrays, and dictionaries.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SeDD0nardXyL"
+ },
+ "source": [
+ "## Running CoLab\n",
+ "\n",
+ "To navigate through different sections, use the table of contents. From View drop-down list, select Table of contents.\n",
+ "\n",
+ "To run a code cell, you can click the Run cell button at the top left of the cell, or by select it and press `Shift+Enter`. Try modifying a code cell and re-running it to see what happens."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "UdMPnMDDkGc8"
+ },
+ "source": [
+ "To begin, we have to set up our environment. Let's install and import Apache Beam by running the cell below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "5cPHukaOgDDM"
+ },
+ "outputs": [],
+ "source": [
+ "# Remember: You can press shift+enter to run this cell\n",
+ "!pip install --quiet apache-beam\n",
+ "import apache_beam as beam"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "30l8_MD-undP"
+ },
+ "outputs": [],
+ "source": [
+ "# Set the logging level to reduce verbose information\n",
+ "import logging\n",
+ "\n",
+ "logging.root.setLevel(logging.ERROR)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gyoo1gLKtZmU"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "---\n",
+ "\n",
+ "\n",
+ "\n",
+ "---\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "N29KJTfdMCtB"
+ },
+ "source": [
+ "# What is Apache Beam?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "B2evhzEuMu8a"
+ },
+ "source": [
+ "Apache Beam is a library for data processing. It is often used for [Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load) jobs, where we:\n",
+ "1. *Extract* from a data source\n",
+ "2. *Transform* that data\n",
+ "3. *Load* that data into a data sink (like a database)\n",
+ "\n",
+ "Apache Beam makes these jobs easy with the ability to process everything at the same time and its unified model and open-source SDKs. There are many more parts of Beam, but throughout these tutorials, we will break down each part to show you how they will all fit together.\n",
+ "\n",
+ "For this tutorial, you will use these Beam SDKs to build your own `Pipeline` to process your data.\n",
+ "\n",
+ "Below, we will run through creating the heart of the `Pipeline`. There are three main abstractions in Beam:\n",
+ "1. `Pipeline`\n",
+ "2. `PCollection`\n",
+ "3. `PTransform`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WecDJbfqpWb1"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "---\n",
+ "\n",
+ "\n",
+ "---\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FditMJAIAp9Q"
+ },
+ "source": [
+ "# 1. Design Your Pipeline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_xYj6e3Hvt1W"
+ },
+ "source": [
+ "## 1.1 What is a Pipeline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "bVJCh9FLBtU9"
+ },
+ "source": [
+ "A `Pipeline` describes the whole cycle of your data processing task, starting from the data sources you'll be processing (`PCollection`) to the processing transforms (`PTransforms`) you will apply to them until your desired output.\n",
Review Comment:
there is a conflation of concepts 'source' and 'pcollection' here. See: https://beam.apache.org/documentation/glossary/#source
##########
examples/notebooks/get-started/learn_beam_basics_by_doing.ipynb:
##########
@@ -0,0 +1,1070 @@
+{
+ "cells": [
Review Comment:
Let's add `Open in Colab` link line in https://github.com/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-py.ipynb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org