You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/06/07 23:52:13 UTC
[GitHub] [beam] davidcavazos opened a new pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
davidcavazos opened a new pull request #14962:
URL: https://github.com/apache/beam/pull/14962
Adds the Windowing notebook for the Tour of Beam.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] anguillanneuf commented on pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
anguillanneuf commented on pull request #14962:
URL: https://github.com/apache/beam/pull/14962#issuecomment-863579457
Overall looks great. Gardening project, moon phases, the notebook has got personality and I like it a lot :)
Some comments as I was going through the [notebook](https://colab.research.google.com/github/davidcavazos/beam/blob/tour-of-beam/examples/notebooks/tour-of-beam/windowing.ipynb):
1. `s/lets/let's`
2. Suggested edit - redundancy: "In our example, the "processing" is done by PrintElementInfo which simply prints the element with its window information. ~For windows of three months every month~, each element is processed three times, one time per window."
3. Suggested edit - use a bulleted list: "Sliding windows allow us to do just that. We need to specify the window size in seconds just like with FixedWindows. We also need to specify a window period in seconds, which is how often we want to emit each window."
4. Suggested edit - redundancy - future tense: "If the next event happens within the next 30 days ~or less, like 20 days after the previous event,~ the session window [will extend and cover..] extends and covers that as well. If there are no new events for the next 30 days, the session window [will close..] closes and is emitted."
5. I had to scroll up to remember what input events look like in the later examples. It would be nice to include a screenshot of the input events to the right of the nice bar charts, then these screenshots can be easily used by other tutorials/talks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #14962:
URL: https://github.com/apache/beam/pull/14962#issuecomment-870792499
R: @pcoet -- for documentation
FYI: @rosetn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #14962:
URL: https://github.com/apache/beam/pull/14962#discussion_r660824581
##########
File path: examples/notebooks/tour-of-beam/windowing.ipynb
##########
@@ -0,0 +1,703 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Windowing -- Tour of Beam",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "code",
+ "metadata": {
+ "cellView": "form",
+ "id": "upmJn_DjcThx"
+ },
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE"
+ },
+ "source": [
+ "# Windowing -- _Tour of Beam_\n",
+ "\n",
+ "Sometimes, we want to [aggregate](https://beam.apache.org/documentation/transforms/python/overview/#aggregation) data, like `GroupByKey` or `Combine`, only at certain intervals, like hourly or daily, instead of processing the entire `PCollection` of data only once.\n",
+ "\n",
+ "We might want to emit a [moving average](https://en.wikipedia.org/wiki/Moving_average) as we're processing data.\n",
+ "\n",
+ "Maybe we want to analyze the user experience for a certain task in a web app, it would be nice to get the app events by sessions of activity.\n",
+ "\n",
+ "Or we could be running a streaming pipeline, and there is no end to the data, so how can we aggregate data?\n",
+ "\n",
+ "_Windows_ in Beam allow us to process only certain data intervals at a time.\n",
+ "In this notebook, we go through different ways of windowing our pipeline.\n",
+ "\n",
+ "Lets begin by installing `apache-beam`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg"
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_OkWHiAvpWDZ"
+ },
+ "source": [
+ "First, lets define some helper functions to simplify the rest of the examples.\n",
+ "\n",
+ "We have a transform to help us analyze an element alongside its window information, and we have another transform to help us analyze how many elements landed into each window.\n",
+ "We use a custom [`DoFn`](https://beam.apache.org/documentation/transforms/python/elementwise/pardo)\n",
+ "to access that information.\n",
+ "\n",
+ "You don't need to understand these, you just need to know they exist đ."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "C9yAN1Hgk3dF"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def human_readable_window(window) -> str:\n",
+ " \"\"\"Formats a window object into a human readable string.\"\"\"\n",
+ " if isinstance(window, beam.window.GlobalWindow):\n",
+ " return str(window)\n",
+ " return f'{window.start.to_utc_datetime()} - {window.end.to_utc_datetime()}'\n",
+ "\n",
+ "class PrintElementInfo(beam.DoFn):\n",
+ " \"\"\"Prints an element with its Window information.\"\"\"\n",
+ " def process(self, element, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):\n",
+ " print(f'[{human_readable_window(window)}] {timestamp.to_utc_datetime()} -- {element}')\n",
+ " yield element\n",
+ "\n",
+ "@beam.ptransform_fn\n",
Review comment:
Thanks, I removed PTransforms and only use DoFn since we need to access the windowing information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #14962:
URL: https://github.com/apache/beam/pull/14962#issuecomment-870217609
What is the next step on this PR? Could you address the open comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos edited a comment on pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos edited a comment on pull request #14962:
URL: https://github.com/apache/beam/pull/14962#issuecomment-870792499
@aaltay sorry for the delay on this one, but comments have been addressed and should be ready for review.
R: @pcoet
FYI: @rosetn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos edited a comment on pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos edited a comment on pull request #14962:
URL: https://github.com/apache/beam/pull/14962#issuecomment-870789854
> Overall looks great. Gardening project, moon phases, the notebook has got personality and I like it a lot :)
>
> Some comments as I was going through the [notebook](https://colab.research.google.com/github/davidcavazos/beam/blob/tour-of-beam/examples/notebooks/tour-of-beam/windowing.ipynb):
>
> 1. `s/lets/let's`
Thanks, done.
> 2. Suggested edit - redundancy: "In our example, the "processing" is done by PrintElementInfo which simply prints the element with its window information. ~For windows of three months every month~, each element is processed three times, one time per window."
Changed
> 3. Suggested edit - use a bulleted list: "Sliding windows allow us to do just that. We need to specify the window size in seconds just like with FixedWindows. We also need to specify a window period in seconds, which is how often we want to emit each window."
Changed
> 4. Suggested edit - redundancy - future tense: "If the next event happens within the next 30 days ~or less, like 20 days after the previous event,~ the session window [will extend and cover..] extends and covers that as well. If there are no new events for the next 30 days, the session window [will close..] closes and is emitted."
Changed
> 5. I had to scroll up to remember what input events look like in the later examples. It would be nice to include a screenshot of the input events to the right of the nice bar charts, then these screenshots can be easily included by other tutorials/talks.
I actually ended up inlining the data on each snippet. It makes each sample a little more self contained, and simplified some indirection. It also makes it easier for people to modify the inputs and see how it affects things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] anguillanneuf edited a comment on pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
anguillanneuf edited a comment on pull request #14962:
URL: https://github.com/apache/beam/pull/14962#issuecomment-863579457
Overall looks great. Gardening project, moon phases, the notebook has got personality and I like it a lot :)
Some comments as I was going through the [notebook](https://colab.research.google.com/github/davidcavazos/beam/blob/tour-of-beam/examples/notebooks/tour-of-beam/windowing.ipynb):
1. `s/lets/let's`
2. Suggested edit - redundancy: "In our example, the "processing" is done by PrintElementInfo which simply prints the element with its window information. ~For windows of three months every month~, each element is processed three times, one time per window."
3. Suggested edit - use a bulleted list: "Sliding windows allow us to do just that. We need to specify the window size in seconds just like with FixedWindows. We also need to specify a window period in seconds, which is how often we want to emit each window."
4. Suggested edit - redundancy - future tense: "If the next event happens within the next 30 days ~or less, like 20 days after the previous event,~ the session window [will extend and cover..] extends and covers that as well. If there are no new events for the next 30 days, the session window [will close..] closes and is emitted."
5. I had to scroll up to remember what input events look like in the later examples. It would be nice to include a screenshot of the input events to the right of the nice bar charts, then these screenshots can be easily included by other tutorials/talks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #14962:
URL: https://github.com/apache/beam/pull/14962#issuecomment-870217609
What is the next step on this PR? Could you address the open comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay merged pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
aaltay merged pull request #14962:
URL: https://github.com/apache/beam/pull/14962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on a change in pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on a change in pull request #14962:
URL: https://github.com/apache/beam/pull/14962#discussion_r660825491
##########
File path: examples/notebooks/tour-of-beam/windowing.ipynb
##########
@@ -0,0 +1,703 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Windowing -- Tour of Beam",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "code",
+ "metadata": {
+ "cellView": "form",
+ "id": "upmJn_DjcThx"
+ },
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE"
+ },
+ "source": [
+ "# Windowing -- _Tour of Beam_\n",
+ "\n",
+ "Sometimes, we want to [aggregate](https://beam.apache.org/documentation/transforms/python/overview/#aggregation) data, like `GroupByKey` or `Combine`, only at certain intervals, like hourly or daily, instead of processing the entire `PCollection` of data only once.\n",
+ "\n",
+ "We might want to emit a [moving average](https://en.wikipedia.org/wiki/Moving_average) as we're processing data.\n",
+ "\n",
+ "Maybe we want to analyze the user experience for a certain task in a web app, it would be nice to get the app events by sessions of activity.\n",
+ "\n",
+ "Or we could be running a streaming pipeline, and there is no end to the data, so how can we aggregate data?\n",
+ "\n",
+ "_Windows_ in Beam allow us to process only certain data intervals at a time.\n",
+ "In this notebook, we go through different ways of windowing our pipeline.\n",
+ "\n",
+ "Lets begin by installing `apache-beam`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg"
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_OkWHiAvpWDZ"
+ },
+ "source": [
+ "First, lets define some helper functions to simplify the rest of the examples.\n",
+ "\n",
+ "We have a transform to help us analyze an element alongside its window information, and we have another transform to help us analyze how many elements landed into each window.\n",
+ "We use a custom [`DoFn`](https://beam.apache.org/documentation/transforms/python/elementwise/pardo)\n",
+ "to access that information.\n",
+ "\n",
+ "You don't need to understand these, you just need to know they exist đ."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "C9yAN1Hgk3dF"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def human_readable_window(window) -> str:\n",
+ " \"\"\"Formats a window object into a human readable string.\"\"\"\n",
+ " if isinstance(window, beam.window.GlobalWindow):\n",
+ " return str(window)\n",
+ " return f'{window.start.to_utc_datetime()} - {window.end.to_utc_datetime()}'\n",
+ "\n",
+ "class PrintElementInfo(beam.DoFn):\n",
+ " \"\"\"Prints an element with its Window information.\"\"\"\n",
+ " def process(self, element, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):\n",
+ " print(f'[{human_readable_window(window)}] {timestamp.to_utc_datetime()} -- {element}')\n",
+ " yield element\n",
+ "\n",
+ "@beam.ptransform_fn\n",
+ "def PrintWindowInfo(pcollection):\n",
+ " \"\"\"Prints the Window information with how many elements landed in that window.\"\"\"\n",
+ " class PrintCountsInfo(beam.DoFn):\n",
+ " def process(self, num_elements, window=beam.DoFn.WindowParam):\n",
+ " print(f'>> Window [{human_readable_window(window)}] has {num_elements} elements')\n",
+ " yield num_elements\n",
+ "\n",
+ " return (\n",
+ " pcollection\n",
+ " | 'Count elements per window' >> beam.combiners.Count.Globally().without_defaults()\n",
+ " | 'Print counts info' >> beam.ParDo(PrintCountsInfo())\n",
+ " )"
+ ],
+ "execution_count": 1,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "CQrojV2QnqIU"
+ },
+ "source": [
+ "Now lets create some data to use in the examples.\n",
+ "\n",
+ "Windows define data intervals based on time, so we need to tell Apache Beam a timestamp for each element.\n",
+ "\n",
+ "We define a `PTransform` for convenience, so we can attach the timestamps automatically.\n",
+ "\n",
+ "Apache Beam requires us to provide the timestamp as [Unix time](https://en.wikipedia.org/wiki/Unix_time), which is a way to represent a date and time as the number of seconds since January 1st, 1970.\n",
+ "\n",
+ "For our data, lets analyze some events about the seasons and moon phases for the year 2021, which might be [useful for a gardening project](https://www.almanac.com/content/planting-by-the-moon).\n",
+ "\n",
+ "To attach timestamps to each element, we can `Map` each element and return a [`TimestmpedValue`](https://beam.apache.org/documentation/transforms/python/elementwise/withtimestamps/)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "Sgzscopvmh1f",
+ "outputId": "e0c6fc19-ab97-4754-8f1f-1601807be940"
+ },
+ "source": [
+ "import time\n",
+ "from apache_beam.options.pipeline_options import PipelineOptions\n",
+ "\n",
+ "def to_unix_time(time_str: str, time_format='%Y-%m-%d %H:%M:%S') -> int:\n",
+ " \"\"\"Converts a time string into Unix time.\"\"\"\n",
+ " time_tuple = time.strptime(time_str, time_format)\n",
+ " return int(time.mktime(time_tuple))\n",
+ "\n",
+ "@beam.ptransform_fn\n",
+ "@beam.typehints.with_input_types(beam.pvalue.PBegin)\n",
+ "@beam.typehints.with_output_types(beam.window.TimestampedValue)\n",
+ "def AstronomicalEvents(pipeline):\n",
+ " return (\n",
+ " pipeline\n",
+ " | 'Create data' >> beam.Create([\n",
+ " ('2021-03-20 03:37:00', 'March Equinox 2021'),\n",
+ " ('2021-04-26 22:31:00', 'Super full moon'),\n",
+ " ('2021-05-11 13:59:00', 'Micro new moon'),\n",
+ " ('2021-05-26 06:13:00', 'Super full moon, total lunar eclipse'),\n",
+ " ('2021-06-20 22:32:00', 'June Solstice 2021'),\n",
+ " ('2021-08-22 07:01:00', 'Blue moon'),\n",
+ " ('2021-09-22 14:21:00', 'September Equinox 2021'),\n",
+ " ('2021-11-04 15:14:00', 'Super new moon'),\n",
+ " ('2021-11-19 02:57:00', 'Micro full moon, partial lunar eclipse'),\n",
+ " ('2021-12-04 01:43:00', 'Super new moon'),\n",
+ " ('2021-12-18 10:35:00', 'Micro full moon'),\n",
+ " ('2021-12-21 09:59:00', 'December Solstice 2021'),\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.MapTuple(\n",
+ " lambda timestamp, element:\n",
+ " beam.window.TimestampedValue(element, to_unix_time(timestamp))\n",
+ " )\n",
+ " )\n",
+ "\n",
+ "# Lets see how the data looks like.\n",
+ "beam_options = PipelineOptions(flags=[], type_check_additional='all')\n",
+ "with beam.Pipeline(options=beam_options) as pipeline:\n",
+ " (\n",
+ " pipeline\n",
+ " | 'Astronomical events' >> AstronomicalEvents()\n",
+ " | 'Print element' >> beam.Map(print)\n",
+ " )"
+ ],
+ "execution_count": 3,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "March Equinox 2021\n",
+ "Super full moon\n",
+ "Micro new moon\n",
+ "Super full moon, total lunar eclipse\n",
+ "June Solstice 2021\n",
+ "Blue moon\n",
+ "September Equinox 2021\n",
+ "December Solstice 2021\n",
+ "Super new moon\n",
+ "Micro full moon, partial lunar eclipse\n",
+ "Super new moon\n",
+ "Micro full moon\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "qI0K3jSA2LbJ"
+ },
+ "source": [
+ "> âšī¸ After running this, it looks like the timestamps disappeared!\n",
+ "> They're actually still _implicitly_ part of the element, just like the windowing information.\n",
+ "> If we need to access it, we can do so via a custom [`DoFn`](https://beam.apache.org/documentation/transforms/python/elementwise/pardo).\n",
+ "> Aggregation transforms use each element's timestamp along with the windowing we specified to create windows of elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ymHF1WCqnG4V"
+ },
+ "source": [
+ "# Global window\n",
+ "\n",
+ "All pipelines use the [`GlobalWindow`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html#apache_beam.transforms.window.GlobalWindow) by default.\n",
+ "This is a single window that covers the entire `PCollection`.\n",
+ "\n",
+ "In many cases, especially for batch pipelines, this is what we want since we want to analyze all the data that we have.\n",
+ "\n",
+ "> âšī¸ `GlobalWindow` is not very useful in a streaming pipeline unless you only need element-wise transforms.\n",
+ "> Aggregations, like `GroupByKey` and `Combine`, need to process the entire window, but a streaming pipeline has no end, so they would never finish."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "xDXdE9uysriw",
+ "outputId": "b39e7fe7-dc13-4d77-89af-f2d1312ab673"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# All elements fall into the GlobalWindow by default.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " (\n",
+ " pipeline\n",
+ " | 'Astrolonomical events' >> AstronomicalEvents()\n",
+ " | 'Print element info' >> beam.ParDo(PrintElementInfo())\n",
+ " | 'Print window info' >> PrintWindowInfo()\n",
+ " )"
+ ],
+ "execution_count": 4,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "[GlobalWindow] 2021-03-20 03:37:00 -- March Equinox 2021\n",
+ "[GlobalWindow] 2021-04-26 22:31:00 -- Super full moon\n",
+ "[GlobalWindow] 2021-05-11 13:59:00 -- Micro new moon\n",
+ "[GlobalWindow] 2021-05-26 06:13:00 -- Super full moon, total lunar eclipse\n",
+ "[GlobalWindow] 2021-06-20 22:32:00 -- June Solstice 2021\n",
+ "[GlobalWindow] 2021-08-22 07:01:00 -- Blue moon\n",
+ "[GlobalWindow] 2021-09-22 14:21:00 -- September Equinox 2021\n",
+ "[GlobalWindow] 2021-12-21 09:59:00 -- December Solstice 2021\n",
+ "[GlobalWindow] 2021-11-04 15:14:00 -- Super new moon\n",
+ "[GlobalWindow] 2021-11-19 02:57:00 -- Micro full moon, partial lunar eclipse\n",
+ "[GlobalWindow] 2021-12-04 01:43:00 -- Super new moon\n",
+ "[GlobalWindow] 2021-12-18 10:35:00 -- Micro full moon\n",
+ ">> Window [GlobalWindow] has 12 elements\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "l3Kod_pR7a7S"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "7WkYLzFCo4Rl"
+ },
+ "source": [
+ "# Fixed time windows\n",
+ "\n",
+ "If we want to analyze our data hourly, daily, monthly, etc. We might want to create evenly spaced intervals.\n",
+ "\n",
+ "[`FixedWindows`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html#apache_beam.transforms.window.FixedWindows)\n",
+ "allow us to create fixed-sized windows.\n",
+ "We only need to specify the _window size_ in seconds.\n",
+ "\n",
+ "In Python, we can use [`timedelta`](https://docs.python.org/3/library/datetime.html#timedelta-objects)\n",
+ "to help us do the conversion of minutes, hours, or days for us.\n",
+ "\n",
+ "> âšī¸ Some time deltas like a month cannot be so easily converted into seconds, since a month can have from 28 to 31 days.\n",
Review comment:
Makes sense, I'm changing all the months mentions to days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos commented on pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos commented on pull request #14962:
URL: https://github.com/apache/beam/pull/14962#issuecomment-870789854
> Overall looks great. Gardening project, moon phases, the notebook has got personality and I like it a lot :)
>
> Some comments as I was going through the [notebook](https://colab.research.google.com/github/davidcavazos/beam/blob/tour-of-beam/examples/notebooks/tour-of-beam/windowing.ipynb):
>
> 1. `s/lets/let's`
Thanks, done.
> 2. Suggested edit - redundancy: "In our example, the "processing" is done by PrintElementInfo which simply prints the element with its window information. ~For windows of three months every month~, each element is processed three times, one time per window."
Changed
> 3. Suggested edit - use a bulleted list: "Sliding windows allow us to do just that. We need to specify the window size in seconds just like with FixedWindows. We also need to specify a window period in seconds, which is how often we want to emit each window."
Changed
> 4. Suggested edit - redundancy - future tense: "If the next event happens within the next 30 days ~or less, like 20 days after the previous event,~ the session window [will extend and cover..] extends and covers that as well. If there are no new events for the next 30 days, the session window [will close..] closes and is emitted."
Changed
> 5. I had to scroll up to remember what input events look like in the later examples. It would be nice to include a screenshot of the input events to the right of the nice bar charts, then these screenshots can be easily included by other tutorials/talks.
I actually ended up inlining the data on each snippet. It makes each sample a little more self contained, and simplified some indirection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] davidcavazos edited a comment on pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
davidcavazos edited a comment on pull request #14962:
URL: https://github.com/apache/beam/pull/14962#issuecomment-870792499
@aaltay sorry for the delay on this one, but comments have been addressed and should be ready for review.
R: @pcoet -- for documentation
FYI: @rosetn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #14962: [BEAM-10937] Tour of Beam Windowing notebook
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #14962:
URL: https://github.com/apache/beam/pull/14962#discussion_r647026286
##########
File path: examples/notebooks/tour-of-beam/windowing.ipynb
##########
@@ -0,0 +1,703 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Windowing -- Tour of Beam",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "code",
+ "metadata": {
+ "cellView": "form",
+ "id": "upmJn_DjcThx"
+ },
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE"
+ },
+ "source": [
+ "# Windowing -- _Tour of Beam_\n",
+ "\n",
+ "Sometimes, we want to [aggregate](https://beam.apache.org/documentation/transforms/python/overview/#aggregation) data, like `GroupByKey` or `Combine`, only at certain intervals, like hourly or daily, instead of processing the entire `PCollection` of data only once.\n",
+ "\n",
+ "We might want to emit a [moving average](https://en.wikipedia.org/wiki/Moving_average) as we're processing data.\n",
+ "\n",
+ "Maybe we want to analyze the user experience for a certain task in a web app, it would be nice to get the app events by sessions of activity.\n",
+ "\n",
+ "Or we could be running a streaming pipeline, and there is no end to the data, so how can we aggregate data?\n",
+ "\n",
+ "_Windows_ in Beam allow us to process only certain data intervals at a time.\n",
+ "In this notebook, we go through different ways of windowing our pipeline.\n",
+ "\n",
+ "Lets begin by installing `apache-beam`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg"
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_OkWHiAvpWDZ"
+ },
+ "source": [
+ "First, lets define some helper functions to simplify the rest of the examples.\n",
+ "\n",
+ "We have a transform to help us analyze an element alongside its window information, and we have another transform to help us analyze how many elements landed into each window.\n",
+ "We use a custom [`DoFn`](https://beam.apache.org/documentation/transforms/python/elementwise/pardo)\n",
+ "to access that information.\n",
+ "\n",
+ "You don't need to understand these, you just need to know they exist đ."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "C9yAN1Hgk3dF"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def human_readable_window(window) -> str:\n",
+ " \"\"\"Formats a window object into a human readable string.\"\"\"\n",
+ " if isinstance(window, beam.window.GlobalWindow):\n",
+ " return str(window)\n",
+ " return f'{window.start.to_utc_datetime()} - {window.end.to_utc_datetime()}'\n",
+ "\n",
+ "class PrintElementInfo(beam.DoFn):\n",
+ " \"\"\"Prints an element with its Window information.\"\"\"\n",
+ " def process(self, element, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):\n",
+ " print(f'[{human_readable_window(window)}] {timestamp.to_utc_datetime()} -- {element}')\n",
+ " yield element\n",
+ "\n",
+ "@beam.ptransform_fn\n",
Review comment:
Maybe we can use either `beam.DoFn` or `@beam.ptransform_fn` consistently to reduce the number of concepts.
##########
File path: examples/notebooks/tour-of-beam/windowing.ipynb
##########
@@ -0,0 +1,703 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Windowing -- Tour of Beam",
+ "provenance": [],
+ "collapsed_sections": [],
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "code",
+ "metadata": {
+ "cellView": "form",
+ "id": "upmJn_DjcThx"
+ },
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5UC_aGanx6oE"
+ },
+ "source": [
+ "# Windowing -- _Tour of Beam_\n",
+ "\n",
+ "Sometimes, we want to [aggregate](https://beam.apache.org/documentation/transforms/python/overview/#aggregation) data, like `GroupByKey` or `Combine`, only at certain intervals, like hourly or daily, instead of processing the entire `PCollection` of data only once.\n",
+ "\n",
+ "We might want to emit a [moving average](https://en.wikipedia.org/wiki/Moving_average) as we're processing data.\n",
+ "\n",
+ "Maybe we want to analyze the user experience for a certain task in a web app, it would be nice to get the app events by sessions of activity.\n",
+ "\n",
+ "Or we could be running a streaming pipeline, and there is no end to the data, so how can we aggregate data?\n",
+ "\n",
+ "_Windows_ in Beam allow us to process only certain data intervals at a time.\n",
+ "In this notebook, we go through different ways of windowing our pipeline.\n",
+ "\n",
+ "Lets begin by installing `apache-beam`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "R_Yhoc6N_Flg"
+ },
+ "source": [
+ "# Install apache-beam with pip.\n",
+ "!pip install --quiet apache-beam"
+ ],
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_OkWHiAvpWDZ"
+ },
+ "source": [
+ "First, lets define some helper functions to simplify the rest of the examples.\n",
+ "\n",
+ "We have a transform to help us analyze an element alongside its window information, and we have another transform to help us analyze how many elements landed into each window.\n",
+ "We use a custom [`DoFn`](https://beam.apache.org/documentation/transforms/python/elementwise/pardo)\n",
+ "to access that information.\n",
+ "\n",
+ "You don't need to understand these, you just need to know they exist đ."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "C9yAN1Hgk3dF"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def human_readable_window(window) -> str:\n",
+ " \"\"\"Formats a window object into a human readable string.\"\"\"\n",
+ " if isinstance(window, beam.window.GlobalWindow):\n",
+ " return str(window)\n",
+ " return f'{window.start.to_utc_datetime()} - {window.end.to_utc_datetime()}'\n",
+ "\n",
+ "class PrintElementInfo(beam.DoFn):\n",
+ " \"\"\"Prints an element with its Window information.\"\"\"\n",
+ " def process(self, element, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):\n",
+ " print(f'[{human_readable_window(window)}] {timestamp.to_utc_datetime()} -- {element}')\n",
+ " yield element\n",
+ "\n",
+ "@beam.ptransform_fn\n",
+ "def PrintWindowInfo(pcollection):\n",
+ " \"\"\"Prints the Window information with how many elements landed in that window.\"\"\"\n",
+ " class PrintCountsInfo(beam.DoFn):\n",
+ " def process(self, num_elements, window=beam.DoFn.WindowParam):\n",
+ " print(f'>> Window [{human_readable_window(window)}] has {num_elements} elements')\n",
+ " yield num_elements\n",
+ "\n",
+ " return (\n",
+ " pcollection\n",
+ " | 'Count elements per window' >> beam.combiners.Count.Globally().without_defaults()\n",
+ " | 'Print counts info' >> beam.ParDo(PrintCountsInfo())\n",
+ " )"
+ ],
+ "execution_count": 1,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "CQrojV2QnqIU"
+ },
+ "source": [
+ "Now lets create some data to use in the examples.\n",
+ "\n",
+ "Windows define data intervals based on time, so we need to tell Apache Beam a timestamp for each element.\n",
+ "\n",
+ "We define a `PTransform` for convenience, so we can attach the timestamps automatically.\n",
+ "\n",
+ "Apache Beam requires us to provide the timestamp as [Unix time](https://en.wikipedia.org/wiki/Unix_time), which is a way to represent a date and time as the number of seconds since January 1st, 1970.\n",
+ "\n",
+ "For our data, lets analyze some events about the seasons and moon phases for the year 2021, which might be [useful for a gardening project](https://www.almanac.com/content/planting-by-the-moon).\n",
+ "\n",
+ "To attach timestamps to each element, we can `Map` each element and return a [`TimestmpedValue`](https://beam.apache.org/documentation/transforms/python/elementwise/withtimestamps/)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "Sgzscopvmh1f",
+ "outputId": "e0c6fc19-ab97-4754-8f1f-1601807be940"
+ },
+ "source": [
+ "import time\n",
+ "from apache_beam.options.pipeline_options import PipelineOptions\n",
+ "\n",
+ "def to_unix_time(time_str: str, time_format='%Y-%m-%d %H:%M:%S') -> int:\n",
+ " \"\"\"Converts a time string into Unix time.\"\"\"\n",
+ " time_tuple = time.strptime(time_str, time_format)\n",
+ " return int(time.mktime(time_tuple))\n",
+ "\n",
+ "@beam.ptransform_fn\n",
+ "@beam.typehints.with_input_types(beam.pvalue.PBegin)\n",
+ "@beam.typehints.with_output_types(beam.window.TimestampedValue)\n",
+ "def AstronomicalEvents(pipeline):\n",
+ " return (\n",
+ " pipeline\n",
+ " | 'Create data' >> beam.Create([\n",
+ " ('2021-03-20 03:37:00', 'March Equinox 2021'),\n",
+ " ('2021-04-26 22:31:00', 'Super full moon'),\n",
+ " ('2021-05-11 13:59:00', 'Micro new moon'),\n",
+ " ('2021-05-26 06:13:00', 'Super full moon, total lunar eclipse'),\n",
+ " ('2021-06-20 22:32:00', 'June Solstice 2021'),\n",
+ " ('2021-08-22 07:01:00', 'Blue moon'),\n",
+ " ('2021-09-22 14:21:00', 'September Equinox 2021'),\n",
+ " ('2021-11-04 15:14:00', 'Super new moon'),\n",
+ " ('2021-11-19 02:57:00', 'Micro full moon, partial lunar eclipse'),\n",
+ " ('2021-12-04 01:43:00', 'Super new moon'),\n",
+ " ('2021-12-18 10:35:00', 'Micro full moon'),\n",
+ " ('2021-12-21 09:59:00', 'December Solstice 2021'),\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.MapTuple(\n",
+ " lambda timestamp, element:\n",
+ " beam.window.TimestampedValue(element, to_unix_time(timestamp))\n",
+ " )\n",
+ " )\n",
+ "\n",
+ "# Lets see how the data looks like.\n",
+ "beam_options = PipelineOptions(flags=[], type_check_additional='all')\n",
+ "with beam.Pipeline(options=beam_options) as pipeline:\n",
+ " (\n",
+ " pipeline\n",
+ " | 'Astronomical events' >> AstronomicalEvents()\n",
+ " | 'Print element' >> beam.Map(print)\n",
+ " )"
+ ],
+ "execution_count": 3,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "March Equinox 2021\n",
+ "Super full moon\n",
+ "Micro new moon\n",
+ "Super full moon, total lunar eclipse\n",
+ "June Solstice 2021\n",
+ "Blue moon\n",
+ "September Equinox 2021\n",
+ "December Solstice 2021\n",
+ "Super new moon\n",
+ "Micro full moon, partial lunar eclipse\n",
+ "Super new moon\n",
+ "Micro full moon\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "qI0K3jSA2LbJ"
+ },
+ "source": [
+ "> âšī¸ After running this, it looks like the timestamps disappeared!\n",
+ "> They're actually still _implicitly_ part of the element, just like the windowing information.\n",
+ "> If we need to access it, we can do so via a custom [`DoFn`](https://beam.apache.org/documentation/transforms/python/elementwise/pardo).\n",
+ "> Aggregation transforms use each element's timestamp along with the windowing we specified to create windows of elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ymHF1WCqnG4V"
+ },
+ "source": [
+ "# Global window\n",
+ "\n",
+ "All pipelines use the [`GlobalWindow`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html#apache_beam.transforms.window.GlobalWindow) by default.\n",
+ "This is a single window that covers the entire `PCollection`.\n",
+ "\n",
+ "In many cases, especially for batch pipelines, this is what we want since we want to analyze all the data that we have.\n",
+ "\n",
+ "> âšī¸ `GlobalWindow` is not very useful in a streaming pipeline unless you only need element-wise transforms.\n",
+ "> Aggregations, like `GroupByKey` and `Combine`, need to process the entire window, but a streaming pipeline has no end, so they would never finish."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "xDXdE9uysriw",
+ "outputId": "b39e7fe7-dc13-4d77-89af-f2d1312ab673"
+ },
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# All elements fall into the GlobalWindow by default.\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " (\n",
+ " pipeline\n",
+ " | 'Astrolonomical events' >> AstronomicalEvents()\n",
+ " | 'Print element info' >> beam.ParDo(PrintElementInfo())\n",
+ " | 'Print window info' >> PrintWindowInfo()\n",
+ " )"
+ ],
+ "execution_count": 4,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "[GlobalWindow] 2021-03-20 03:37:00 -- March Equinox 2021\n",
+ "[GlobalWindow] 2021-04-26 22:31:00 -- Super full moon\n",
+ "[GlobalWindow] 2021-05-11 13:59:00 -- Micro new moon\n",
+ "[GlobalWindow] 2021-05-26 06:13:00 -- Super full moon, total lunar eclipse\n",
+ "[GlobalWindow] 2021-06-20 22:32:00 -- June Solstice 2021\n",
+ "[GlobalWindow] 2021-08-22 07:01:00 -- Blue moon\n",
+ "[GlobalWindow] 2021-09-22 14:21:00 -- September Equinox 2021\n",
+ "[GlobalWindow] 2021-12-21 09:59:00 -- December Solstice 2021\n",
+ "[GlobalWindow] 2021-11-04 15:14:00 -- Super new moon\n",
+ "[GlobalWindow] 2021-11-19 02:57:00 -- Micro full moon, partial lunar eclipse\n",
+ "[GlobalWindow] 2021-12-04 01:43:00 -- Super new moon\n",
+ "[GlobalWindow] 2021-12-18 10:35:00 -- Micro full moon\n",
+ ">> Window [GlobalWindow] has 12 elements\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "l3Kod_pR7a7S"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "7WkYLzFCo4Rl"
+ },
+ "source": [
+ "# Fixed time windows\n",
+ "\n",
+ "If we want to analyze our data hourly, daily, monthly, etc. We might want to create evenly spaced intervals.\n",
+ "\n",
+ "[`FixedWindows`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html#apache_beam.transforms.window.FixedWindows)\n",
+ "allow us to create fixed-sized windows.\n",
+ "We only need to specify the _window size_ in seconds.\n",
+ "\n",
+ "In Python, we can use [`timedelta`](https://docs.python.org/3/library/datetime.html#timedelta-objects)\n",
+ "to help us do the conversion of minutes, hours, or days for us.\n",
+ "\n",
+ "> âšī¸ Some time deltas like a month cannot be so easily converted into seconds, since a month can have from 28 to 31 days.\n",
Review comment:
I think you can either:
- Remove the month concept, drop this comment and and all the other approximately x months comments, and simply say 30 day, 90 day etc.
- Or introduce custom windows later on which can do more sophisticated things.
Probably the first one (simplifying) would be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org