You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/01/30 02:54:37 UTC
[beam-site] branch mergebot updated (db51ee5 -> 99936db)
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.
from db51ee5 This closes #370
add f0e45db Prepare repository for deployment.
new 51836e2 Fix broken links due to code directory moves
new 99936db This closes #380
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
content/blog/2018/01/09/beam-a-look-back.html | 280 +++++++++++++++++++++
content/blog/index.html | 21 ++
content/feed.xml | 176 ++++++++++---
.../images/blog/2017-look-back/timeline.png | Bin
content/index.html | 10 +-
.../2016-10-12-strata-hadoop-world-and-beam.md | 2 +-
src/_posts/2016-10-20-test-stream.md | 4 +-
src/documentation/io/testing.md | 2 +-
src/documentation/programming-guide.md | 8 +-
src/get-started/mobile-gaming-example.md | 30 +--
10 files changed, 469 insertions(+), 64 deletions(-)
create mode 100644 content/blog/2018/01/09/beam-a-look-back.html
copy {src => content}/images/blog/2017-look-back/timeline.png (100%)
--
To stop receiving notification emails like this one, please contact
mergebot-role@apache.org.
[beam-site] 01/02: Fix broken links due to code directory moves
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 51836e26f82b8f5cef3fac14cb158cdc487a0b07
Author: melissa <me...@google.com>
AuthorDate: Mon Jan 29 10:28:50 2018 -0800
Fix broken links due to code directory moves
---
.../2016-10-12-strata-hadoop-world-and-beam.md | 2 +-
src/_posts/2016-10-20-test-stream.md | 4 +--
src/documentation/io/testing.md | 2 +-
src/documentation/programming-guide.md | 8 +++---
src/get-started/mobile-gaming-example.md | 30 +++++++++++-----------
5 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/_posts/2016-10-12-strata-hadoop-world-and-beam.md b/src/_posts/2016-10-12-strata-hadoop-world-and-beam.md
index 1642edd..3b8eb37 100644
--- a/src/_posts/2016-10-12-strata-hadoop-world-and-beam.md
+++ b/src/_posts/2016-10-12-strata-hadoop-world-and-beam.md
@@ -12,7 +12,7 @@ Tyler Akidau and I gave a [three-hour tutorial](http://conferences.oreilly.com/s
<img src="{{ "/images/blog/IMG_20160927_170956.jpg" | prepend: site.baseurl }}" alt="Exercise time">
-If you want to take a look at the tutorial materials, we’ve put them up [on GitHub](https://github.com/eljefe6a/beamexample). This includes the [actual slides](https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/slides.pdf) as well as the [exercises](https://github.com/eljefe6a/beamexample/tree/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game) that we covered. If you’re looking to learn a little about Beam, this is a good way to start. The exercises a [...]
+If you want to take a look at the tutorial materials, we’ve put them up [on GitHub](https://github.com/eljefe6a/beamexample). This includes the [actual slides](https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/slides.pdf) as well as the [exercises](https://github.com/eljefe6a/beamexample/tree/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game) that we covered. If you’re looking to learn a little about Beam, this is a good way to start. The exercises a [...]
I want to share some of takeaways I had about Beam during the conference.
diff --git a/src/_posts/2016-10-20-test-stream.md b/src/_posts/2016-10-20-test-stream.md
index 500680a..a65a8b7 100644
--- a/src/_posts/2016-10-20-test-stream.md
+++ b/src/_posts/2016-10-20-test-stream.md
@@ -45,8 +45,8 @@ from the Mobile Gaming example series.
## LeaderBoard and the Mobile Gaming Example
-[LeaderBoard](https://github.com/apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java#L177)
-is part of the [Beam mobile gaming examples](https://github.com/apache/beam/tree/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game)
+[LeaderBoard](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java#L177)
+is part of the [Beam mobile gaming examples](https://github.com/apache/beam/tree/master/examples/java/src/main/java/org/apache/beam/examples/complete/game)
(and [walkthroughs]({{ site.baseurl }}/get-started/mobile-gaming-example/))
which produces a continuous accounting of user and team scores. User scores are
calculated over the lifetime of the program, while team scores are calculated
diff --git a/src/documentation/io/testing.md b/src/documentation/io/testing.md
index 0c3f439..cac7b8a 100644
--- a/src/documentation/io/testing.md
+++ b/src/documentation/io/testing.md
@@ -561,7 +561,7 @@ You can do this by:
1. Creating two Kubernetes scripts: one for a small instance of the data store, and one for a large instance.
1. Having your test take a pipeline option that decides whether to generate a small or large amount of test data (where small and large are sizes appropriate to your data store)
-An example of this is [HadoopInputFormatIO](https://github.com/apache/beam/tree/master/sdks/java/io/hadoop/input-format)'s tests.
+An example of this is [HadoopInputFormatIO](https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-input-format)'s tests.
<!--
# Next steps
diff --git a/src/documentation/programming-guide.md b/src/documentation/programming-guide.md
index 796f664..9910ef6 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -876,7 +876,7 @@ data contains names and phone numbers.
</span>
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/test/java/org/apache/beam/examples/website_snippets/SnippetsTest.java tag:CoGroupByKeyTupleInputs
+{% github_sample /apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/website_snippets/SnippetsTest.java tag:CoGroupByKeyTupleInputs
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_group_by_key_cogroupbykey_tuple_inputs
@@ -886,7 +886,7 @@ After `CoGroupByKey`, the resulting data contains all data associated with each
unique key from any of the input collections.
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/test/java/org/apache/beam/examples/website_snippets/SnippetsTest.java tag:CoGroupByKeyTupleOutputs
+{% github_sample /apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/website_snippets/SnippetsTest.java tag:CoGroupByKeyTupleOutputs
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_group_by_key_cogroupbykey_tuple_outputs
@@ -897,7 +897,7 @@ followed by a `ParDo` to consume the result. Then, the code uses tags to look up
and format data from each collection.
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/website_snippets/Snippets.java tag:CoGroupByKeyTuple
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/website_snippets/Snippets.java tag:CoGroupByKeyTuple
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:model_group_by_key_cogroupbykey_tuple
@@ -906,7 +906,7 @@ and format data from each collection.
The formatted data looks like this:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/test/java/org/apache/beam/examples/website_snippets/SnippetsTest.java tag:CoGroupByKeyTupleFormattedOutputs
+{% github_sample /apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/website_snippets/SnippetsTest.java tag:CoGroupByKeyTupleFormattedOutputs
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_group_by_key_cogroupbykey_tuple_formatted_outputs
diff --git a/src/get-started/mobile-gaming-example.md b/src/get-started/mobile-gaming-example.md
index 9a734c0..49d1530 100644
--- a/src/get-started/mobile-gaming-example.md
+++ b/src/get-started/mobile-gaming-example.md
@@ -60,7 +60,7 @@ The Mobile Gaming example pipelines vary in complexity, from simple batch analys
The `UserScore` pipeline is the simplest example for processing mobile game data. `UserScore` determines the total score per user over a finite data set (for example, one day's worth of scores stored on the game server). Pipelines like `UserScore` are best run periodically after all relevant data has been gathered. For example, `UserScore` could run as a nightly job over data gathered during that day.
{:.language-java}
-> **Note:** See [UserScore on GitHub](https://github.com/apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/UserScore.java) for the complete example pipeline program.
+> **Note:** See [UserScore on GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java) for the complete example pipeline program.
{:.language-py}
> **Note:** See [UserScore on GitHub](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/user_score.py) for the complete example pipeline program.
@@ -93,7 +93,7 @@ This example uses batch processing, and the diagram's Y axis represents processi
After reading the score events from the input file, the pipeline groups all of those user/score pairs together and sums the score values into one total value per unique user. `UserScore` encapsulates the core logic for that step as the [user-defined composite transform]({{ site.baseurl }}/documentation/programming-guide/#composite-transforms) `ExtractAndSumScore`:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/UserScore.java tag:DocInclude_USExtractXform
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java tag:DocInclude_USExtractXform
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/user_score.py tag:extract_and_sum_score
@@ -104,7 +104,7 @@ After reading the score events from the input file, the pipeline groups all of t
Here's the main method of `UserScore`, showing how we apply all three steps of the pipeline:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/UserScore.java tag:DocInclude_USMain
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java tag:DocInclude_USMain
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/user_score.py tag:main
@@ -131,7 +131,7 @@ The `HourlyTeamScore` pipeline expands on the basic batch analysis principles us
Like `UserScore`, `HourlyTeamScore` is best thought of as a job to be run periodically after all the relevant data has been gathered (such as once per day). The pipeline reads a fixed data set from a file, and writes the results to a Google Cloud BigQuery table.
{:.language-java}
-> **Note:** See [HourlyTeamScore on GitHub](https://github.com/apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java) for the complete example pipeline program.
+> **Note:** See [HourlyTeamScore on GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java) for the complete example pipeline program.
{:.language-py}
> **Note:** See [HourlyTeamScore on GitHub](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py) for the complete example pipeline program.
@@ -173,7 +173,7 @@ Beam's windowing feature uses the [intrinsic timestamp information]({{ site.base
The following code shows this:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java tag:DocInclude_HTSAddTsAndWindow
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java tag:DocInclude_HTSAddTsAndWindow
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py tag:add_timestamp_and_window
@@ -192,7 +192,7 @@ It also lets the pipeline include relevant **late data**—data events with vali
The following code shows how `HourlyTeamScore` uses the `Filter` transform to filter events that occur either before or after the relevant analysis period:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java tag:DocInclude_HTSFilters
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java tag:DocInclude_HTSFilters
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py tag:filter_by_time_range
@@ -203,7 +203,7 @@ The following code shows how `HourlyTeamScore` uses the `Filter` transform to fi
`HourlyTeamScore` uses the same `ExtractAndSumScores` transform as the `UserScore` pipeline, but passes a different key (team, as opposed to user). Also, because the pipeline applies `ExtractAndSumScores` _after_ applying fixed-time 1-hour windowing to the input data, the data gets grouped by both team _and_ window. You can see the full sequence of transforms in `HourlyTeamScore`'s main method:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java tag:DocInclude_HTSMain
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java tag:DocInclude_HTSMain
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py tag:main
@@ -224,7 +224,7 @@ The `LeaderBoard` pipeline also demonstrates how to process game score data with
Because the `LeaderBoard` pipeline reads the game data from an unbounded source as that data is generated, you can think of the pipeline as an ongoing job running concurrently with the game process. `LeaderBoard` can thus provide low-latency insights into how users are playing the game at any given moment — useful if, for example, we want to provide a live web-based scoreboard so that users can track their progress against other users as they play.
{:.language-java}
-> **Note:** See [LeaderBoard on GitHub](https://github.com/apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java) for the complete example pipeline program.
+> **Note:** See [LeaderBoard on GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java) for the complete example pipeline program.
{:.language-py}
> **Note:** See [LeaderBoard on GitHub](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/leader_board.py) for the complete example pipeline program.
@@ -261,7 +261,7 @@ As processing time advances and more scores are processed, the trigger outputs t
The following code example shows how `LeaderBoard` sets the processing time trigger to output the data for user scores:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java tag:DocInclude_ProcTimeTrigger
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java tag:DocInclude_ProcTimeTrigger
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/leader_board.py tag:processing_time_trigger
@@ -295,7 +295,7 @@ Data arriving above the solid watermark line is _late data_ — this is a score
The following code example shows how `LeaderBoard` applies fixed-time windowing with the appropriate triggers to have our pipeline perform the calculations we want:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java tag:DocInclude_WindowAndTrigger
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java tag:DocInclude_WindowAndTrigger
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/leader_board.py tag:window_and_trigger
@@ -310,7 +310,7 @@ While `LeaderBoard` demonstrates how to use basic windowing and triggers to perf
Like `LeaderBoard`, `GameStats` reads data from an unbounded source. It is best thought of as an ongoing job that provides insight into the game as users play.
{:.language-java}
-> **Note:** See [GameStats on GitHub](https://github.com/apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java) for the complete example pipeline program.
+> **Note:** See [GameStats on GitHub](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/GameStats.java) for the complete example pipeline program.
{:.language-py}
> **Note:** See [GameStats on GitHub](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py) for the complete example pipeline program.
@@ -335,7 +335,7 @@ Since the average depends on the pipeline data, we need to calculate it, and the
The following code example shows the composite transform that handles abuse detection. The transform uses the `Sum.integersPerKey` transform to sum all scores per user, and then the `Mean.globally` transform to determine the average score for all users. Once that's been calculated (as a `PCollectionView` singleton), we can pass it to the filtering `ParDo` using `.withSideInputs`:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java tag:DocInclude_AbuseDetect
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/GameStats.java tag:DocInclude_AbuseDetect
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py tag:abuse_detect
@@ -344,7 +344,7 @@ The following code example shows the composite transform that handles abuse dete
The abuse-detection transform generates a view of users supected to be spambots. Later in the pipeline, we use that view to filter out any such users when we calculate the team score per hour, again by using the side input mechanism. The following code example shows where we insert the spam filter, between windowing the scores into fixed windows and extracting the team scores:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java tag:DocInclude_FilterAndCalc
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/GameStats.java tag:DocInclude_FilterAndCalc
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py tag:filter_and_calc
@@ -368,7 +368,7 @@ between instances are.*
We can use the session-windowed data to determine the average length of uninterrupted play time for all of our users, as well as the total score they achieve during each session. We can do this in the code by first applying session windows, summing the score per user and session, and then using a transform to calculate the length of each individual session:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java tag:DocInclude_SessionCalc
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/GameStats.java tag:DocInclude_SessionCalc
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py tag:session_calc
@@ -377,7 +377,7 @@ We can use the session-windowed data to determine the average length of uninterr
This gives us a set of user sessions, each with an attached duration. We can then calculate the _average_ session length by re-windowing the data into fixed time windows, and then calculating the average for all sessions that end in each hour:
```java
-{% github_sample /apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java tag:DocInclude_Rewindow
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/GameStats.java tag:DocInclude_Rewindow
%}```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py tag:rewindow
--
To stop receiving notification emails like this one, please contact
mergebot-role@apache.org.
[beam-site] 02/02: This closes #380
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 99936db217fee3682f78852cb57f2813aa2d779d
Merge: f0e45db 51836e2
Author: Mergebot <me...@apache.org>
AuthorDate: Mon Jan 29 18:54:00 2018 -0800
This closes #380
.../2016-10-12-strata-hadoop-world-and-beam.md | 2 +-
src/_posts/2016-10-20-test-stream.md | 4 +--
src/documentation/io/testing.md | 2 +-
src/documentation/programming-guide.md | 8 +++---
src/get-started/mobile-gaming-example.md | 30 +++++++++++-----------
5 files changed, 23 insertions(+), 23 deletions(-)
--
To stop receiving notification emails like this one, please contact
mergebot-role@apache.org.