You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2019/09/04 17:00:26 UTC
[beam] branch asf-site updated: Publishing website 2019/09/04
17:00:12 at commit d251702
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new da2ed38 Publishing website 2019/09/04 17:00:12 at commit d251702
da2ed38 is described below
commit da2ed38e5fe9402d76d691f1f733a91e9fbc49d0
Author: jenkins <bu...@apache.org>
AuthorDate: Wed Sep 4 17:00:13 2019 +0000
Publishing website 2019/09/04 17:00:12 at commit d251702
---
.../{index.html => blog/2019/09/04/gsoc-19.html} | 299 +++++++--------------
website/generated-content/blog/index.html | 32 +++
website/generated-content/feed.xml | 162 ++++++-----
website/generated-content/index.html | 10 +-
4 files changed, 232 insertions(+), 271 deletions(-)
diff --git a/website/generated-content/index.html b/website/generated-content/blog/2019/09/04/gsoc-19.html
similarity index 53%
copy from website/generated-content/index.html
copy to website/generated-content/blog/2019/09/04/gsoc-19.html
index 57b28e2..f651080 100644
--- a/website/generated-content/index.html
+++ b/website/generated-content/blog/2019/09/04/gsoc-19.html
@@ -28,9 +28,8 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Apache Beam</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
+ <title>Google Summer of Code '19</title>
+ <meta name="description" content="Google Summer of Code was an amazing learning experience for me.I contributed to open source, learned about Apache Beam’s internals and worked with the best ...">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
<link rel="stylesheet" href="/css/site.css">
<script src="https://code.jquery.com/jquery-2.2.4.min.js"></script>
@@ -39,7 +38,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/" data-proofer-ignore>
+ <link rel="canonical" href="https://beam.apache.org/blog/2019/09/04/gsoc-19.html" data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css" integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous">
@@ -53,7 +52,7 @@
</script>
</head>
- <body class="body body--index">
+ <body class="body ">
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -141,7 +140,7 @@
GitHub links will not resolve until the markdown source is available on the master branch.
New pages would fail validation during development / PR test automation.
-->
- <a href="https://github.com/apache/beam/edit/master/website/src/index.md" data-proofer-ignore>
+ <a href="https://github.com/apache/beam/edit/master/website/src/_posts/2019-09-04-gsoc-19.md" data-proofer-ignore>
<i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on GitHub"></i>
</a>
</li>
@@ -151,6 +150,31 @@
<div class="body__contained">
<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Google Summer of Code '19</h1>
+ <p class="post-meta"><time datetime="2019-09-04T01:00:01-07:00" itemprop="datePublished">Sep 4, 2019</time> •
+ Tanay Tummalapalli [<a href="https://twitter.com/ttanay100">@ttanay100</a>]
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -163,207 +187,80 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<div class="hero-bg">
- <div class="hero section">
- <div class="hero__cols">
- <div class="hero__cols__col">
- <div class="hero__cols__col__content">
- <div class="hero__title">
- Apache Beam: An advanced unified programming model
- </div>
- <div class="hero__subtitle">
- Implement batch and streaming data processing jobs that run on any execution engine.
- </div>
- <div class="hero__ctas hero__ctas--first">
- <a class="button button--primary" href="/get-started/beam-overview/">Learn more</a>
- <a class="button button--primary" href="/get-started/try-apache-beam/">Try Beam</a>
- <a class="button button--primary" href="/get-started/downloads/">Download Beam SDK 2.15.0</a>
- </div>
- <div class="hero__ctas">
- <a class="button" href="/get-started/quickstart-java/">Java Quickstart</a>
- <a class="button" href="/get-started/quickstart-py/">Python Quickstart</a>
- <a class="button" href="/get-started/quickstart-go/">Go Quickstart</a>
- </div>
- </div>
- </div>
- <div class="hero__cols__col">
- <div class="hero__blog">
- <div class="hero__blog__title">
- The latest from the blog
- </div>
- <div class="hero__blog__cards">
-
- <a class="hero__blog__cards__card" href="/blog/2019/08/22/beam-2.15.0.html">
- <div class="hero__blog__cards__card__title">Apache Beam 2.15.0</div>
- <div class="hero__blog__cards__card__date">Aug 22, 2019</div>
- </a>
-
- <a class="hero__blog__cards__card" href="/blog/2019/07/31/beam-2.14.0.html">
- <div class="hero__blog__cards__card__title">Apache Beam 2.14.0</div>
- <div class="hero__blog__cards__card__date">Jul 31, 2019</div>
- </a>
-
- <a class="hero__blog__cards__card" href="/blog/2019/06/11/looping-timers.html">
- <div class="hero__blog__cards__card__title">Looping timers in Apache Beam</div>
- <div class="hero__blog__cards__card__date">Jun 11, 2019</div>
- </a>
-
- </div>
- </div>
- </div>
- </div>
- </div>
-</div>
-<div class="pillars section">
- <div class="pillars__title">
- All about Apache Beam
- </div>
- <div class="pillars__cols">
-
- <div class="pillars__cols__col">
- <div class="pillars__cols__col__title">
- Unified
- </div>
- <div class="pillars__cols__col__body">
- Use a single programming model for both batch and streaming use cases.
- </div>
- </div>
-
- <div class="pillars__cols__col">
- <div class="pillars__cols__col__title">
- Portable
- </div>
- <div class="pillars__cols__col__body">
- Execute pipelines on multiple execution environments.
- </div>
- </div>
-
- <div class="pillars__cols__col">
- <div class="pillars__cols__col__title">
- Extensible
- </div>
- <div class="pillars__cols__col__body">
- Write and share new SDKs, IO connectors, and transformation libraries.
- </div>
- </div>
-
- </div>
-</div>
+<p>Google Summer of Code was an amazing learning experience for me.
+I contributed to open source, learned about Apache Beam’s internals and worked with the best engineers in the world.</p>
-<div class="graphic section">
-<div class="graphic__image">
-<img src="/images/beam_architecture.png" alt="Beam architecture" />
-</div>
-</div>
+<!--more-->
-<div class="logos section">
- <div class="logos__title">
- Works with
- </div>
- <div class="logos__logos">
-
- <div class="logos__logos__logo">
- <a href="http://apex.apache.org"><img src="/images/logo_apex.png" alt="APEX" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://flink.apache.org"><img src="/images/logo_flink.png" alt="Flink" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://spark.apache.org/"><img src="/images/logo_spark.png" alt="Spark" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="https://cloud.google.com/dataflow/"><img src="/images/logo_google_cloud.png" alt="Google Cloud Dataflow" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://gearpump.apache.org/"><img src="/images/logo_gearpump.png" alt="Gearpump" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://samza.apache.org/"><img src="/images/logo_samza.png" alt="Samza" /></a>
- </div>
-
- </div>
-</div>
+<h2 id="motivation">Motivation</h2>
+<p>Two of my friends had participated in GSoC in 2018. I was intrigued by their experience.
+The idea of working on open-source software that could potentially be used by developers across the world, while being mentored by the best people in a field was exciting!
+So, I decided to give Google Summer of Code a shot this year.</p>
-<div class="cards section section--wide">
- <div class="section__contained">
- <div class="cards__title">
- Testimonials
- </div>
- <div class="cards__cards">
-
- <div class="cards__cards__card">
- <div class="cards__cards__card__body">
- A framework that delivers the flexibility and advanced functionality our customers need.
- </div>
- <div class="cards__cards__card__user">
- <!-- TODO: Implement icons.
- <div class="cards__cards__card__user__icon">
- </div>
- -->
- <div class="cards__cards__card__user__name">
- –Talend
- </div>
- </div>
- </div>
-
- <div class="cards__cards__card">
- <div class="cards__cards__card__body">
- Apache Beam has powerful semantics that solve real-world challenges of stream processing.
- </div>
- <div class="cards__cards__card__user">
- <!-- TODO: Implement icons.
- <div class="cards__cards__card__user__icon">
- </div>
- -->
- <div class="cards__cards__card__user__name">
- –PayPal
- </div>
- </div>
- </div>
-
- <div class="cards__cards__card">
- <div class="cards__cards__card__body">
- Apache Beam represents a principled approach for analyzing data streams.
- </div>
- <div class="cards__cards__card__user">
- <!-- TODO: Implement icons.
- <div class="cards__cards__card__user__icon">
- </div>
- -->
- <div class="cards__cards__card__user__name">
- –data Artisans
- </div>
- </div>
- </div>
-
- </div>
- <div class="cards__body">
- Beam is an open source community and contributions are greatly appreciated!
- If you’d like to contribute, please see the <a href="/contribute/">Contribute</a> section.
- </div>
- </div>
-</div>
+<h2 id="what-is-google-summer-of-code">What is Google Summer of Code?</h2>
+<p><a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> is a global program hosted by Google focused on introducing students to open source software development.
+Students work on a 3 month programming project with an open source organization during their break from university.</p>
+
+<h2 id="why-apache-beam">Why Apache Beam?</h2>
+<p>While interning at <a href="https://atlan.com/">Atlan</a>, I discovered the field of Data Engineering. I found the challenges and the discussions of the engineers there interesting. While researching for my internship project, I came across the Streaming Systems book. It introduced me to the unified model of Apache Beam for Batch and Streaming Systems, which I was fascinated by.
+I wanted to explore Data Engineering, so for GSoC, I wanted to work on a project in that field. Towards the end of my internship, I started contributing to Apache Airflow(very cool project) and Apache Beam, hoping one of them would participate in GSoC. I got lucky!</p>
+
+<p><a href="https://youtu.be/U2eWLb-LD44">Also, Spotify’s Discover Weekly uses Apache Beam!</a></p>
+
+<h2 id="preparation">Preparation</h2>
+<p>I had already read the <a href="http://streamingsystems.net/">Streaming Systems book</a>. So, I had an idea of the concepts that Beam is built on, but had never actually used Beam.
+Before actually submitting a proposal, I went through a bunch of resources to make sure I had a concrete understanding of Beam.
+I read the <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">Streaming 101</a> and <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102">Streaming 102</a> blogs by Tyler Akidau. They are the perfect introduction to Beam’s unified model for Batch and Streaming.
+In addition, I watched all Beam talks on YouTube. You can find them on the <a href="https://beam.apache.org/documentation/resources/videos-and-podcasts/">Beam Website</a>.
+Beam has really good documentation. The <a href="https://beam.apache.org/documentation/programming-guide/">Programming Guide</a> lays out all of Beam’s concepts really well. <a href="https://beam.apache.org/documentation/execution-model/">Beam’s execution model</a> is also documented well and is a must-read to understand how Beam processes data.
+<a href="https://www.waitingforcode.com/apache-beam">waitingforcode.com</a> also has good blog posts about Beam concepts.
+To get a better sense of the Beam codebase, I played around with it and worked on some PRs to understand Beam better and got familiar with the test suite and workflows.</p>
+
+<h2 id="gsoc-journey">GSoC Journey</h2>
+<p>GSoC has 2 phases. The first is the Community Bonding period in which students get familiar with the project and the community. The other is the actual Coding Period in which students work on their projects. Since the Coding Period has three evaluations spaced out by a month, I divided my project into three parts focusing on the implementation, tests, and documentation or improvements.</p>
+
+<h3 id="project">Project</h3>
+<p>My project(<a href="https://issues.apache.org/jira/browse/BEAM-6611">BEAM-6611</a>) added support for File Loads method of inserting data into BigQuery for streaming pipelines. It builds on PR - <a href="https://github.com/apache/beam/pull/7655">#7655</a> for <a href="https://issues.apache.org/jira/browse/BEAM-6553">BEAM-6553</a> that added support in the Python SDK for writing to BigQuery using File Loads method for Batch pipelines. Streaming pipelines with non-default Windowing, Tri [...]
+You can find my proposal <a href="https://docs.google.com/document/d/15Peyd3Z_wu5rvGWw8lMLpZuTyyreM_JOAEFFWvF97YY/edit?usp=sharing">here</a>.</p>
+
+<h3 id="community-bonding">Community Bonding</h3>
+<p>When GSoC started, my semester end exams had not yet finished. As a result, I couldn’t get much done. I worked on three PTransforms for the Python SDK - Latest, WithKeys and Reify.</p>
+
+<h3 id="coding-period-i">Coding Period I</h3>
+<p>In this period, I wrote some Integration Tests for the BigQuery sink using Streaming Inserts in streaming mode. I worked on a failing integration test for my project. I also finished the implementation of my project. But, one PostCommit test didn’t pass. I realized that the matcher for the Integration Test that queried BigQuery for the results was intended to be used in Batch mode. So, I wrote a version of the matcher to work in streaming mode.</p>
+
+<h3 id="coding-period-ii">Coding Period II</h3>
+<p>Even after I had added the matcher for streaming mode, the PostComit tests did not pass. A test was being run even though it was not specified. I isolated the failure to a <a href="https://nose.readthedocs.io/en/latest/doc_tests/test_multiprocess/multiprocess.html#other-differences-in-test-running">limitation</a> of the multiprocess plugin for <a href="https://nose.readthedocs.io/en/latest/">nose(a Python test framework)</a> due to which it found more tests than had been specified. It [...]
+I also worked on small issues related to testing.</p>
+
+<p>This period was marked by a few exciting events:</p>
+<ul>
+ <li>Ending up in the top #100 contributors to apache/beam.</li>
+ <li>My first ever PR Review on an open source project.</li>
+</ul>
+
+<p><img src="https://pbs.twimg.com/media/D_XNSC-UIAUmswG?format=png&name=small" alt="Weird flex but ok" /></p>
+
+<h3 id="coding-period-iii">Coding Period III</h3>
+<p>This was the final coding period before the program ended. Since my project was merged earlier than expected, my mentor suggested another issue(<a href="https://issues.apache.org/jira/browse/BEAM-7742">BEAM-7742</a>) in the same area - BigQueryIO, that I found interesting. So, I worked on partitioning written files in BigQuery to ensure that all load jobs triggered adhere to the load job size limitations specified for BigQuery.
+While working on my project, I was using a pipeline that uses PubSub as a source and BigQuery as a sink to validate my changes. My mentor suggested we add them to the Beam test suite as it would be the ultimate test for BigQueryIO. I also worked on adding this test to Beam.</p>
+
+<p>You can find the list of PRs I worked on <a href="https://github.com/apache/beam/pulls?utf8=%E2%9C%93&q=is%3Apr+author%3Attanay">here</a>.</p>
+
+<h2 id="conclusion">Conclusion</h2>
+<p>GSoC has been a lesson in discipline and goal-setting for me. Deciding what I wanted to work on and how much I wanted to get done each week was an important lesson.
+I had never worked remotely, so this was a new experience. Although I struggled with it initially, I appreciate the flexibility that it comes with.
+I also had a lot of fun learning about Apache Beam’s internals, and other tools in the same ecosystem.
+This was also the first time I had written code with a test-first approach.</p>
+
+<p>I thank my mentor - Pablo Estrada, Apache Beam, The Apache Software Foundation and Google Summer of Code for this opportunity. I am also grateful to my mentor for helping me with everything I needed and more, and the Apache Beam community for being supportive and encouraging.</p>
+
+<p>With the right effort, perseverance, conviction, and a plan, anything is possible. Anything.</p>
-<div class="ctas section">
- <div class="ctas__title">
- Get started
- </div>
- <div class="ctas__ctas ctas__ctas--top">
- <a class="button button--primary" href="/get-started/beam-overview/">Learn more</a>
- <a class="button button--primary" href="/get-started/downloads/">Download Beam SDK 2.15.0</a>
- </div>
- <div class="ctas__ctas">
- <a class="button" href="/get-started/quickstart-java/">Java Quickstart</a>
- <a class="button" href="/get-started/quickstart-py/">Python Quickstart</a>
- <a class="button" href="/get-started/quickstart-go/">Go Quickstart</a>
</div>
-</div>
+
+</article>
</div>
<!--
diff --git a/website/generated-content/blog/index.html b/website/generated-content/blog/index.html
index 0352aea..6d6b1ff 100644
--- a/website/generated-content/blog/index.html
+++ b/website/generated-content/blog/index.html
@@ -169,6 +169,38 @@ limitations under the License.
<p>This is the blog for the Apache Beam project. This blog contains news and updates
for the project.</p>
+<h3 id="google-summer-of-code-19"><a class="post-link" href="/blog/2019/09/04/gsoc-19.html">Google Summer of Code ‘19</a></h3>
+<p><i>Sep 4, 2019 •
+ Tanay Tummalapalli [<a href="https://twitter.com/ttanay100">@ttanay100</a>]
+</i></p>
+
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>Google Summer of Code was an amazing learning experience for me.
+I contributed to open source, learned about Apache Beam’s internals and worked with the best engineers in the world.</p>
+
+<!-- Render a "read more" button if the post is longer than the excerpt -->
+
+<p>
+<a class="btn btn-default btn-sm" href="/blog/2019/09/04/gsoc-19.html" role="button">
+Read more <span class="glyphicon glyphicon-menu-right" aria-hidden="true"></span>
+</a>
+</p>
+
+<hr />
+
<h3 id="apache-beam-2150"><a class="post-link" href="/blog/2019/08/22/beam-2.15.0.html">Apache Beam 2.15.0</a></h3>
<p><i>Aug 22, 2019 •</i></p>
diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml
index ce1f99f..9879203 100644
--- a/website/generated-content/feed.xml
+++ b/website/generated-content/feed.xml
@@ -20,6 +20,103 @@
<generator>Jekyll v3.2.0</generator>
<item>
+ <title>Google Summer of Code '19</title>
+ <description><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>Google Summer of Code was an amazing learning experience for me.
+I contributed to open source, learned about Apache Beam’s internals and worked with the best engineers in the world.</p>
+
+<!--more-->
+
+<h2 id="motivation">Motivation</h2>
+<p>Two of my friends had participated in GSoC in 2018. I was intrigued by their experience.
+The idea of working on open-source software that could potentially be used by developers across the world, while being mentored by the best people in a field was exciting!
+So, I decided to give Google Summer of Code a shot this year.</p>
+
+<h2 id="what-is-google-summer-of-code">What is Google Summer of Code?</h2>
+<p><a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> is a global program hosted by Google focused on introducing students to open source software development.
+Students work on a 3 month programming project with an open source organization during their break from university.</p>
+
+<h2 id="why-apache-beam">Why Apache Beam?</h2>
+<p>While interning at <a href="https://atlan.com/">Atlan</a>, I discovered the field of Data Engineering. I found the challenges and the discussions of the engineers there interesting. While researching for my internship project, I came across the Streaming Systems book. It introduced me to the unified model of Apache Beam for Batch and Streaming Systems, which I was fascinated by.
+I wanted to explore Data Engineering, so for GSoC, I wanted to work on a project in that field. Towards the end of my internship, I started contributing to Apache Airflow(very cool project) and Apache Beam, hoping one of them would participate in GSoC. I got lucky!</p>
+
+<p><a href="https://youtu.be/U2eWLb-LD44">Also, Spotify’s Discover Weekly uses Apache Beam!</a></p>
+
+<h2 id="preparation">Preparation</h2>
+<p>I had already read the <a href="http://streamingsystems.net/">Streaming Systems book</a>. So, I had an idea of the concepts that Beam is built on, but had never actually used Beam.
+Before actually submitting a proposal, I went through a bunch of resources to make sure I had a concrete understanding of Beam.
+I read the <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">Streaming 101</a> and <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102">Streaming 102</a> blogs by Tyler Akidau. They are the perfect introduction to Beam’s unified model for Batch and Streaming.
+In addition, I watched all Beam talks on YouTube. You can find them on the <a href="https://beam.apache.org/documentation/resources/videos-and-podcasts/">Beam Website</a>.
+Beam has really good documentation. The <a href="https://beam.apache.org/documentation/programming-guide/">Programming Guide</a> lays out all of Beam’s concepts really well. <a href="https://beam.apache.org/documentation/execution-model/">Beam’s execution model</a> is also documented well and is a must-read to understand how Beam processes data.
+<a href="https://www.waitingforcode.com/apache-beam">waitingforcode.com</a> also has good blog posts about Beam concepts.
+To get a better sense of the Beam codebase, I played around with it and worked on some PRs to understand Beam better and got familiar with the test suite and workflows.</p>
+
+<h2 id="gsoc-journey">GSoC Journey</h2>
+<p>GSoC has 2 phases. The first is the Community Bonding period in which students get familiar with the project and the community. The other is the actual Coding Period in which students work on their projects. Since the Coding Period has three evaluations spaced out by a month, I divided my project into three parts focusing on the implementation, tests, and documentation or improvements.</p>
+
+<h3 id="project">Project</h3>
+<p>My project(<a href="https://issues.apache.org/jira/browse/BEAM-6611">BEAM-6611</a>) added support for File Loads method of inserting data into BigQuery for streaming pipelines. It builds on PR - <a href="https://github.com/apache/beam/pull/7655">#7655</a> for <a href="https://issues.apache.org/jira/browse/BEAM-6553">BEAM-6553</a> that added support in the Python SDK for writing to BigQuery using File Loads method [...]
+You can find my proposal <a href="https://docs.google.com/document/d/15Peyd3Z_wu5rvGWw8lMLpZuTyyreM_JOAEFFWvF97YY/edit?usp=sharing">here</a>.</p>
+
+<h3 id="community-bonding">Community Bonding</h3>
+<p>When GSoC started, my semester end exams had not yet finished. As a result, I couldn’t get much done. I worked on three PTransforms for the Python SDK - Latest, WithKeys and Reify.</p>
+
+<h3 id="coding-period-i">Coding Period I</h3>
+<p>In this period, I wrote some Integration Tests for the BigQuery sink using Streaming Inserts in streaming mode. I worked on a failing integration test for my project. I also finished the implementation of my project. But, one PostCommit test didn’t pass. I realized that the matcher for the Integration Test that queried BigQuery for the results was intended to be used in Batch mode. So, I wrote a version of the matcher to work in streaming mode.</p>
+
+<h3 id="coding-period-ii">Coding Period II</h3>
+<p>Even after I had added the matcher for streaming mode, the PostComit tests did not pass. A test was being run even though it was not specified. I isolated the failure to a <a href="https://nose.readthedocs.io/en/latest/doc_tests/test_multiprocess/multiprocess.html#other-differences-in-test-running">limitation</a> of the multiprocess plugin for <a href="https://nose.readthedocs.io/en/latest/">nose(a Python test framework)</a> due to whi [...]
+I also worked on small issues related to testing.</p>
+
+<p>This period was marked by a few exciting events:</p>
+<ul>
+ <li>Ending up in the top #100 contributors to apache/beam.</li>
+ <li>My first ever PR Review on an open source project.</li>
+</ul>
+
+<p><img src="https://pbs.twimg.com/media/D_XNSC-UIAUmswG?format=png&amp;name=small" alt="Weird flex but ok" /></p>
+
+<h3 id="coding-period-iii">Coding Period III</h3>
+<p>This was the final coding period before the program ended. Since my project was merged earlier than expected, my mentor suggested another issue(<a href="https://issues.apache.org/jira/browse/BEAM-7742">BEAM-7742</a>) in the same area - BigQueryIO, that I found interesting. So, I worked on partitioning written files in BigQuery to ensure that all load jobs triggered adhere to the load job size limitations specified for BigQuery.
+While working on my project, I was using a pipeline that uses PubSub as a source and BigQuery as a sink to validate my changes. My mentor suggested we add them to the Beam test suite as it would be the ultimate test for BigQueryIO. I also worked on adding this test to Beam.</p>
+
+<p>You can find the list of PRs I worked on <a href="https://github.com/apache/beam/pulls?utf8=%E2%9C%93&amp;q=is%3Apr+author%3Attanay">here</a>.</p>
+
+<h2 id="conclusion">Conclusion</h2>
+<p>GSoC has been a lesson in discipline and goal-setting for me. Deciding what I wanted to work on and how much I wanted to get done each week was an important lesson.
+I had never worked remotely, so this was a new experience. Although I struggled with it initially, I appreciate the flexibility that it comes with.
+I also had a lot of fun learning about Apache Beam’s internals, and other tools in the same ecosystem.
+This was also the first time I had written code with a test-first approach.</p>
+
+<p>I thank my mentor - Pablo Estrada, Apache Beam, The Apache Software Foundation and Google Summer of Code for this opportunity. I am also grateful to my mentor for helping me with everything I needed and more, and the Apache Beam community for being supportive and encouraging.</p>
+
+<p>With the right effort, perseverance, conviction, and a plan, anything is possible. Anything.</p>
+</description>
+ <pubDate>Wed, 04 Sep 2019 01:00:01 -0700</pubDate>
+ <link>https://beam.apache.org/blog/2019/09/04/gsoc-19.html</link>
+ <guid isPermaLink="true">https://beam.apache.org/blog/2019/09/04/gsoc-19.html</guid>
+
+
+ <category>blog</category>
+
+ <category>gsoc</category>
+
+ </item>
+
+ <item>
<title>Apache Beam 2.15.0</title>
<description><!--
Licensed under the Apache License, Version 2.0 (the "License");
@@ -1214,70 +1311,5 @@ Valentyn Tymofieiev, Xinyu Liu, Yifan Zou, Yueyang Qiu</p>
</item>
- <item>
- <title>Apache Beam is applying to Season of Docs</title>
- <description><!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<p>The Apache Beam community is thrilled to announce its application to the first edition of Season of Docs 2019!</p>
-
-<!--more-->
-
-<p><img src="/images/blog/SoD.png" alt="Season of Docs 2019 flyer" height="455" width="640" /></p>
-
-<p><a href="https://developers.google.com/season-of-docs/">Season of Docs</a> is a unique program that pairs technical writers with open source mentors to contribute to open source. This creates an opportunity to introduce the technical writer to an open source community and provide guidance while the writer works on a real world open source project. We, in the Apache Beam community, would love to take this chance and invite technical writers to collaborate wi [...]
-
-<p>Apache Beam does have help from excellent technical writers, but the documentation needs of the project often exceed their bandwidth. This is why we are excited about this program.</p>
-
-<p>After discussing ideas in the community, we have been able to find mentors, and frame two ideas that we think would be a great fit for an incoming tech writer to tackle. We hope you will find this opportunity interesting - and if you do, please get in touch by emailing the Apache Beam mailing list at <a href="mailto:dev@beam.apache.org">dev@beam.apache.org</a> (you will need to subscribe first by emailing to <a href="mailto:dev-subscribe@beam.apache [...]
-
-<p>The project ideas available in Apache Beam are described below. Please take a look and ask any questions that you may have. We will be very happy to help you get onboarded with the project.</p>
-
-<h3 id="project-ideas">Project ideas</h3>
-
-<p><strong>Deployment of Flink and Spark Clusters for use with Portable Beam</strong></p>
-
-<p>The Apache Beam vision has been to provide a framework for users to write and execute pipelines on the programming language of your choice, and the runner of your choice. As the reality of Beam has evolved towards this vision, the way in which Beam is run on top of runners such as Apache Spark and Apache Flink has changed.</p>
-
-<p>These changes are documented in the wiki and in design documents, and are accessible for Beam contributors; but they are not available in the user-facing documentation. This has been a barrier of adoption for other users of Beam.</p>
-
-<p>This project involves improving the <a href="https://beam.apache.org/documentation/runners/flink/">Flink Runner page</a> to include strategies to deploy Beam on a few different environments: A Kubernetes cluster, a Google Cloud Dataproc cluster, and an AWS EMR cluster. There are other places in the documentation that should be updated in this regard, such as the <a href="https://beam.apache.org/documentation/sdks/python-streaming/">Python st [...]
-
-<p>After working on the Flink Runner, then similar updates should be made to the <a href="https://beam.apache.org/documentation/runners/spark/">Spark Runner page</a>, and the <a href="https://beam.apache.org/get-started/beam-overview/">getting started documentation</a>.</p>
-
-<p><strong>The runner comparison page / capability matrix update</strong></p>
-
-<p>Beam maintains a <a href="https://beam.apache.org/documentation/runners/capability-matrix/">capability matrix</a> to track which Beam features are supported by which set of language SDKs + Runners.
-This project involves a number of <a href="https://issues.apache.org/jira/browse/BEAM-2888">corrections and improvements to the capability matrix</a>; followed by a few larger set of changes, involving:</p>
-
-<ul>
- <li>Plain english summaries for each runner’s support of the Beam model.</li>
- <li>A paragraph-length description of the production-readiness for each runner.</li>
- <li>Comparisons for non-model differences between runners.</li>
- <li>Comparison for support of the portability framework for each runner.</li>
-</ul>
-
-<p>Thank you, and we are looking forward to hearing from you!</p>
-</description>
- <pubDate>Fri, 19 Apr 2019 01:00:01 -0700</pubDate>
- <link>https://beam.apache.org/blog/2019/04/19/season-of-docs.html</link>
- <guid isPermaLink="true">https://beam.apache.org/blog/2019/04/19/season-of-docs.html</guid>
-
-
- <category>blog</category>
-
- </item>
-
</channel>
</rss>
diff --git a/website/generated-content/index.html b/website/generated-content/index.html
index 57b28e2..9eb300f 100644
--- a/website/generated-content/index.html
+++ b/website/generated-content/index.html
@@ -193,6 +193,11 @@ limitations under the License.
</div>
<div class="hero__blog__cards">
+ <a class="hero__blog__cards__card" href="/blog/2019/09/04/gsoc-19.html">
+ <div class="hero__blog__cards__card__title">Google Summer of Code '19</div>
+ <div class="hero__blog__cards__card__date">Sep 4, 2019</div>
+ </a>
+
<a class="hero__blog__cards__card" href="/blog/2019/08/22/beam-2.15.0.html">
<div class="hero__blog__cards__card__title">Apache Beam 2.15.0</div>
<div class="hero__blog__cards__card__date">Aug 22, 2019</div>
@@ -203,11 +208,6 @@ limitations under the License.
<div class="hero__blog__cards__card__date">Jul 31, 2019</div>
</a>
- <a class="hero__blog__cards__card" href="/blog/2019/06/11/looping-timers.html">
- <div class="hero__blog__cards__card__title">Looping timers in Apache Beam</div>
- <div class="hero__blog__cards__card__date">Jun 11, 2019</div>
- </a>
-
</div>
</div>
</div>