You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/07/27 21:29:07 UTC
[beam-site] branch asf-site updated (8a7108d -> 6d7fd1a)
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.
from 8a7108d Prepare repository for deployment.
add a278295 Add post-commit tests policies summary page.
add a0c756e Fix links
add 21b087e Remove [TOC]
add 491c456 Update postcommits guides
add 67a2c16 This closes #511
new 6d7fd1a Prepare repository for deployment.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
content/contribute/become-a-committer/index.html | 6 +
content/contribute/committer-guide/index.html | 6 +
content/contribute/dependencies/index.html | 6 +
content/contribute/design-documents/index.html | 6 +
content/contribute/docker-images/index.html | 6 +
content/contribute/eclipse/index.html | 6 +
content/contribute/index.html | 6 +
content/contribute/intellij/index.html | 6 +
content/contribute/portability/index.html | 6 +
.../index.html | 151 ++++++++++-----------
.../index.html | 122 +++++++++--------
.../{eclipse => postcommits-policies}/index.html | 151 ++++++++++-----------
.../contribute/ptransform-style-guide/index.html | 6 +
content/contribute/release-guide/index.html | 6 +
content/contribute/runner-guide/index.html | 6 +
content/contribute/testing/index.html | 105 ++++++++++++++
.../contribute/website-contributions/index.html | 6 +
src/_includes/section-menu/contribute.html | 6 +
src/contribute/postcommits-guides.md | 74 ++++++++++
src/contribute/postcommits-policies-details.md | 93 +++++++++++++
src/contribute/postcommits-policies.md | 82 +++++++++++
src/contribute/testing.md | 81 +++++++++++
22 files changed, 725 insertions(+), 218 deletions(-)
copy content/contribute/{website-contributions => postcommits-guides}/index.html (79%)
copy content/contribute/{website-contributions => postcommits-policies-details}/index.html (76%)
copy content/contribute/{eclipse => postcommits-policies}/index.html (76%)
create mode 100644 src/contribute/postcommits-guides.md
create mode 100644 src/contribute/postcommits-policies-details.md
create mode 100644 src/contribute/postcommits-policies.md
[beam-site] 01/01: Prepare repository for deployment.
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 6d7fd1a3fe225a73e08c32f6ec15e78868252913
Author: Mergebot <me...@apache.org>
AuthorDate: Fri Jul 27 21:29:05 2018 +0000
Prepare repository for deployment.
---
content/contribute/become-a-committer/index.html | 6 +
content/contribute/committer-guide/index.html | 6 +
content/contribute/dependencies/index.html | 6 +
content/contribute/design-documents/index.html | 6 +
content/contribute/docker-images/index.html | 6 +
content/contribute/eclipse/index.html | 6 +
content/contribute/index.html | 6 +
content/contribute/intellij/index.html | 6 +
content/contribute/portability/index.html | 6 +
.../index.html | 151 ++++++++++-----------
.../index.html | 122 +++++++++--------
.../{eclipse => postcommits-policies}/index.html | 151 ++++++++++-----------
.../contribute/ptransform-style-guide/index.html | 6 +
content/contribute/release-guide/index.html | 6 +
content/contribute/runner-guide/index.html | 6 +
content/contribute/testing/index.html | 105 ++++++++++++++
.../contribute/website-contributions/index.html | 6 +
17 files changed, 389 insertions(+), 218 deletions(-)
diff --git a/content/contribute/become-a-committer/index.html b/content/contribute/become-a-committer/index.html
index 2a9bbc0..1ccffc9 100644
--- a/content/contribute/become-a-committer/index.html
+++ b/content/contribute/become-a-committer/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/committer-guide/index.html b/content/contribute/committer-guide/index.html
index 72770da..58e5ed5 100644
--- a/content/contribute/committer-guide/index.html
+++ b/content/contribute/committer-guide/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/dependencies/index.html b/content/contribute/dependencies/index.html
index 57c2c13..a4a306f 100644
--- a/content/contribute/dependencies/index.html
+++ b/content/contribute/dependencies/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/design-documents/index.html b/content/contribute/design-documents/index.html
index eee9c3b..93e2fab 100644
--- a/content/contribute/design-documents/index.html
+++ b/content/contribute/design-documents/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/docker-images/index.html b/content/contribute/docker-images/index.html
index ec15c74..dc3c803 100644
--- a/content/contribute/docker-images/index.html
+++ b/content/contribute/docker-images/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/eclipse/index.html b/content/contribute/eclipse/index.html
index 5545811..6e26d29 100644
--- a/content/contribute/eclipse/index.html
+++ b/content/contribute/eclipse/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/index.html b/content/contribute/index.html
index 580062e..7a5ad79 100644
--- a/content/contribute/index.html
+++ b/content/contribute/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/intellij/index.html b/content/contribute/intellij/index.html
index 449b20f..f35bbfa 100644
--- a/content/contribute/intellij/index.html
+++ b/content/contribute/intellij/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/portability/index.html b/content/contribute/portability/index.html
index f0f247d..09d0e57 100644
--- a/content/contribute/portability/index.html
+++ b/content/contribute/portability/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/website-contributions/index.html b/content/contribute/postcommits-guides/index.html
similarity index 79%
copy from content/contribute/website-contributions/index.html
copy to content/contribute/postcommits-guides/index.html
index 7c52811..be19269 100644
--- a/content/contribute/website-contributions/index.html
+++ b/content/contribute/postcommits-guides/index.html
@@ -28,7 +28,7 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Beam Website Contributions</title>
+ <title>Post-commit tests processes guides</title>
<meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
@@ -39,7 +39,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/contribute/website-contributions/" data-proofer-ignore>
+ <link rel="canonical" href="https://beam.apache.org/contribute/postcommits-guides/" data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
<script>
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
@@ -189,9 +195,9 @@
<ul class="nav">
- <li><a href="#one-time-setup">One-time Setup</a></li>
- <li><a href="#working-on-your-change">Working on your change</a></li>
- <li><a href="#committing-website-changes-committers-only">Committing website changes (committers only)</a></li>
+ <li><a href="#find_specialist">Finding someone to triage a post-commit test failure</a></li>
+ <li><a href="#rollback">Rolling back a commit</a></li>
+ <li><a href="#disabling">Disabling a failing test</a></li>
</ul>
@@ -212,79 +218,70 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-<h1 id="website-contributions">Website Contributions</h1>
-
-<p>Contributions to the website require the same steps as contributions to the
-main repository, so start with the instructions for <a href="/contribute/contribution-guide">Contributing
-Code</a>.</p>
-
-<p>The Beam website is in the <a href="https://github.com/apache/beam-site">Beam Site GitHub
-mirror</a> repository in the <code class="highlighter-rouge">asf-site</code>
-branch (<em>not</em> <code class="highlighter-rouge">master</code>).</p>
-
-<p>Issues are tracked in the
-<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20website">website</a>
-component in JIRA.</p>
-
-<h2 id="one-time-setup">One-time Setup</h2>
-
-<p>The <a href="https://github.com/apache/beam-site/blob/asf-site/README.md">README file</a>
-in the website repository has more information on how to set up the required
-dependencies for your development environment.</p>
-
-<p>The general guidelines for cloning a repository can be adjusted to use the
-<code class="highlighter-rouge">asf-site</code> branch of <code class="highlighter-rouge">beam-site</code>:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ git clone -b asf-site https://github.com/apache/beam-site.git
-$ cd beam-site
-$ git remote add <GitHub_user> git@github.com:<GitHub_user>/beam-site.git
-$ git fetch --all
-$ git checkout -b <my-branch> origin/asf-site
-</code></pre>
-</div>
-
-<h2 id="working-on-your-change">Working on your change</h2>
-
-<p>While you are working on your pull request, you can test and develop live by
-running the following command in the root folder of the website:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ bundle exec jekyll serve --incremental
-</code></pre>
-</div>
-
-<p>Jekyll will start a webserver on port 4000. As you make changes to the content,
-Jekyll will rebuild it automatically.</p>
-
-<p>In addition, you can run the tests to valid your links using:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ bundle exec rake test
-</code></pre>
-</div>
-
-<p>Both of these commands will cause the <code class="highlighter-rouge">content/</code> directory to be generated.
-Merging autogenerated content can get tricky, so please leave this directory
-out of your commits and pull request by doing:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>$ git checkout -- content
-</code></pre>
-</div>
-
-<p>When you are ready, submit a pull request using the <a href="https://github.com/apache/beam-site">Beam Site GitHub
-mirror</a>, including the JIRA issue as
-usual.</p>
-
-<p>During review, committers will patch in your PR, generate the static
-<code class="highlighter-rouge">content/</code>, and review the changes.</p>
-
-<h2 id="committing-website-changes-committers-only">Committing website changes (committers only)</h2>
-
-<p>We have a bot that runs tests and merges changes to the website.
-When the pull request is fully ready to be merged, a committer
-can comment</p>
+<h1 id="post-commit-test-task-guides">Post-commit test task guides</h1>
+
+<p>These guides provide steps for common post-commit test failure tasks.</p>
+
+<h2 id="find_specialist">Finding someone to triage a post-commit test failure</h2>
+
+<p>To find the proper person to triage a test failure, you can use these
+suggestions:</p>
+
+<ol>
+ <li>If you can triage it yourself, go for it.</li>
+ <li>Look at the GitHub blame for the files with problematic code.</li>
+ <li>Ask in the <a href="https://the-asf.slack.com/messages/C9H0YNP3P/apps/A0F7VRFKN/">Beam Slack chat</a>.</li>
+ <li>Write to the dev list: dev@beam.apache.org</li>
+</ol>
+
+<h2 id="rollback">Rolling back a commit</h2>
+
+<p>Rolling back is usually the fastest way to fix a failing test. However it is
+is often inconvenient for the original author. To help the author fix the
+issue, follow these steps when you rollback someone’s change.</p>
+
+<ol>
+ <li>Rollback the PR.</li>
+ <li>Create a JIRA issue that contains the following information:
+ <ul>
+ <li>the reason for the rollback</li>
+ <li>a link to the test failure’s JIRA issue</li>
+ <li>triage information</li>
+ <li>any other relevant details</li>
+ </ul>
+ </li>
+ <li>Assign the new JIRA issue to the original PR author.</li>
+ <li>Consider re-opening the JIRA issue associated with the original PR (if
+there is one).</li>
+ <li>Send a notification email with information about the rollback, links to the
+original PR and the rollback PR, and the reasons for the rollback to:
+ <ul>
+ <li>dev@beam.apache.org</li>
+ <li>the original PR author and the committer of the PR</li>
+ </ul>
+ </li>
+ <li>Close the test failure JIRA issue. Your work is done here!</li>
+</ol>
+
+<h2 id="disabling">Disabling a failing test</h2>
+
+<p>If a test fails, our first priority is to rollback the problematic code and fix
+the issue. However, if both: rollback and fix will take awhile to implement, it
+is safer to temporarily disable the test until the fix is ready.</p>
+
+<p>Use caution when deciding to disable a test. When tests are disabled,
+contributors are no longer developing on top of fully tested code. If you decide
+to disable a test, use the following guidelines:</p>
+
+<ul>
+ <li>Notify the dev@beam.apache.org mailing list. Describe the problem and let
+everyone know which test you are disabling.</li>
+ <li>Implement the fix and get the test back online as soon as possible.</li>
+</ul>
-<div class="highlighter-rouge"><pre class="highlight"><code>@asfgit merge
-</code></pre>
-</div>
+<p>While the test is disabled, contributors should not push code to the failing
+test’s coverage area. The code area is not properly tested until you fix the
+test.</p>
</div>
diff --git a/content/contribute/website-contributions/index.html b/content/contribute/postcommits-policies-details/index.html
similarity index 76%
copy from content/contribute/website-contributions/index.html
copy to content/contribute/postcommits-policies-details/index.html
index 7c52811..b91f5e2 100644
--- a/content/contribute/website-contributions/index.html
+++ b/content/contribute/postcommits-policies-details/index.html
@@ -28,7 +28,7 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Beam Website Contributions</title>
+ <title>Post-commit policies details</title>
<meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
@@ -39,7 +39,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/contribute/website-contributions/" data-proofer-ignore>
+ <link rel="canonical" href="https://beam.apache.org/contribute/postcommits-policies-details/" data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
<script>
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
@@ -189,9 +195,11 @@
<ul class="nav">
- <li><a href="#one-time-setup">One-time Setup</a></li>
- <li><a href="#working-on-your-change">Working on your change</a></li>
- <li><a href="#committing-website-changes-committers-only">Committing website changes (committers only)</a></li>
+ <li><a href="#rollback_first">Rollback first</a></li>
+ <li><a href="#failing_test_is_critical_bug">A failing test is a critical/P1 issue</a></li>
+ <li><a href="#flake_is_failing">A flaky test is a critical/P1 issue</a></li>
+ <li><a href="#remove_flake">Flaky tests must be fixed or removed</a></li>
+ <li><a href="#precommit_for_postcommit">Add new pre-commit tests as part of a post-commit fix</a></li>
</ul>
@@ -212,79 +220,75 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-<h1 id="website-contributions">Website Contributions</h1>
-
-<p>Contributions to the website require the same steps as contributions to the
-main repository, so start with the instructions for <a href="/contribute/contribution-guide">Contributing
-Code</a>.</p>
+<h1 id="post-commit-policies-details">Post-commit policies details</h1>
-<p>The Beam website is in the <a href="https://github.com/apache/beam-site">Beam Site GitHub
-mirror</a> repository in the <code class="highlighter-rouge">asf-site</code>
-branch (<em>not</em> <code class="highlighter-rouge">master</code>).</p>
+<p>A post-commit test failure means that there is a bug in the code. The longer the
+bug exists, the harder it is to fix it due to ongoing code contributions. As a
+result, we want to fix bugs quickly. The Beam community’s post-commit test
+policies help keep our code and test results in a good state.</p>
-<p>Issues are tracked in the
-<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20website">website</a>
-component in JIRA.</p>
+<h2 id="rollback_first">Rollback first</h2>
-<h2 id="one-time-setup">One-time Setup</h2>
+<p>Beam uses a “rollback first” approach: the first action to resolve a test
+failure is to rollback the culprit code change. The two main benefits of this
+approach are short implementation time and high reliability. When we rollback
+first, we quickly return to a previously verified good state.</p>
-<p>The <a href="https://github.com/apache/beam-site/blob/asf-site/README.md">README file</a>
-in the website repository has more information on how to set up the required
-dependencies for your development environment.</p>
+<p>At a high level, this approach consists of the following steps:</p>
-<p>The general guidelines for cloning a repository can be adjusted to use the
-<code class="highlighter-rouge">asf-site</code> branch of <code class="highlighter-rouge">beam-site</code>:</p>
+<ol>
+ <li>Revert the culprit commit.</li>
+ <li>Re-run the post-commit tests to verify the tests pass.</li>
+ <li>Push the revert commit.</li>
+</ol>
-<div class="highlighter-rouge"><pre class="highlight"><code>$ git clone -b asf-site https://github.com/apache/beam-site.git
-$ cd beam-site
-$ git remote add <GitHub_user> git@github.com:<GitHub_user>/beam-site.git
-$ git fetch --all
-$ git checkout -b <my-branch> origin/asf-site
-</code></pre>
-</div>
+<p>For background on this policy, see the
+<a href="https://lists.apache.org/thread.html/3bb4aa777751da2e2d7e22666aa6a2e18ae31891cb09d91718b75e74@%3Cdev.beam.apache.org%3E">mailing list thread</a>
+and <a href="https://docs.google.com/document/d/1sczGwnCvdHiboVajGVdnZL0rfnr7ViXXAebBAf_uQME/edit">design doc</a>.</p>
-<h2 id="working-on-your-change">Working on your change</h2>
+<h2 id="failing_test_is_critical_bug">A failing test is a critical/P1 issue</h2>
-<p>While you are working on your pull request, you can test and develop live by
-running the following command in the root folder of the website:</p>
+<p>It is difficult to properly verify new changes made on top of buggy code. In
+some cases, adding additional code can make the problem worse. To avoid this
+situation, fixing failing tests is our highest priority.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>$ bundle exec jekyll serve --incremental
-</code></pre>
-</div>
+<h2 id="flake_is_failing">A flaky test is a critical/P1 issue</h2>
-<p>Jekyll will start a webserver on port 4000. As you make changes to the content,
-Jekyll will rebuild it automatically.</p>
+<p>Flaky tests are considered failing tests, and fixing a flaky test is a
+critical/P1 issue.</p>
-<p>In addition, you can run the tests to valid your links using:</p>
+<p>Flaky tests are tests that randomly succeed or fail while using the same code
+version. Flaky test failures are one of the most dangerous types of failures
+because they are easy to ignore – another run of the flaky test might pass
+successfully. However, these failures can hide real bugs and flaky tests often
+slowly accumulate. Someone must repeatedly triage the failures, and flaky tests
+are often the hardest ones to fix.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>$ bundle exec rake test
-</code></pre>
-</div>
+<p>Flaky tests do not provide a reliable quality signal, so it is important to
+quickly fix the flakiness. If a fix will take awhile to implement, it is safer
+to disable the test until the fix is ready.</p>
-<p>Both of these commands will cause the <code class="highlighter-rouge">content/</code> directory to be generated.
-Merging autogenerated content can get tricky, so please leave this directory
-out of your commits and pull request by doing:</p>
+<p>Martin Fowler has a good <a href="https://martinfowler.com/articles/nonDeterminism.html">article</a>
+about non-determinism in tests.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>$ git checkout -- content
-</code></pre>
-</div>
+<h2 id="remove_flake">Flaky tests must be fixed or removed</h2>
-<p>When you are ready, submit a pull request using the <a href="https://github.com/apache/beam-site">Beam Site GitHub
-mirror</a>, including the JIRA issue as
-usual.</p>
+<p>Flaky tests do not provide a reliable quality signal, which has a harmful effect
+on all tests and can lead to a loss of trust in our test suite. As a result,
+contributors might start to ignore test failures.</p>
-<p>During review, committers will patch in your PR, generate the static
-<code class="highlighter-rouge">content/</code>, and review the changes.</p>
+<p>We want everyone to trust our tests, so it is important to diligently fix all
+flaky tests. If it is not possible to fix a flaky test, we must remove the test.</p>
-<h2 id="committing-website-changes-committers-only">Committing website changes (committers only)</h2>
+<h2 id="precommit_for_postcommit">Add new pre-commit tests as part of a post-commit fix</h2>
-<p>We have a bot that runs tests and merges changes to the website.
-When the pull request is fully ready to be merged, a committer
-can comment</p>
+<p>Post-commit tests are an important fail-safe, but we want to fail fast. Failing
+fast means that we want to detect bugs in pre-commit tests, and <em>not</em> in
+post-commit tests.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>@asfgit merge
-</code></pre>
-</div>
+<p>When you implement a fix for a post-commit test failure, add a new pre-commit
+test that will detect similar failures in the future. For example, you can
+implement a new unit test that covers a problematic code branch.</p>
</div>
diff --git a/content/contribute/eclipse/index.html b/content/contribute/postcommits-policies/index.html
similarity index 76%
copy from content/contribute/eclipse/index.html
copy to content/contribute/postcommits-policies/index.html
index 5545811..ffe36b1 100644
--- a/content/contribute/eclipse/index.html
+++ b/content/contribute/postcommits-policies/index.html
@@ -28,7 +28,7 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Beam Eclipse Tips</title>
+ <title>Post-commit tests policies</title>
<meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
@@ -39,7 +39,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/contribute/eclipse/" data-proofer-ignore>
+ <link rel="canonical" href="https://beam.apache.org/contribute/postcommits-policies/" data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
<script>
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
@@ -189,11 +195,16 @@
<ul class="nav">
- <li><a href="#eclipse-version">Eclipse version</a></li>
- <li><a href="#initial-setup">Initial setup</a></li>
- <li><a href="#building">Building</a></li>
- <li><a href="#checkstyle">Checkstyle</a></li>
- <li><a href="#code-style">Code Style</a></li>
+ <li><a href="#policies">Policies</a></li>
+ <li><a href="#post-commit-test-failure-scenarios">Post-commit test failure scenarios</a>
+ <ul>
+ <li><a href="#found-failing-test">I found a test failure</a></li>
+ <li><a href="#assigned-failing-test">I was assigned a JIRA issue for a test failure</a></li>
+ <li><a href="#pr-rolled-back">My change was rolled back due to a test failure</a></li>
+ </ul>
+ </li>
+ <li><a href="#useful-links">Useful links</a></li>
+ <li><a href="#references">References</a></li>
</ul>
@@ -214,105 +225,81 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-<h1 id="eclipse-tips">Eclipse Tips</h1>
+<h1 id="post-commit-tests-policies">Post-commit tests policies</h1>
-<blockquote>
- <p>These are best-effort community-contributed tips, and are not…</p>
+<p>Post-commit tests validate that Beam works correctly in a live environment. The
+tests also catch errors that are hard to predict in the design and
+implementation stages</p>
- <ul>
- <li>… guaranteed to work with any particular Eclipse setup.</li>
- <li>… the only or best way to work.</li>
- <li>… an endorsement of Eclipse over alternatives.</li>
- </ul>
-
- <p>Mastering Eclipse is, ultimately, your responsibility.</p>
-</blockquote>
+<p>Even though post-commit tests run after the code is merged into the repository,
+it is important that the tests pass reliably. Jenkins executes post-commit tests
+against the HEAD of the master branch. If post-commit tests fail, there is a
+problem with the HEAD build. In addition, post-commit tests are time consuming
+to run, and it is often hard to triage test failures.</p>
-<h2 id="eclipse-version">Eclipse version</h2>
+<h2 id="policies">Policies</h2>
-<p>Use a recent Eclipse version that includes
-<a href="https://projects.eclipse.org/projects/tools.buildship">Buildship</a> for Gradle
-integration. Currently we recommend Eclipse Oxygen. Start Eclipse with a fresh
-workspace in a separate directory from your checkout.</p>
+<p>To ensure that Beam’s post-commit tests are reliable and healthy, the Beam
+community follows these post-commit test policies:</p>
-<h2 id="initial-setup">Initial setup</h2>
+<ul>
+ <li><a href="/contribute/postcommits-policies-details/index.html#rollback_first">Rollback first</a></li>
+ <li><a href="/contribute/postcommits-policies-details/index.html#failing_test_is_critical_bug">A failing test is a critical bug</a></li>
+ <li><a href="/contribute/postcommits-policies-details/index.html#flake_is_failing">A flaky test is a critical bug</a></li>
+ <li><a href="/contribute/postcommits-policies-details/index.html#remove_flake">Flaky tests must either be fixed or removed</a></li>
+ <li><a href="/contribute/postcommits-policies-details/index.html#precommit_for_postcommit">Fixes for post-commit failures should include a corresponding new pre-commit test</a></li>
+</ul>
-<p>Before setting up Eclipse integration, verify that you can successfully build
-from the commandline by building a sample package:</p>
+<h2 id="post-commit-test-failure-scenarios">Post-commit test failure scenarios</h2>
-<div class="highlighter-rouge"><pre class="highlight"><code>./gradlew :beam-examples-java:build
-</code></pre>
-</div>
+<p>When a post-commit test fails, follow the provided steps for your situation.</p>
-<p>If you receive any errors, first verify your environment setup:</p>
+<h3 id="found-failing-test">I found a test failure</h3>
<ol>
- <li>If running on Mac or Linux, launch Eclipse from a terminal. This is necessary
-to ensure that Eclipse has proper environment setup from user profile
-scripts, i.e. .bashrc.</li>
- <li>Install <a href="https://marketplace.eclipse.org/content/buildship-gradle-integration">Buildship Gradle
-Integration</a>.
-This will allow importing and interacting with the Gradle build.</li>
- <li>Open the project import wizard in Eclipse via “File” > “Import”.</li>
- <li>From the “Import” screen, select “Gradle” > “Existing Gradle Project”, and click
-Next.</li>
- <li>From the “Import Gradle Project” screen, fill in the Project root directory
-with your local git path, and click Finish.</li>
+ <li>Create a <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20test-failures">JIRA issue</a>
+and assign it to yourself.</li>
+ <li>Do high level triage of the failure.</li>
+ <li><a href="/contribute/postcommits-guides/index.html#find_specialist">Assign the JIRA issue to a relevant person</a>.</li>
</ol>
-<p>Eclipse will scan the project tree and import each as a separate Package.</p>
-
-<p>Verify that your workspace is correctly configured by invoking
-‘beam-runners-direct-java:build’ from the “Gradle Tasks” pane. The build should
-succeed with no errors.</p>
+<h3 id="assigned-failing-test">I was assigned a JIRA issue for a test failure</h3>
-<h2 id="building">Building</h2>
-
-<p>After your Eclipse workspace is properly setup, you will have a “Gradle
-Tasks” window with a set of operations. If you don’t see the pane, open it
-from “Window” > “Show View” > Other.. > “Gradle” > “Gradle Tasks”.</p>
-
-<p>From the “Gradle Tasks” window, you can build any task registered with Gradle.
-For example, if you are working on Kinesis IO, select ‘beam-sdks-java-io-kinesis:build’.</p>
+<ol>
+ <li><a href="/contribute/postcommits-guides/index.html#rollback">Rollback the culprit change</a>.</li>
+ <li>If you determine that rollback will take longer than 8 hours, <a href="/contribute/postcommits-guides/index.html#disabling">disable the
+test temporarily</a> while you rollback or create a
+fix.</li>
+</ol>
-<h2 id="checkstyle">Checkstyle</h2>
+<blockquote>
+ <p>Note: Rollback is always the first course of action. If a fix is trivial,
+open a pull request with the proposed fix while doing rollback.</p>
+</blockquote>
-<p>Eclipse supports checkstyle within the IDE using the Checkstyle plugin.</p>
+<h3 id="pr-rolled-back">My change was rolled back due to a test failure</h3>
<ol>
- <li>Install the <a href="https://marketplace.eclipse.org/content/checkstyle-plug">Checkstyle
-plugin</a>.</li>
- <li>Configure the Checkstyle plugin by going to Preferences -> Checkstyle.
- <ol>
- <li>Click “New…”</li>
- <li>Select “External Configuration File” for type</li>
- <li>Click “Browse…” and select
-<code class="highlighter-rouge">sdks/java/build-tools/src/main/resources/beam/checkstyle.xml</code></li>
- <li>Enter “Beam Checks” under “Name:”</li>
- <li>Click “OK”, then “OK”</li>
- </ol>
- </li>
+ <li>Look at the JIRA issue to find the reason for the rollback.</li>
+ <li>Fix your code and re-run the post-commit tests.</li>
+ <li>Implement new pre-commit tests that will catch similar bugs before future
+code is merged into the repository.</li>
+ <li>Open a new PR that contains your fix and the new pre-commit tests.</li>
</ol>
-<h2 id="code-style">Code Style</h2>
+<h2 id="useful-links">Useful links</h2>
+
+<ul>
+ <li><a href="/contribute/testing/index.html#best_practices">Best practices for writing tests</a></li>
+</ul>
-<p>Eclipse supports code styles within the IDE. Use one or both of the following
-to ensure your code style matches the project’s checkstyle enforcement.</p>
+<h2 id="references">References</h2>
<ol>
- <li>The simplest way to have uniform code style is to use the <a href="https://github.com/google/google-java-format#eclipse">Google
-Java Format plugin</a></li>
- <li>You can also configure Eclipse to use <code class="highlighter-rouge">beam-codestyle.xml</code>
- <ol>
- <li>Go to Preferences -> Java -> Code Style -> Formatter</li>
- <li>Click “Import…” and select
-<code class="highlighter-rouge">sdks/java/build-tools/src/main/resources/beam/beam-codestyle.xml</code></li>
- <li>Click “Apply” and “OK”</li>
- </ol>
- </li>
+ <li><a href="https://lists.apache.org/thread.html/3bb4aa777751da2e2d7e22666aa6a2e18ae31891cb09d91718b75e74@%3Cdev.beam.apache.org%3E">Keeping post-commit tests green</a>
+mailing list proposal thread.</li>
</ol>
-
</div>
</div>
<!--
diff --git a/content/contribute/ptransform-style-guide/index.html b/content/contribute/ptransform-style-guide/index.html
index c614f6a..0fc122a 100644
--- a/content/contribute/ptransform-style-guide/index.html
+++ b/content/contribute/ptransform-style-guide/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/release-guide/index.html b/content/contribute/release-guide/index.html
index 198d55a..06d0a3b 100644
--- a/content/contribute/release-guide/index.html
+++ b/content/contribute/release-guide/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/runner-guide/index.html b/content/contribute/runner-guide/index.html
index c410582..1639ce1 100644
--- a/content/contribute/runner-guide/index.html
+++ b/content/contribute/runner-guide/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
diff --git a/content/contribute/testing/index.html b/content/contribute/testing/index.html
index 67584ee..85a50d8 100644
--- a/content/contribute/testing/index.html
+++ b/content/contribute/testing/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">
@@ -217,6 +223,14 @@
<li><a href="#api-surface-testing">API Surface testing</a></li>
</ul>
</li>
+ <li><a href="#best_practices">Best practices for writing tests</a>
+ <ul>
+ <li><a href="#aim-for-one-failure-path">Aim for one failure path</a></li>
+ <li><a href="#avoid-non-deterministic-code">Avoid non-deterministic code</a></li>
+ <li><a href="#use-descriptive-test-names">Use descriptive test names</a></li>
+ <li><a href="#use-a-pre-commit-test-if-possible">Use a pre-commit test if possible</a></li>
+ </ul>
+ </li>
</ul>
@@ -268,6 +282,13 @@ limitations under the License.
<li><a href="#api-surface-testing" id="markdown-toc-api-surface-testing">API Surface testing</a></li>
</ul>
</li>
+ <li><a href="#best_practices" id="markdown-toc-best_practices">Best practices for writing tests</a> <ul>
+ <li><a href="#aim-for-one-failure-path" id="markdown-toc-aim-for-one-failure-path">Aim for one failure path</a></li>
+ <li><a href="#avoid-non-deterministic-code" id="markdown-toc-avoid-non-deterministic-code">Avoid non-deterministic code</a></li>
+ <li><a href="#use-descriptive-test-names" id="markdown-toc-use-descriptive-test-names">Use descriptive test names</a></li>
+ <li><a href="#use-a-pre-commit-test-if-possible" id="markdown-toc-use-a-pre-commit-test-if-possible">Use a pre-commit test if possible</a></li>
+ </ul>
+ </li>
</ul>
<h2 id="overview">Overview</h2>
@@ -811,6 +832,90 @@ classes which reside under <code class="highlighter-rouge">"org[.]apache[.]beam[
of the packages: <code class="highlighter-rouge">org.apache.beam.x</code>, <code class="highlighter-rouge">org.apache.beam.y</code>, <code class="highlighter-rouge">org.apache.beam.z</code>,
nor equal to <code class="highlighter-rouge">Other.class</code>.</p>
+<h2 id="best_practices">Best practices for writing tests</h2>
+
+<p>The following best practices help you to write reliable and maintainable tests.</p>
+
+<h3 id="aim-for-one-failure-path">Aim for one failure path</h3>
+
+<p>An ideal test has one failure path. When you create your tests, minimize the
+possible reasons for a test failure. A developer can debug a problem more
+easily when there are fewer failure paths.</p>
+
+<h3 id="avoid-non-deterministic-code">Avoid non-deterministic code</h3>
+
+<p>Reliable tests are predictable and deterministic. Tests that contain
+non-deterministic code are hard to debug and are often flaky. Non-deterministic
+code includes the use of randomness, time, and multithreading.</p>
+
+<p>To avoid non-deterministic code, mock the corresponding methods or classes.</p>
+
+<h3 id="use-descriptive-test-names">Use descriptive test names</h3>
+
+<p>Helpful test names contain details about your test, such as test parameters and
+the expected result. Ideally, a developer can read the test name and know where
+the buggy code is and how to reproduce the bug.</p>
+
+<p>An easy and effective way to name your methods is to use these three questions:</p>
+
+<ul>
+ <li>What you are testing?</li>
+ <li>What are the parameters of the test?</li>
+ <li>What is the expected result of the test?</li>
+</ul>
+
+<p>For example, consider a scenario where you want to add a test for the
+<code class="highlighter-rouge">Divide</code> method:</p>
+
+<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kt">float</span> <span class="nf">Divide</span><span class="o">(</span><span class="kt">float</span> <span class="n">dividend</span><span class="o">,</span> <span class="kt">float</span> <span class="n">divisor</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">dividend</span> <span class="o">/</span> <span class="n">divisor</span><span class="o">;</span>
+<span class="o">}</span>
+
+<span class="o">...</span>
+
+<span class="nd">@Test</span>
+<span class="kt">void</span> <span class="o"><--</span><span class="n">TestMethodName</span><span class="o">-->()</span> <span class="o">{</span>
+ <span class="n">assertThrows</span><span class="o">(</span><span class="n">Divide</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="mi">0</span><span class="o">))</span>
+<span class="o">}</span>
+</code></pre>
+</div>
+
+<p>If you use a simple test name, such as <code class="highlighter-rouge">testDivide()</code>, you are missing important
+information such as the expected action, parameter information, and expected
+test result. As a result, triaging a test failure requires you to look at the
+test implementation to see what the test does.</p>
+
+<p>Instead, use a name such as <code class="highlighter-rouge">invokingDivideWithDivisorEqualToZeroThrowsException()</code>,
+which specifies:</p>
+
+<ul>
+ <li>the expected action of the test (<code class="highlighter-rouge">invokingDivide</code>)</li>
+ <li>details about important parameters (the divisor is zero)</li>
+ <li>the expected result (the test throws an exception)</li>
+</ul>
+
+<p>If this test fails, you can look at the descriptive test name to find the most
+probable cause of the failure. In addition, test frameworks and test result
+dashboards use the test name when reporting test results. Descriptive names
+enable contributors to look at test suite results and easily see what
+features are failing.</p>
+
+<p>Long method names are not a problem for test code. Test names are rarely used
+(usually when you triage and debug), and when you do need to look at a
+test, it is helpful to have descriptive names.</p>
+
+<h3 id="use-a-pre-commit-test-if-possible">Use a pre-commit test if possible</h3>
+
+<p>Post-commit tests validate that Beam works correctly in broad variety of
+scenarios. The tests catch errors that are hard to predict in the design and
+implementation stages</p>
+
+<p>However, we often write a test to verify a specific scenario. In this situation,
+it is usually possible to implement the test as a unit test or a component test.
+You can add your unit tests or component tests to the pre-commit test suite, and
+the pre-commit test results give you faster code health feedback during the
+development stage, when a bug is cheap to fix.</p>
+
</div>
</div>
<!--
diff --git a/content/contribute/website-contributions/index.html b/content/contribute/website-contributions/index.html
index 7c52811..a0da1a9 100644
--- a/content/contribute/website-contributions/index.html
+++ b/content/contribute/website-contributions/index.html
@@ -160,6 +160,12 @@
</ul>
</li>
<li>
+ <span class="section-nav-list-title">Policies</span>
+ <ul class="section-nav-list">
+ <li><a href="/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ </ul>
+</li>
+<li>
<span class="section-nav-list-title">Committers</span>
<ul class="section-nav-list">