You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/08/20 18:01:18 UTC
[beam-site] branch mergebot updated (fc0a5bb -> 320da92)
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.
discard fc0a5bb This closes #539
discard d613018 Edit suggestions
discard 2731a23 Update to LTS wording to remove every Nth release clause.
new 6fe36cc Add precommit policies and triage guide.
new b9156a0 Add dashes to pre-commit and post-commit.
new 320da92 This closes #519
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (fc0a5bb)
\
N -- N -- N refs/heads/mergebot (320da92)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
src/_includes/section-menu/contribute.html | 6 +-
src/community/policies.md | 2 +-
src/contribute/index.md | 9 +-
src/contribute/precommit-policies.md | 66 ++++++++++++++
src/contribute/precommit-triage-guide.md | 137 ++++++++++++++++++++++++++++
src/contribute/testing.md | 75 ++++++++-------
src/images/precommit_durations.png | Bin 0 -> 45673 bytes
src/images/precommit_graph_queuing_time.png | Bin 0 -> 25809 bytes
8 files changed, 255 insertions(+), 40 deletions(-)
create mode 100644 src/contribute/precommit-policies.md
create mode 100644 src/contribute/precommit-triage-guide.md
create mode 100644 src/images/precommit_durations.png
create mode 100644 src/images/precommit_graph_queuing_time.png
[beam-site] 03/03: This closes #519
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 320da925a57cd8883322510ad333ef7cbe9448a9
Merge: 7981cfc b9156a0
Author: Mergebot <me...@apache.org>
AuthorDate: Mon Aug 20 18:00:40 2018 +0000
This closes #519
src/_includes/section-menu/contribute.html | 6 +-
src/contribute/index.md | 9 +-
src/contribute/precommit-policies.md | 66 ++++++++++++++
src/contribute/precommit-triage-guide.md | 137 ++++++++++++++++++++++++++++
src/contribute/testing.md | 75 ++++++++-------
src/images/precommit_durations.png | Bin 0 -> 45673 bytes
src/images/precommit_graph_queuing_time.png | Bin 0 -> 25809 bytes
7 files changed, 254 insertions(+), 39 deletions(-)
[beam-site] 01/03: Add precommit policies and triage guide.
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 6fe36cc22a3b67fa0a4d4e635a463646e607577f
Author: Udi Meiri <eh...@google.com>
AuthorDate: Thu Aug 2 18:20:51 2018 -0700
Add precommit policies and triage guide.
Also update some paragraphs regarding precommits and postcommits in the
testing guide.
---
src/_includes/section-menu/contribute.html | 4 +
src/contribute/precommit-policies.md | 66 +++++++++++++++
src/contribute/precommit-triage-guide.md | 125 ++++++++++++++++++++++++++++
src/contribute/testing.md | 51 +++++++-----
src/images/precommit_durations.png | Bin 0 -> 45673 bytes
src/images/precommit_graph_queuing_time.png | Bin 0 -> 25809 bytes
6 files changed, 224 insertions(+), 22 deletions(-)
diff --git a/src/_includes/section-menu/contribute.html b/src/_includes/section-menu/contribute.html
index 07affbc..7a70f62 100644
--- a/src/_includes/section-menu/contribute.html
+++ b/src/_includes/section-menu/contribute.html
@@ -25,6 +25,9 @@
<ul class="section-nav-list">
<li><a href="{{ site.baseurl }}/contribute/testing/">Testing guide</a></li>
+ <ul>
+ <li><a href="{{ site.baseurl }}/contribute/precommit-triage-guide/">Precommit Slowness Triage Guide</a></li>
+ </ul>
<li><a href="{{ site.baseurl }}/contribute/ptransform-style-guide/">PTransform style guide</a></li>
<li><a href="{{ site.baseurl }}/contribute/runner-guide/">Runner authoring guide</a></li>
<li><a href="{{ site.baseurl }}/contribute/portability/">Portability Framework</a></li>
@@ -36,6 +39,7 @@
<li>
<span class="section-nav-list-title">Policies</span>
<ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/contribute/precommit-policies/">Precommit test policies</a></li>
<li><a href="{{ site.baseurl }}/contribute/postcommits-policies/">Post-commit tests policies</a></li>
</ul>
</li>
diff --git a/src/contribute/precommit-policies.md b/src/contribute/precommit-policies.md
new file mode 100644
index 0000000..7261283
--- /dev/null
+++ b/src/contribute/precommit-policies.md
@@ -0,0 +1,66 @@
+---
+layout: section
+title: "Precommit Test Policies"
+permalink: /contribute/precommit-policies/
+section_menu: section-menu/contribute.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Precommit test policies
+
+## Definitions
+
+- Precommit test - Any single test in a precommit test suite.
+- Precommit test suite - A collection of precommit tests that have a common
+denominator. A test suite runs in a single Jenkins job. Currently, suites are
+grouped by SDK languages, e.g., Python, Java, and Go.
+
+## Policies
+
+### Pull Requests
+
+- A PR must pass precommit tests before being committed to the main Beam repo.
+ - The relevant precommit test suites are automatically launched according to
+ PR contents.
+
+### Problems
+
+#### Breakage
+
+Breakage is when one or more tests in a precommit test suite fails or
+is flaky (occasionally fails).
+
+- Breakages should be fixed within 8 hours.
+
+#### Slowness
+
+Slowness is when the total time to run a precommit suite exceeds 30 minutes\*,
+including the time the job spends in the Jenkins queue.
+
+- Slowness should be fixed within 24 hours.
+
+\* See the [Precommit Slowness Triage
+Guide](/contribute/precommit-triage-guide/) for a precise definition of slowness
+and for information on dealing with slowness.
+
+### Problem Resolution
+
+For any problem, the options are, one of:
+
+- Roll back the culprit PR.
+- Roll out a fix within 24 hours.
+- Disable the slow test or feature temporarily (make sure there's a tracking
+ issue to re-enable it).
+
diff --git a/src/contribute/precommit-triage-guide.md b/src/contribute/precommit-triage-guide.md
new file mode 100644
index 0000000..4fc67a8
--- /dev/null
+++ b/src/contribute/precommit-triage-guide.md
@@ -0,0 +1,125 @@
+---
+layout: section
+title: "Precommit Slowness Triage Guide"
+permalink: /contribute/precommit-triage-guide/
+section_menu: section-menu/contribute.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Precommit Slowness Triage Guide
+
+Beam precommit jobs are suites of tests run automatically on Jenkins build
+machines for each pull request (PR) submitted to
+[apache/beam](https://github.com/apache/beam). For more information and the
+difference between precommits and postcommits, see
+[testing](/contribute/testing/).
+
+## What are fast precommits?
+
+Precommit tests are required to pass before a pull request (PR) may be merged.
+When these tests are slow they slow down Beam's development process.
+
+The aim is to have 95% of precommit jobs complete within 30 minutes
+(failing or passing).
+Technically, the 95th percentile of running time should be below 30 minutes over
+the past 4 weeks, where running time is the duration of time the job spends in
+the Jenkins queue + the actual time it spends running.
+
+## Detemining Slowness
+
+The current method for determining if precommmits are slow is to look at the
+[Jupyter
+notebook](https://github.com/apache/beam/tree/master/.test-infra/jupyter)
+`precommit_job_times.ipynb`.
+
+Run the notebook. It should output a table with running times. The numbers in
+the column `totalDurationMinutes_all` and the rows with a `job_name` like `4
+weeks 95th` contain the target numbers for determining slowness.
+If any of these numbers are above 30, triaging is required.
+
+### Example
+Here's an example table of running times:
+
+
+In this example, Go precommits are taking approximately 14 minutes, which is
+fast. Java and Python precommits are taking 78 and 32 minutes respectively,
+which is slow. Both Java and Python precommits require triage.
+
+## Triage Process
+
+1. [Search for existing
+ issues](https://issues.apache.org/jira/issues/?filter=12344461)
+1. Create a new issue if needed: [Apache
+ JIRA](https://issues.apache.org/jira/issues)
+ - Project: Beam
+ - Components: testing, anything else relevant
+ - Label: precommit
+ - Reference this page in the description.
+1. Determine where the slowness is coming from and identify issues. Open
+ additional issues if needed (such as for multiple issues).
+1. Assign the issue as appropriate, e.g., to the test's or PR's author.
+
+## Resolution
+
+It is expected that slowness is resolved promptly. See [precommit test
+policies](/contribute/precommit-policies/) for details.
+
+## Possible Causes and Solutions
+
+This section lists some starting off points for fixing precommit slowness.
+
+### Jenkins
+
+Have a look at the graphs in the Jupyter notebook. Does the rise in total
+duration match the rise in queuing time? If so, the slowness might be unrelated
+to this specific precommit job.
+
+Example of when total and queuing durations rise and fall together (mostly):
+
+
+Since Jenkins machines are a limited resource, other jobs can
+affect precommit queueing times. Try to figure out if other jobs have been
+recently slower, increased in frequency, or new jobs have been introduced.
+
+Another option is to look at adding more Jenkins machines.
+
+### Slow individual tests
+
+Sometimes a precommit job is slowed down due to one or more tests. One way of
+determining if this is the case is by looking at individual test timings.
+
+Where to find individual test timings:
+
+- Look at the `Gradle Build Scan` link on the precommit job's Jenkins page. This
+ page will contain individual test timings for Java tests only (2018-08).
+- Look at the `Test Result` link on the precommit job's Jenkins page. This
+ should be available for Java and Python tests (2018-08).
+
+Sometimes tests can be made faster by refactoring. A test that spends a lot of
+time waiting (such as an integration test) could be made to run concurrently with
+the other tests.
+
+If a test is determined to be too slow to be part of precommit tests, it could
+be removed from precommit and placed in postcommit instead (but it should be in
+postcommit already). In addition, ensure that the code covered by the removed
+test is covered by a unit test in precommit.
+
+### Slow integration tests
+
+Integration test slowdowns may be caused by dependent services.
+
+## References
+
+- [Beam Fast Precommits design doc](https://docs.google.com/document/d/1udtvggmS2LTMmdwjEtZCcUQy6aQAiYTI3OrTP8CLfJM/edit?usp=sharing)
diff --git a/src/contribute/testing.md b/src/contribute/testing.md
index ef0814b..301b931 100644
--- a/src/contribute/testing.md
+++ b/src/contribute/testing.md
@@ -26,30 +26,37 @@ systems at the bottom.
## Testing Scenarios
-With the tools at our disposal, we have a good set of utilities which we can use
-to verify Beam correctness. To ensure an ongoing high quality of code, we use
-precommit and postcommit testing.
+Ideally, all available tests should be run against a pull request (PR) before
+it's allowed to be committed to Beam's [Github](https://github.com/apache/beam)
+repo. This is not possible, however, due to a combination of time and resource
+constraints. Running all tests for each PR would take hours or even days using
+available resources, which would slow down development considerably.
+
+Thus tests are split into *precommit* and *postcommit* suites. Precommit is
+fast, while postcommit is comprehensive. (Or at least that's the idea.) As their
+names imply, precommit tests are run on each PR before it is committed, while
+postcommits run periodically against the master branch (i.e. on already
+committed PRs).
+
+Beam uses [Jenkins](https://builds.apache.org/view/A-D/view/Beam/) to run
+precommit and postcommit tests.
### Precommit
-For precommit testing, Beam uses
-[Jenkins](https://builds.apache.org/view/A-D/view/Beam/) and a code coverage tool
-called [Coveralls](https://coveralls.io/github/apache/beam), hooked up
-to [Github](https://github.com/apache/beam), to ensure that pull
-requests meet a certain quality bar. These precommits verify correctness via two
-of the below testing tools: unit tests (with coverage monitored by Coveralls)
-and E2E tests. We run the full slate of unit tests in precommit, ensuring
-correctness at a basic level, and then run the WordCount E2E test in both batch
-and streaming (coming soon!) against each supported SDK / runner combination as
-a smoke test, to verify that a basic level of functionality exists. We think
-that this hits the appropriate tradeoff between a desire for short (ideally
-\<30m) precommit times and a desire to verify that pull requests going into Beam
-function in the way in which they are intended.
-
-Precommit tests are kicked off when a user makes a Pull Request against the
-`apache/beam` repository and the Jenkins and Coveralls statuses are displayed at
-the bottom of the pull request page. Clicking on “Details” will open the status
-page in the selected tool; there, test status and output can be viewed.
+The precommit test suite verifies correctness via two testing tools: unit tests
+and end-to-end (E2E) tests. Unit tests ensure correctness at a basic level,
+while WordCount E2E tests are run againsts each supported SDK / runner
+combination as a smoke test, to verify that a basic level of functionality
+exists.
+
+This combination of tests hits the appropriate tradeoff between a desire for
+short (ideally \<30m) precommit times and a desire to verify that PRs going into
+Beam function in the way in which they are intended.
+
+Precommit jobs are kicked off when a contributor makes a PR against the
+`apache/beam` repository. Job statuses are displayed at the bottom of the PR
+page. Clicking on “Details” will open the status page in the selected tool;
+there, test status and output can be viewed.
### Postcommit
@@ -87,7 +94,7 @@ To run all unit tests, execute the following command in the ``sdks/python``
subdirectory
```
-python setup.py test [-s apache_beam.package.module.TestClass.test_method]
+$ python setup.py test [-s apache_beam.package.module.TestClass.test_method]
```
We also provide a [tox](https://tox.readthedocs.io/en/latest/) configuration
diff --git a/src/images/precommit_durations.png b/src/images/precommit_durations.png
new file mode 100644
index 0000000..c659677
Binary files /dev/null and b/src/images/precommit_durations.png differ
diff --git a/src/images/precommit_graph_queuing_time.png b/src/images/precommit_graph_queuing_time.png
new file mode 100644
index 0000000..5082943
Binary files /dev/null and b/src/images/precommit_graph_queuing_time.png differ
[beam-site] 02/03: Add dashes to pre-commit and post-commit.
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit b9156a0b825da71af402b1830e5bbc6b1d5eb948
Author: Udi Meiri <eh...@google.com>
AuthorDate: Wed Aug 15 09:42:53 2018 -0700
Add dashes to pre-commit and post-commit.
Address other review comments as well.
---
src/_includes/section-menu/contribute.html | 6 +--
src/contribute/index.md | 9 ++--
src/contribute/precommit-policies.md | 22 ++++-----
src/contribute/precommit-triage-guide.md | 78 +++++++++++++++++-------------
src/contribute/testing.md | 38 +++++++--------
5 files changed, 83 insertions(+), 70 deletions(-)
diff --git a/src/_includes/section-menu/contribute.html b/src/_includes/section-menu/contribute.html
index 7a70f62..9ec08b8 100644
--- a/src/_includes/section-menu/contribute.html
+++ b/src/_includes/section-menu/contribute.html
@@ -26,7 +26,7 @@
<ul class="section-nav-list">
<li><a href="{{ site.baseurl }}/contribute/testing/">Testing guide</a></li>
<ul>
- <li><a href="{{ site.baseurl }}/contribute/precommit-triage-guide/">Precommit Slowness Triage Guide</a></li>
+ <li><a href="{{ site.baseurl }}/contribute/precommit-triage-guide/">Pre-commit Slowness Triage Guide</a></li>
</ul>
<li><a href="{{ site.baseurl }}/contribute/ptransform-style-guide/">PTransform style guide</a></li>
<li><a href="{{ site.baseurl }}/contribute/runner-guide/">Runner authoring guide</a></li>
@@ -39,8 +39,8 @@
<li>
<span class="section-nav-list-title">Policies</span>
<ul class="section-nav-list">
- <li><a href="{{ site.baseurl }}/contribute/precommit-policies/">Precommit test policies</a></li>
- <li><a href="{{ site.baseurl }}/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ <li><a href="{{ site.baseurl }}/contribute/precommit-policies/">Pre-commit test policies</a></li>
+ <li><a href="{{ site.baseurl }}/contribute/postcommits-policies/">Post-commit test policies</a></li>
</ul>
</li>
<li>
diff --git a/src/contribute/index.md b/src/contribute/index.md
index e4dafe7..b202214 100644
--- a/src/contribute/index.md
+++ b/src/contribute/index.md
@@ -143,14 +143,15 @@ To find a committer for your area, look for similar code merges or ask on
Use @mention in the pull request to notify the reviewer.
-The pull request and any changes pushed to it will trigger precommit jobs.
-If a test fails and appears unrelated to your change, you can cause tests
-to be re-run by adding a single line comment on your PR
+The pull request and any changes pushed to it will trigger [pre-commit
+jobs](/contribute/testing/). If a test fails and appears unrelated to your
+change, you can cause tests to be re-run by adding a single line comment on your
+PR
retest this please
There are other trigger phrases for post-commit tests found in
-.testinfra/jenkins, but use these sparingly because postcommit
+.testinfra/jenkins, but use these sparingly because post-commit
tests consume shared development resources.
### Developing with the Python SDK
diff --git a/src/contribute/precommit-policies.md b/src/contribute/precommit-policies.md
index 7261283..d903269 100644
--- a/src/contribute/precommit-policies.md
+++ b/src/contribute/precommit-policies.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Precommit Test Policies"
+title: "Pre-commit Test Policies"
permalink: /contribute/precommit-policies/
section_menu: section-menu/contribute.html
---
@@ -18,40 +18,40 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Precommit test policies
+# Pre-commit test policies
## Definitions
-- Precommit test - Any single test in a precommit test suite.
-- Precommit test suite - A collection of precommit tests that have a common
-denominator. A test suite runs in a single Jenkins job. Currently, suites are
-grouped by SDK languages, e.g., Python, Java, and Go.
+- Pre-commit test - Any single test in a pre-commit test suite.
+- Pre-commit test suite - A collection of pre-commit tests that have a common
+ denominator. A test suite runs in a single Jenkins job. Currently, suites are
+ grouped by SDK languages, e.g., Python, Java, and Go.
## Policies
### Pull Requests
-- A PR must pass precommit tests before being committed to the main Beam repo.
- - The relevant precommit test suites are automatically launched according to
+- A PR must pass pre-commit tests before being committed to the main Beam repo.
+ - The relevant pre-commit test suites are automatically launched according to
PR contents.
### Problems
#### Breakage
-Breakage is when one or more tests in a precommit test suite fails or
+Breakage is when one or more tests in a pre-commit test suite fails or
is flaky (occasionally fails).
- Breakages should be fixed within 8 hours.
#### Slowness
-Slowness is when the total time to run a precommit suite exceeds 30 minutes\*,
+Slowness is when the total time to run a pre-commit suite exceeds 30 minutes\*,
including the time the job spends in the Jenkins queue.
- Slowness should be fixed within 24 hours.
-\* See the [Precommit Slowness Triage
+\* See the [Pre-commit Slowness Triage
Guide](/contribute/precommit-triage-guide/) for a precise definition of slowness
and for information on dealing with slowness.
diff --git a/src/contribute/precommit-triage-guide.md b/src/contribute/precommit-triage-guide.md
index 4fc67a8..e1ba1e0 100644
--- a/src/contribute/precommit-triage-guide.md
+++ b/src/contribute/precommit-triage-guide.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Precommit Slowness Triage Guide"
+title: "Pre-commit Slowness Triage Guide"
permalink: /contribute/precommit-triage-guide/
section_menu: section-menu/contribute.html
---
@@ -18,49 +18,61 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Precommit Slowness Triage Guide
+# Pre-commit Slowness Triage Guide
-Beam precommit jobs are suites of tests run automatically on Jenkins build
+Beam pre-commit jobs are suites of tests run automatically on Jenkins build
machines for each pull request (PR) submitted to
[apache/beam](https://github.com/apache/beam). For more information and the
-difference between precommits and postcommits, see
+difference between pre-commits and post-commits, see
[testing](/contribute/testing/).
-## What are fast precommits?
+## What are fast pre-commits?
-Precommit tests are required to pass before a pull request (PR) may be merged.
+Pre-commit tests are required to pass before a pull request (PR) is merged.
When these tests are slow they slow down Beam's development process.
-
-The aim is to have 95% of precommit jobs complete within 30 minutes
+The aim is to have 95% of pre-commit jobs complete within 30 minutes
(failing or passing).
+
Technically, the 95th percentile of running time should be below 30 minutes over
the past 4 weeks, where running time is the duration of time the job spends in
the Jenkins queue + the actual time it spends running.
-## Detemining Slowness
+## Determining Slowness
+
+There are two main signs of slowness:
+
+1. Pre-commit jobs are timing out after 30 minutes. This can be determined from
+ the console log of a job.
+1. Pre-commits aren't timing out, but the total wait time for pre-commit results
+ is >30m. This can be determined using a Jupyter notebook.
-The current method for determining if precommmits are slow is to look at the
-[Jupyter
-notebook](https://github.com/apache/beam/tree/master/.test-infra/jupyter)
-`precommit_job_times.ipynb`.
+### Notebook
+
+The Beam repo contains a [Jupyter
+notebook](https://github.com/apache/beam/tree/master/.test-infra/jupyter) named
+`precommit_job_times.ipynb` that can be used to gather and visualize statistics
+about pre-commit running times.
Run the notebook. It should output a table with running times. The numbers in
the column `totalDurationMinutes_all` and the rows with a `job_name` like `4
weeks 95th` contain the target numbers for determining slowness.
If any of these numbers are above 30, triaging is required.
-### Example
+#### Example
Here's an example table of running times:
-
-In this example, Go precommits are taking approximately 14 minutes, which is
-fast. Java and Python precommits are taking 78 and 32 minutes respectively,
-which is slow. Both Java and Python precommits require triage.
+
+
+(Note that this example was created when pre-commits did not have 30m timeouts.)
+
+In this example, Go pre-commits are taking approximately 14 minutes, which is
+fast. Java and Python pre-commits are taking 78 and 32 minutes respectively,
+which is slow. Both Java and Python pre-commits require triage.
## Triage Process
1. [Search for existing
- issues](https://issues.apache.org/jira/issues/?filter=12344461)
+ issues](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20labels%20%3D%20precommit%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC)
1. Create a new issue if needed: [Apache
JIRA](https://issues.apache.org/jira/issues)
- Project: Beam
@@ -73,48 +85,48 @@ which is slow. Both Java and Python precommits require triage.
## Resolution
-It is expected that slowness is resolved promptly. See [precommit test
+It is important that we quickly fix slow pre-commit tests. See [pre-commit test
policies](/contribute/precommit-policies/) for details.
## Possible Causes and Solutions
-This section lists some starting off points for fixing precommit slowness.
+This section lists some starting points for fixing pre-commit slowness.
-### Jenkins
+### Resource Exhaustion
Have a look at the graphs in the Jupyter notebook. Does the rise in total
duration match the rise in queuing time? If so, the slowness might be unrelated
-to this specific precommit job.
+to this specific pre-commit job.
Example of when total and queuing durations rise and fall together (mostly):
-
+
Since Jenkins machines are a limited resource, other jobs can
-affect precommit queueing times. Try to figure out if other jobs have been
+affect pre-commit queueing times. Try to figure out if other jobs have been
recently slower, increased in frequency, or new jobs have been introduced.
Another option is to look at adding more Jenkins machines.
### Slow individual tests
-Sometimes a precommit job is slowed down due to one or more tests. One way of
+Sometimes a pre-commit job is slowed down due to one or more tests. One way of
determining if this is the case is by looking at individual test timings.
Where to find individual test timings:
-- Look at the `Gradle Build Scan` link on the precommit job's Jenkins page. This
- page will contain individual test timings for Java tests only (2018-08).
-- Look at the `Test Result` link on the precommit job's Jenkins page. This
+- Look at the `Gradle Build Scan` link on the pre-commit job's Jenkins page.
+ This page will contain individual test timings for Java tests only (2018-08).
+- Look at the `Test Result` link on the pre-commit job's Jenkins page. This
should be available for Java and Python tests (2018-08).
Sometimes tests can be made faster by refactoring. A test that spends a lot of
time waiting (such as an integration test) could be made to run concurrently with
the other tests.
-If a test is determined to be too slow to be part of precommit tests, it could
-be removed from precommit and placed in postcommit instead (but it should be in
-postcommit already). In addition, ensure that the code covered by the removed
-test is covered by a unit test in precommit.
+If a test is determined to be too slow to be part of pre-commit tests, it should
+be removed from pre-commit and placed in post-commit instead. In addition,
+ensure that the code covered by the removed test is [covered by a unit test in
+pre-commit](/contribute/postcommits-policies-details/#precommit_for_postcommit).
### Slow integration tests
diff --git a/src/contribute/testing.md b/src/contribute/testing.md
index 301b931..2d847a7 100644
--- a/src/contribute/testing.md
+++ b/src/contribute/testing.md
@@ -32,49 +32,49 @@ repo. This is not possible, however, due to a combination of time and resource
constraints. Running all tests for each PR would take hours or even days using
available resources, which would slow down development considerably.
-Thus tests are split into *precommit* and *postcommit* suites. Precommit is
-fast, while postcommit is comprehensive. (Or at least that's the idea.) As their
-names imply, precommit tests are run on each PR before it is committed, while
-postcommits run periodically against the master branch (i.e. on already
-committed PRs).
+Thus tests are split into *pre-commit* and *post-commit* suites. Pre-commit is
+fast, while post-commit is comprehensive. As their names imply, pre-commit tests
+are run on each PR before it is committed, while post-commits run periodically
+against the master branch (i.e. on already committed PRs).
Beam uses [Jenkins](https://builds.apache.org/view/A-D/view/Beam/) to run
-precommit and postcommit tests.
+pre-commit and post-commit tests.
-### Precommit
+### Pre-commit
-The precommit test suite verifies correctness via two testing tools: unit tests
+The pre-commit test suite verifies correctness via two testing tools: unit tests
and end-to-end (E2E) tests. Unit tests ensure correctness at a basic level,
while WordCount E2E tests are run againsts each supported SDK / runner
combination as a smoke test, to verify that a basic level of functionality
exists.
This combination of tests hits the appropriate tradeoff between a desire for
-short (ideally \<30m) precommit times and a desire to verify that PRs going into
-Beam function in the way in which they are intended.
+short (ideally \<30m) pre-commit times and a desire to verify that PRs going
+into Beam function in the way in which they are intended.
-Precommit jobs are kicked off when a contributor makes a PR against the
+Pre-commit jobs are kicked off when a contributor makes a PR against the
`apache/beam` repository. Job statuses are displayed at the bottom of the PR
page. Clicking on “Details” will open the status page in the selected tool;
-there, test status and output can be viewed.
+there, you can view test status and output.
-### Postcommit
+### Post-commit
-Running in postcommit removes as stringent of a time constraint, which gives us
-the ability to do some more comprehensive testing. In postcommit we have a test
+Running in post-commit removes as stringent of a time constraint, which gives us
+the ability to do some more comprehensive testing. In post-commit we have a test
suite running the ValidatesRunner tests against each supported runner, and
another for running the full set of E2E tests against each runner.
Currently-supported runners are Dataflow, Flink, Spark, and Gearpump, with
others soon to follow. Work is ongoing to enable Flink, Spark, and Gearpump in
-the E2E framework, with full support targeted for end of August 2016. Postcommit
-tests run periodically, with timing defined in their Jenkins configurations.
+the E2E framework, with full support targeted for end of August 2016.
+Post-commit tests run periodically, with timing defined in their Jenkins
+configurations.
-Adding new postcommit E2E tests is generally as easy as adding a \*IT.java file
+Adding new post-commit E2E tests is generally as easy as adding a \*IT.java file
to the repository - Failsafe will notice it and run it - but if you want to do
more interesting things, take a look at
[WordCountIT.java](https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java).
-Postcommit test results can be found in
+Post-commit test results can be found in
[Jenkins](https://builds.apache.org/view/A-D/view/Beam/).
## Testing Types