You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/08/20 18:01:20 UTC
[beam-site] 02/03: Add dashes to pre-commit and post-commit.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit b9156a0b825da71af402b1830e5bbc6b1d5eb948
Author: Udi Meiri <eh...@google.com>
AuthorDate: Wed Aug 15 09:42:53 2018 -0700
Add dashes to pre-commit and post-commit.
Address other review comments as well.
---
src/_includes/section-menu/contribute.html | 6 +--
src/contribute/index.md | 9 ++--
src/contribute/precommit-policies.md | 22 ++++-----
src/contribute/precommit-triage-guide.md | 78 +++++++++++++++++-------------
src/contribute/testing.md | 38 +++++++--------
5 files changed, 83 insertions(+), 70 deletions(-)
diff --git a/src/_includes/section-menu/contribute.html b/src/_includes/section-menu/contribute.html
index 7a70f62..9ec08b8 100644
--- a/src/_includes/section-menu/contribute.html
+++ b/src/_includes/section-menu/contribute.html
@@ -26,7 +26,7 @@
<ul class="section-nav-list">
<li><a href="{{ site.baseurl }}/contribute/testing/">Testing guide</a></li>
<ul>
- <li><a href="{{ site.baseurl }}/contribute/precommit-triage-guide/">Precommit Slowness Triage Guide</a></li>
+ <li><a href="{{ site.baseurl }}/contribute/precommit-triage-guide/">Pre-commit Slowness Triage Guide</a></li>
</ul>
<li><a href="{{ site.baseurl }}/contribute/ptransform-style-guide/">PTransform style guide</a></li>
<li><a href="{{ site.baseurl }}/contribute/runner-guide/">Runner authoring guide</a></li>
@@ -39,8 +39,8 @@
<li>
<span class="section-nav-list-title">Policies</span>
<ul class="section-nav-list">
- <li><a href="{{ site.baseurl }}/contribute/precommit-policies/">Precommit test policies</a></li>
- <li><a href="{{ site.baseurl }}/contribute/postcommits-policies/">Post-commit tests policies</a></li>
+ <li><a href="{{ site.baseurl }}/contribute/precommit-policies/">Pre-commit test policies</a></li>
+ <li><a href="{{ site.baseurl }}/contribute/postcommits-policies/">Post-commit test policies</a></li>
</ul>
</li>
<li>
diff --git a/src/contribute/index.md b/src/contribute/index.md
index e4dafe7..b202214 100644
--- a/src/contribute/index.md
+++ b/src/contribute/index.md
@@ -143,14 +143,15 @@ To find a committer for your area, look for similar code merges or ask on
Use @mention in the pull request to notify the reviewer.
-The pull request and any changes pushed to it will trigger precommit jobs.
-If a test fails and appears unrelated to your change, you can cause tests
-to be re-run by adding a single line comment on your PR
+The pull request and any changes pushed to it will trigger [pre-commit
+jobs](/contribute/testing/). If a test fails and appears unrelated to your
+change, you can cause tests to be re-run by adding a single line comment on your
+PR
retest this please
There are other trigger phrases for post-commit tests found in
-.testinfra/jenkins, but use these sparingly because postcommit
+.testinfra/jenkins, but use these sparingly because post-commit
tests consume shared development resources.
### Developing with the Python SDK
diff --git a/src/contribute/precommit-policies.md b/src/contribute/precommit-policies.md
index 7261283..d903269 100644
--- a/src/contribute/precommit-policies.md
+++ b/src/contribute/precommit-policies.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Precommit Test Policies"
+title: "Pre-commit Test Policies"
permalink: /contribute/precommit-policies/
section_menu: section-menu/contribute.html
---
@@ -18,40 +18,40 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Precommit test policies
+# Pre-commit test policies
## Definitions
-- Precommit test - Any single test in a precommit test suite.
-- Precommit test suite - A collection of precommit tests that have a common
-denominator. A test suite runs in a single Jenkins job. Currently, suites are
-grouped by SDK languages, e.g., Python, Java, and Go.
+- Pre-commit test - Any single test in a pre-commit test suite.
+- Pre-commit test suite - A collection of pre-commit tests that have a common
+ denominator. A test suite runs in a single Jenkins job. Currently, suites are
+ grouped by SDK languages, e.g., Python, Java, and Go.
## Policies
### Pull Requests
-- A PR must pass precommit tests before being committed to the main Beam repo.
- - The relevant precommit test suites are automatically launched according to
+- A PR must pass pre-commit tests before being committed to the main Beam repo.
+ - The relevant pre-commit test suites are automatically launched according to
PR contents.
### Problems
#### Breakage
-Breakage is when one or more tests in a precommit test suite fails or
+Breakage is when one or more tests in a pre-commit test suite fails or
is flaky (occasionally fails).
- Breakages should be fixed within 8 hours.
#### Slowness
-Slowness is when the total time to run a precommit suite exceeds 30 minutes\*,
+Slowness is when the total time to run a pre-commit suite exceeds 30 minutes\*,
including the time the job spends in the Jenkins queue.
- Slowness should be fixed within 24 hours.
-\* See the [Precommit Slowness Triage
+\* See the [Pre-commit Slowness Triage
Guide](/contribute/precommit-triage-guide/) for a precise definition of slowness
and for information on dealing with slowness.
diff --git a/src/contribute/precommit-triage-guide.md b/src/contribute/precommit-triage-guide.md
index 4fc67a8..e1ba1e0 100644
--- a/src/contribute/precommit-triage-guide.md
+++ b/src/contribute/precommit-triage-guide.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Precommit Slowness Triage Guide"
+title: "Pre-commit Slowness Triage Guide"
permalink: /contribute/precommit-triage-guide/
section_menu: section-menu/contribute.html
---
@@ -18,49 +18,61 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-# Precommit Slowness Triage Guide
+# Pre-commit Slowness Triage Guide
-Beam precommit jobs are suites of tests run automatically on Jenkins build
+Beam pre-commit jobs are suites of tests run automatically on Jenkins build
machines for each pull request (PR) submitted to
[apache/beam](https://github.com/apache/beam). For more information and the
-difference between precommits and postcommits, see
+difference between pre-commits and post-commits, see
[testing](/contribute/testing/).
-## What are fast precommits?
+## What are fast pre-commits?
-Precommit tests are required to pass before a pull request (PR) may be merged.
+Pre-commit tests are required to pass before a pull request (PR) is merged.
When these tests are slow they slow down Beam's development process.
-
-The aim is to have 95% of precommit jobs complete within 30 minutes
+The aim is to have 95% of pre-commit jobs complete within 30 minutes
(failing or passing).
+
Technically, the 95th percentile of running time should be below 30 minutes over
the past 4 weeks, where running time is the duration of time the job spends in
the Jenkins queue + the actual time it spends running.
-## Detemining Slowness
+## Determining Slowness
+
+There are two main signs of slowness:
+
+1. Pre-commit jobs are timing out after 30 minutes. This can be determined from
+ the console log of a job.
+1. Pre-commits aren't timing out, but the total wait time for pre-commit results
+ is >30m. This can be determined using a Jupyter notebook.
-The current method for determining if precommmits are slow is to look at the
-[Jupyter
-notebook](https://github.com/apache/beam/tree/master/.test-infra/jupyter)
-`precommit_job_times.ipynb`.
+### Notebook
+
+The Beam repo contains a [Jupyter
+notebook](https://github.com/apache/beam/tree/master/.test-infra/jupyter) named
+`precommit_job_times.ipynb` that can be used to gather and visualize statistics
+about pre-commit running times.
Run the notebook. It should output a table with running times. The numbers in
the column `totalDurationMinutes_all` and the rows with a `job_name` like `4
weeks 95th` contain the target numbers for determining slowness.
If any of these numbers are above 30, triaging is required.
-### Example
+#### Example
Here's an example table of running times:
-
-In this example, Go precommits are taking approximately 14 minutes, which is
-fast. Java and Python precommits are taking 78 and 32 minutes respectively,
-which is slow. Both Java and Python precommits require triage.
+
+
+(Note that this example was created when pre-commits did not have 30m timeouts.)
+
+In this example, Go pre-commits are taking approximately 14 minutes, which is
+fast. Java and Python pre-commits are taking 78 and 32 minutes respectively,
+which is slow. Both Java and Python pre-commits require triage.
## Triage Process
1. [Search for existing
- issues](https://issues.apache.org/jira/issues/?filter=12344461)
+ issues](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20labels%20%3D%20precommit%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC)
1. Create a new issue if needed: [Apache
JIRA](https://issues.apache.org/jira/issues)
- Project: Beam
@@ -73,48 +85,48 @@ which is slow. Both Java and Python precommits require triage.
## Resolution
-It is expected that slowness is resolved promptly. See [precommit test
+It is important that we quickly fix slow pre-commit tests. See [pre-commit test
policies](/contribute/precommit-policies/) for details.
## Possible Causes and Solutions
-This section lists some starting off points for fixing precommit slowness.
+This section lists some starting points for fixing pre-commit slowness.
-### Jenkins
+### Resource Exhaustion
Have a look at the graphs in the Jupyter notebook. Does the rise in total
duration match the rise in queuing time? If so, the slowness might be unrelated
-to this specific precommit job.
+to this specific pre-commit job.
Example of when total and queuing durations rise and fall together (mostly):
-
+
Since Jenkins machines are a limited resource, other jobs can
-affect precommit queueing times. Try to figure out if other jobs have been
+affect pre-commit queueing times. Try to figure out if other jobs have been
recently slower, increased in frequency, or new jobs have been introduced.
Another option is to look at adding more Jenkins machines.
### Slow individual tests
-Sometimes a precommit job is slowed down due to one or more tests. One way of
+Sometimes a pre-commit job is slowed down due to one or more tests. One way of
determining if this is the case is by looking at individual test timings.
Where to find individual test timings:
-- Look at the `Gradle Build Scan` link on the precommit job's Jenkins page. This
- page will contain individual test timings for Java tests only (2018-08).
-- Look at the `Test Result` link on the precommit job's Jenkins page. This
+- Look at the `Gradle Build Scan` link on the pre-commit job's Jenkins page.
+ This page will contain individual test timings for Java tests only (2018-08).
+- Look at the `Test Result` link on the pre-commit job's Jenkins page. This
should be available for Java and Python tests (2018-08).
Sometimes tests can be made faster by refactoring. A test that spends a lot of
time waiting (such as an integration test) could be made to run concurrently with
the other tests.
-If a test is determined to be too slow to be part of precommit tests, it could
-be removed from precommit and placed in postcommit instead (but it should be in
-postcommit already). In addition, ensure that the code covered by the removed
-test is covered by a unit test in precommit.
+If a test is determined to be too slow to be part of pre-commit tests, it should
+be removed from pre-commit and placed in post-commit instead. In addition,
+ensure that the code covered by the removed test is [covered by a unit test in
+pre-commit](/contribute/postcommits-policies-details/#precommit_for_postcommit).
### Slow integration tests
diff --git a/src/contribute/testing.md b/src/contribute/testing.md
index 301b931..2d847a7 100644
--- a/src/contribute/testing.md
+++ b/src/contribute/testing.md
@@ -32,49 +32,49 @@ repo. This is not possible, however, due to a combination of time and resource
constraints. Running all tests for each PR would take hours or even days using
available resources, which would slow down development considerably.
-Thus tests are split into *precommit* and *postcommit* suites. Precommit is
-fast, while postcommit is comprehensive. (Or at least that's the idea.) As their
-names imply, precommit tests are run on each PR before it is committed, while
-postcommits run periodically against the master branch (i.e. on already
-committed PRs).
+Thus tests are split into *pre-commit* and *post-commit* suites. Pre-commit is
+fast, while post-commit is comprehensive. As their names imply, pre-commit tests
+are run on each PR before it is committed, while post-commits run periodically
+against the master branch (i.e. on already committed PRs).
Beam uses [Jenkins](https://builds.apache.org/view/A-D/view/Beam/) to run
-precommit and postcommit tests.
+pre-commit and post-commit tests.
-### Precommit
+### Pre-commit
-The precommit test suite verifies correctness via two testing tools: unit tests
+The pre-commit test suite verifies correctness via two testing tools: unit tests
and end-to-end (E2E) tests. Unit tests ensure correctness at a basic level,
while WordCount E2E tests are run againsts each supported SDK / runner
combination as a smoke test, to verify that a basic level of functionality
exists.
This combination of tests hits the appropriate tradeoff between a desire for
-short (ideally \<30m) precommit times and a desire to verify that PRs going into
-Beam function in the way in which they are intended.
+short (ideally \<30m) pre-commit times and a desire to verify that PRs going
+into Beam function in the way in which they are intended.
-Precommit jobs are kicked off when a contributor makes a PR against the
+Pre-commit jobs are kicked off when a contributor makes a PR against the
`apache/beam` repository. Job statuses are displayed at the bottom of the PR
page. Clicking on “Details” will open the status page in the selected tool;
-there, test status and output can be viewed.
+there, you can view test status and output.
-### Postcommit
+### Post-commit
-Running in postcommit removes as stringent of a time constraint, which gives us
-the ability to do some more comprehensive testing. In postcommit we have a test
+Running in post-commit removes as stringent of a time constraint, which gives us
+the ability to do some more comprehensive testing. In post-commit we have a test
suite running the ValidatesRunner tests against each supported runner, and
another for running the full set of E2E tests against each runner.
Currently-supported runners are Dataflow, Flink, Spark, and Gearpump, with
others soon to follow. Work is ongoing to enable Flink, Spark, and Gearpump in
-the E2E framework, with full support targeted for end of August 2016. Postcommit
-tests run periodically, with timing defined in their Jenkins configurations.
+the E2E framework, with full support targeted for end of August 2016.
+Post-commit tests run periodically, with timing defined in their Jenkins
+configurations.
-Adding new postcommit E2E tests is generally as easy as adding a \*IT.java file
+Adding new post-commit E2E tests is generally as easy as adding a \*IT.java file
to the repository - Failsafe will notice it and run it - but if you want to do
more interesting things, take a look at
[WordCountIT.java](https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java).
-Postcommit test results can be found in
+Post-commit test results can be found in
[Jenkins](https://builds.apache.org/view/A-D/view/Beam/).
## Testing Types