Re: File JIRAs for all flaky test failures

[Marcelo Vanzin <va...@cloudera.com>]

Hey all,

I just wanted to bring up Kay's old e-mail about this.

If you see a flaky test during a PR, don't just ask for a re-test.
File a bug so that we know that test is flaky and someone will
eventually take a look at it. A lot of them also make great newbie
bugs.

I've filed a bunch of these in the past months, and every time I look
for the test in jira, there was nothing filed yet. And most of those
ended up fixed. Visibility into these things helps getting them fixed.


On Wed, Feb 15, 2017 at 12:10 PM, Kay Ousterhout
<ka...@gmail.com> wrote:
> Hi all,
>
> I've noticed the Spark tests getting increasingly flaky -- it seems more
> common than not now that the tests need to be re-run at least once on PRs
> before they pass.  This is both annoying and problematic because it makes it
> harder to tell when a PR is introducing new flakiness.
>
> To try to clean this up, I'd propose filing a JIRA *every time* Jenkins
> fails on a PR (for a reason unrelated to the PR).  Just provide a quick
> description of the failure -- e.g., "Flaky test: DagSchedulerSuite" or
> "Tests failed because 250m timeout expired", a link to the failed build, and
> include the "Tests" component.  If there's already a JIRA for the issue,
> just comment with a link to the latest failure.  I know folks don't always
> have time to track down why a test failed, but this it at least helpful to
> someone else who, later on, is trying to diagnose when the issue started to
> find the problematic code / test.
>
> If this seems like too high overhead, feel free to suggest alternative ways
> to make the tests less flaky!
>
> -Kay



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org