Flaky tests

[Todd Lipcon <to...@cloudera.com>]

I've noticed that precommits have been especially flaky the last day or so.
In particular, a troubling assertion keeps popping up in mt-tablet-test on
release builds, seemingly related to compaction.

I'm guessing this isn't a regression, but rather just a different timing
that happens more frequently now that we are running precommits on GCE VMs
instead of AWS. Different instance sizes, disk speeds, etc. The flaky test
detection is still based on data coming from EC2 VMs, so it hasn't marked
mt-tablet-test as especially flaky (
http://dist-test.cloudera.org:8080/test_drilldown?test_name=mt-tablet-test
).

I'll work on looping this one tomorrow, and also see about moving our flaky
test data gatherer to run periodically run on GCE.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Flaky tests

[Todd Lipcon <to...@cloudera.com>]

BTW this particular flakiness appears to be a bug that binglin hit a while
back on a real cluster: https://issues.cloudera.org/browse/KUDU-1131

-Todd

On Tue, Feb 2, 2016 at 12:36 AM, Todd Lipcon <to...@cloudera.com> wrote:

> I've noticed that precommits have been especially flaky the last day or
> so. In particular, a troubling assertion keeps popping up in mt-tablet-test
> on release builds, seemingly related to compaction.
>
> I'm guessing this isn't a regression, but rather just a different timing
> that happens more frequently now that we are running precommits on GCE VMs
> instead of AWS. Different instance sizes, disk speeds, etc. The flaky test
> detection is still based on data coming from EC2 VMs, so it hasn't marked
> mt-tablet-test as especially flaky (
> http://dist-test.cloudera.org:8080/test_drilldown?test_name=mt-tablet-test
> ).
>
> I'll work on looping this one tomorrow, and also see about moving our
> flaky test data gatherer to run periodically run on GCE.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera