travis PR testing going red

[David P Grove <gr...@us.ibm.com>]

Lots of failed PR testing in the last 24 hours.
https://travis-ci.org/apache/incubator-openwhisk/pull_requests

I'm pretty sure at least some of these failures are not actually due to the
PRs themselves.  Some of the failures look like test cases making strong
assumptions about non-concurrent CRUD operations.

What's the best way to unblock PR testing while we deal with the underlying
issue?  Disable test cases?  Something else?

--dave

Re: travis PR testing going red

[Rodric Rabbah <ro...@gmail.com>]

I suggest for failing tests that we create heisenbug issues and investigate
them. Turning off tests shouldn't be an option.

I'm not sure it's entirely CRUD operations that are the issue. For example
in this build
https://travis-ci.org/apache/incubator-openwhisk/builds/308624838 (PR
3017), the failing test does NO CRUD operations. It writes directly to
CouchDb and then queries the db for the records. So here I suspect
clustered couchdb.

We have nearly doubled the number of containers we're deploying in Travis:
2 couchdb instances, 2 kafka instances, 2 controllers, 2 invokers, and 2x+
stem cell containers at start up. This could be putting resource pressure
on the VM that slows things down.

-r