Prevent common causes of flaky tests

[Kirk Lund <kl...@apache.org>]

How to prevent common causes of flakiness in tests:

1. Use the correct timeout
  a. Always use GeodeAwaitility.getTimeout() for all timeouts in tests even
if you think it should be shorter.

2. Use a free port or disable the port instead of using any default ports
  a. Use AvailablePortHelper to get free ports to use instead of using
default ports. Remember: if you didn’t specify or disable the port, it’s
using the default.
  b. [server] Specify --server-port=n when starting servers in a test with
clients. Use --disable-default-server to disable it.
  c. [locator] Specify --port=n when starting locators. (I don’t think this
can be disabled)
  d. [locator] Specify --http-service-port=n when starting locators that
need Pulse/REST/HTTP. Use zero as value to disable it.
  e. [locator] Specify --J=-Dgemfire.jmx-manager-port=n in a test that uses
JMX or GFSH. Use --
J=-Dgemfire.jmx-manager=false to disable it.
  f. If the test needs to use APIs and gemfire.properties instead of Gfsh
switches:
    i. --server-port is specified using CacheServerFactory and CacheServer
APIs
    ii. --port is specified using the Locator and InternalLocator APIs
    iii. --http-service-port is specified using http-service-port
    iv. JMX Manager is configured with jmx-manager-port and jmx-manager

3. Enable only the services the test actually uses
  a. Don’t enable cluster config, REST API, etc if the test doesn’t use the
service.

4. Restart needs to await full-stop before restarting with the same ports
  a. Use GfshStopper for some new handy utilities (introduced by
GEODE-10327).