Running tests faster; use more machine CPU

[David Smiley <ds...@apache.org>]

TLDR: What can we do to make the full test suite complete faster,
especially in a CI-like environment?

After hearing about Ishan's fast machine, I was reminded of the "Crave.io"
service I tried some time ago, which makes it super easy to launch builds
from one's command line on remote resources.  It's smart about Git, what
changes you have locally, etc.  It enables me to be more productive from my
aging personal laptop.  I've been chatting with the CEO of Crave.io about
helping Solr more.  He has generously switched the machine to a larger
beast: 48 cores, ~300-something GB RAM, and some SSD.  I also have a
dedicated account/login and can look at pretty charts of the utilization of
the build machine during a build on a number of dimensions.

./gradlew --max-workers=24 test
Took 11:46.  Using 48 was only slightly faster.

Looking at the CPU utilization chart, the utilization gets to maybe 65%
quickly and then lowers down to nothing by the end of the build.

I am aware Gradle "test" isn't intelligently sharing the test load across
the workers, and that it's apparently hard for them to address this.

Do folks have ideas / recommendations on speeding this up?

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley