[jira] [Updated] (SOLR-14857) Parallelize Solr Docker tests

["Chris M. Hostetter (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-14857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris M. Hostetter updated SOLR-14857:
--------------------------------------
    Attachment: SOLR-14857.patch
        Status: Open  (was: Open)

I took a stab at implementing this...

The gradle {{WorkerExecutor}} is that lovely combination of seemingly simple, extremely bare bones in functionality, and has confusingly undocumented limitations.

Most problematic: there doesn't seem to be any way to control the level of parallelization independently of the project level {{org.gradle.workers.max}} setting – and while I typically run with {{org.gradle.workers.max=5}} on my laptop, when each of those workers is spinning up docker containers running full JVMs, that pretty much crushes my box, and causes multiple (docker) test failures – most seem to be because background tasks (like starting solr, or committing documents) don't finish as quickly as the bash test scripts expect.

One thing that can be done fairly easily is to {{await()}} on the {{WorkQueue}} – which can be done after every task is submitted as quick& dirty way to ensure only one test is run at a time, so that's how I implemented this as an optional feature...
{noformat}
# default: run tests one at a time (regardless of `org.gradle.workers.max`) ...
./gradlew testDocker 

# optional: run concurrent tests, up to `org.gradle.workers.max`
./gradlew testDocker -Dorg.gradle.workers.max=3 -Psolr.docker.tests.concurrent=true
{noformat}
As part of this work, I changed how the bash scripts are invoked by gradle so that the stdout/stderr is always written to test case specific files, never to the gradle console. (but if any tests fail, the name of the test case work dir, which includes those files, is mentioned in the failure message)

[~houston] - WDYT?

> Parallelize Solr Docker tests
> -----------------------------
>
>                 Key: SOLR-14857
>                 URL: https://issues.apache.org/jira/browse/SOLR-14857
>             Project: Solr
>          Issue Type: Improvement
>          Components: Docker, Tests
>    Affects Versions: main (9.0)
>            Reporter: Houston Putman
>            Priority: Minor
>         Attachments: SOLR-14857.patch
>
>
> Currently the Solr docker tests are run concurrently. The contexts and image names are all unique, so there is no reason they can't be run in parallel.
> I would imagine that this would be a configuration passed to the gradle task, which already has command line options for specifying tests to skip or tests to run.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org