[jira] [Reopened] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count

["Michael Stack (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Stack reopened HBASE-23779:
-----------------------------------

Reopening. Was resolved because thought there no elbow room here but have been making progress on this master project off in side issues. Let me bring the work together back here under this issue.

High-level, trying to run w/ more parallelism -- forkcount and the mvn -T -- we ran into limits. Experiments trying to up our forkcount to 0.5C (8 cores on apache jenkins) ran into container memory and host/user ulimit upper-bounds.  Returning fork count down to 0.25C re-stabilized builds (along w/ test fixes). Minimally would at least like to add -T0.25C to maven before resolving this issue again. Maximally, would like to get build running on jenkins at 0.5C (In local tests build takes twice as long if forkcount+ T arg are halved... say from 0.5C to 0.25C).

In local tests, I can run with 1.0C.  On a 40CPU linux machine, it is a bit wobbly.  It was wobbly on big GCE instances but I should revisit. On a 16CPU mac, it seems fine as on a 4CPU VM. For Jenkins, 0.5C might be doable. Lets try again now we know more.

> Up the default fork count to make builds complete faster; make count relative to CPU count
> ------------------------------------------------------------------------------------------
>
>                 Key: HBASE-23779
>                 URL: https://issues.apache.org/jira/browse/HBASE-23779
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>            Reporter: Michael Stack
>            Assignee: Michael Stack
>            Priority: Major
>             Fix For: 3.0.0, 2.3.0
>
>         Attachments: addendum2.patch, test_yetus_934.0.patch
>
>
> Tests take a long time. Our fork count running all tests are conservative -- 1 (small) for first part and 5 for second part (medium and large). Rather than hardcoding we should set the fork count to be relative to machine size. Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.
> Looking up at jenkins, it seems like the boxes are 24 cores... at least going by my random survey. The load reported on a few seems low though this not representative (looking at machine/uptime).
> More parallelism willl probably mean more test failure. Let me take a look see.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)