[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count

["Michael Stack (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027649#comment-17027649 ] 

Michael Stack commented on HBASE-23779:
---------------------------------------

Chatting w/ Sean out on slack:
{code}
saintstack Yesterday at 3:57 PM
Looking at faster builds (prompted by a colleagues' exploration). Was looking to up the default fork count. Our jenkins builds seem to use default. Was going to do a fraction of CPUs. Looking at random build, there are 24 CPUs in the build boxes. Looking at the uptime load, says '2'. Could look around more but was thinking of going from default of 4 forks to 0.5CPU#. Testing locally to make sure pass profile is not distrubted (Our test classification has gotten messy... cleaning that up too). Shout if input. Thanks.
6 replies
busbey  16 hours ago
Remember there are 2 executors per node, so optimistically we get 12 cores
busbey  15 hours ago
And if we land on a node with Hadoop tests then maybe only like half that
saintstack  3 minutes ago
So, 0.5CPU sounds like a good number then as its pretty much what we have now
saintstack  3 minutes ago
... only it has advantage of varying w/ machine cpu count.
saintstack  2 minutes ago
...rather than being hardcoded.
{code}

... so let me down what I have here from .75C to 0.5C so we are close to the settings we already have (The patch has 0.75 -- not 0.5 as I suggest in the comment).

> Up the default fork count to make builds complete faster; make count relative to CPU count
> ------------------------------------------------------------------------------------------
>
>                 Key: HBASE-23779
>                 URL: https://issues.apache.org/jira/browse/HBASE-23779
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>            Reporter: Michael Stack
>            Priority: Major
>
> Tests take a long time. Our fork count running all tests are conservative -- 1 (small) for first part and 5 for second part (medium and large). Rather than hardcoding we should set the fork count to be relative to machine size. Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.
> Looking up at jenkins, it seems like the boxes are 24 cores... at least going by my random survey. The load reported on a few seems low though this not representative (looking at machine/uptime).
> More parallelism willl probably mean more test failure. Let me take a look see.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)