[jira] [Resolved] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread

["Michael Stack (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Stack resolved HBASE-24072.
-----------------------------------
    Fix Version/s: 2.3.0
                   3.0.0
     Release Note: Hadoop hosts have had their ulimit -u raised from 10000 to 30000 (per user, by INFRA). The Docker build container has had its limit raised from 10000 to 12500.  (was: Hadoop hosts have had their ulimit -u raised from 10000 to 30000 (per user). The build container has had its limit raised from 10000 to 12500.)
         Assignee: Michael Stack
       Resolution: Fixed

We don't see these anymore. Two containers running on a single hadoop host seems likely culprit. We shouldn't see this anymore since we won't be upping our parallelism on jenkins builds since we seem to be operating at the edge of whats fair; i.e. max of half CPUs on hadoop host (hadoop hosts run two jenkins executors per host).

> Nightlies reporting OutOfMemoryError: unable to create new native thread
> ------------------------------------------------------------------------
>
>                 Key: HBASE-24072
>                 URL: https://issues.apache.org/jira/browse/HBASE-24072
>             Project: HBase
>          Issue Type: Task
>          Components: test
>            Reporter: Michael Stack
>            Assignee: Michael Stack
>            Priority: Major
>             Fix For: 3.0.0, 2.3.0
>
>         Attachments: 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, print_ulimit.patch
>
>
> Seeing this kind of thing in nightly...
> {code}
> java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
> 	at org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> 	at org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
> {code}
> Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing our docker container. It is set to 20G (The hosts are 48G). Fork count is 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add the maven 4G and we could be over the top.
> Play w/ downing the fork size (in earlier study we didn't seem to need this much RAM when running a fat long test). Let me also take th ms off the mvn allocation to see if that helps.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)