[jira] [Created] (PHOENIX-3289) Region servers crashing randomly with "Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory"

["Ankit Singhal (JIRA)" <ji...@apache.org>]

Ankit Singhal created PHOENIX-3289:
--------------------------------------

             Summary: Region servers crashing randomly with "Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory"
                 Key: PHOENIX-3289
                 URL: https://issues.apache.org/jira/browse/PHOENIX-3289
             Project: Phoenix
          Issue Type: Bug
            Reporter: Ankit Singhal


for GROUP BY case, we try to keep the data in memory but if we exceed the total memory given for global cache(phoenix.query.maxGlobalMemorySize), then we start spilling the data to the disk. We spill and map them with MappedByteBuffer  and add bloom filter so that they can be accessed faster.

MappedByteBuffer doesn't release memory immediately on close(),  GC needs to collect them when it find such buffers with no references. Though, we are closing the channel and file properly and deleting the file at the end, it's possible that GCs has not run for a while and this map count is increasing.

PFB, parent ticket talking about the memory leak with the FileChannel.map() function.
http://bugs.java.com/view_bug.do?bug_id=4724038

And ,related ticket
http://bugs.java.com/view_bug.do?bug_id=6558368

But now the problems is that we are keeping page size of 4KB only for each mmap, which seems to be very small. I don't know the rationale behind keeping it so low, may be for performance to get a single key all the time but it can result in multiple mmap for even small file , so we can thought of keeping it as 64KB page size or more as default but we need to see the impact on performance.

Workaround to mask the issue, increasing the phoenix.query.maxGlobalMemorySize and increasing vm.max_map_count is a solution for get going 

thanks [~rmaruthiyodan] for reducing the territory of the problem by analyzing VM hostspot error file and detecting the increase in mmap count during GroupBy queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)