You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by as...@googlecode.com on 2015/08/12 05:03:02 UTC
Re: Issue 934 in asterixdb: java.lang.OutOfMemoryError: Java heap
space when creating an n-gram index
Comment #2 on issue 934 by jianfeng...@gmail.com:
java.lang.OutOfMemoryError: Java heap space when creating an n-gram index
https://code.google.com/p/asterixdb/issues/detail?id=934
The int array is the pointer array storing the offset to the frame tuples
which is created when the FrameMemoryBuffer is full. If the pointer is that
big then the objects it pointed to could be even bigger.
The source of the problem is that the `frameLimit` parameter that sent to
initialize the `ExternalSortRunGenerator` is too larger.
Can you find out how big is that `frameLimit`?
And what is the `compiler.framesize` and `compiler.sortmemory` in your
settings?
--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings
Re: Issue 934 in asterixdb: java.lang.OutOfMemoryError: Java heap
space when creating an n-gram index
Posted by as...@googlecode.com.
Comment #6 on issue 934 by jianfeng...@gmail.com:
java.lang.OutOfMemoryError: Java heap space when creating an n-gram index
https://code.google.com/p/asterixdb/issues/detail?id=934
While, the OOM exception is an overall effect. Each operator may hold
several hundred of MB, sort operator is the last straw.
--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings
Re: Issue 934 in asterixdb: java.lang.OutOfMemoryError: Java heap
space when creating an n-gram index
Posted by as...@googlecode.com.
Comment #5 on issue 934 by wangs...@gmail.com: java.lang.OutOfMemoryError:
Java heap space when creating an n-gram index
https://code.google.com/p/asterixdb/issues/detail?id=934
Yes. I will try that. 320MB for 6GB java heap was too much, maybe.
--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings
Re: Issue 934 in asterixdb: java.lang.OutOfMemoryError: Java heap
space when creating an n-gram index
Posted by as...@googlecode.com.
Comment #4 on issue 934 by jianfeng...@gmail.com:
java.lang.OutOfMemoryError: Java heap space when creating an n-gram index
https://code.google.com/p/asterixdb/issues/detail?id=934
Yes, it makes sense if the `sortmemory` is 320MB. I guess your tuple is
also very small, then the size of the pointer array is not neglectable. It
will be a very tricky problems if we want to calculate the memory
consumption very precisely, though it is a separate issue.
You can try to lower the `sortmemory` to 120MB or smaller to trigger the
2nd round ExternalSort.
--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings
Re: Issue 934 in asterixdb: java.lang.OutOfMemoryError: Java heap
space when creating an n-gram index
Posted by as...@googlecode.com.
Comment #3 on issue 934 by wangs...@gmail.com: java.lang.OutOfMemoryError:
Java heap space when creating an n-gram index
https://code.google.com/p/asterixdb/issues/detail?id=934
Thanks. Based on the sort memory, it makes sense? However, the problem is
we have five partitions, and an OOM exception happens.
compiler.framesize: 655,360 (650KB)
compiler.sortmemory: 335,544,320 (320MB)
I will try to check the frameLimit.
--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings