You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "geekboood (via GitHub)" <gi...@apache.org> on 2023/06/22 09:57:31 UTC
[GitHub] [pinot] geekboood opened a new issue, #10960: Large off-heap memory usage?
geekboood opened a new issue, #10960:
URL: https://github.com/apache/pinot/issues/10960
Hi, I deployed Pinot on k8s, and each pod has 32GB memory allocated. The java heap memory is limited to 16GB by using -Xmx16G option.
When I run a query (through Trino) and scan a large json value column (each segment has 300MB data), it allocates too many memory and triggers k8s oom killer. What could be the problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geekboood commented on issue #10960: Large off-heap memory usage?
Posted by "geekboood (via GitHub)" <gi...@apache.org>.
geekboood commented on issue #10960:
URL: https://github.com/apache/pinot/issues/10960#issuecomment-1634674328
@Jackie-Jiang I tried to figure out the memory leak by using pmap to investigate the memory layout. I found that it allocated a lot of anonymous memory, each with 6x MB. After using gdb to dump these memory regions, I found my data for a column.
While reading through the code, I discovered that ColumnValueReader implements the Closeable interface, but DataFetcher put it into a hashmap. Will it automatically call the close method sometime?
If no one invokes these close methods, the direct memory allocated by ChunkReaderContext will not be freed. These are used in VarByteChunkSVForwardIndexReader to decompress string column data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] gortiz commented on issue #10960: Large off-heap memory usage?
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #10960:
URL: https://github.com/apache/pinot/issues/10960#issuecomment-1614239155
Newer images will be built using amazon corretto and microsoft base images and the plan is to try to keep them updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10960: Large off-heap memory usage?
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10960:
URL: https://github.com/apache/pinot/issues/10960#issuecomment-1603753296
When configuring xmx16G, java will use up to 16G heap and up to 16G direct memory, mapped memory is not tracked though. The mapped memory can go over the limit (OS should reclaim mapped memory when it is short on memory, but that might be too late).
One best practice is to not put limit for the memory assuming pinot server is the only pod running on the node
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jbewing commented on issue #10960: Large off-heap memory usage?
Posted by "jbewing (via GitHub)" <gi...@apache.org>.
jbewing commented on issue #10960:
URL: https://github.com/apache/pinot/issues/10960#issuecomment-1604721257
One thing to look out for is that some of the 0.12.0/1 runtime images for pinot are based on an older, deprecated [openjdk:11-jdk-slim docker image](https://hub.docker.com/layers/library/openjdk/11-jdk-slim/images/sha256-d2b6af2093e823ba0cdee4bcd45a905afe3fa054d08bde55b1d850515da69a08?context=explore) which use Java 11.0.16 which has [a C2 compiler bug which results in memory leaks under load](https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-controllers/memory-leak-c2-jit-compiler-java-11.0.16). We found that this led to mysterious OOM's in k8s. Since it's a spike under load, the bug is hard to notice via metrics or direct memory profiles.
Ironically, there's another java runtime (11.0.18) in that image that you can use at `/usr/lib/jvm/java-11-openjdk-amd64` (just remove the openjdk:11-jdk-slim 11.0.16 java from the PATH and update your JAVA_HOME/JAVA_VERSION to point at the other JDK11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geekboood commented on issue #10960: Large off-heap memory usage?
Posted by "geekboood (via GitHub)" <gi...@apache.org>.
geekboood commented on issue #10960:
URL: https://github.com/apache/pinot/issues/10960#issuecomment-1635724282
@Jackie-Jiang I studied the issue further and found that DirectByteBuffer would be freed eventually when the corresponding Java object on the heap is garbage collected. However, since the Java heap pressure is not heavy, it seems like this would take quite a long time. I discovered that there exists a Java option, `-XX:MaxDirectMemorySize`, which can limit the total allocated off-heap memory. The default value is set to the heap's maximum memory, which means that when I set `-Xmx24G`, it could use a total of 48GB of memory and lead to an OOM (out of memory) error. After I set `-XX:MaxDirectMemorySize=4G`, everything seems fine.
However, after setting this option, I encountered an OutOfMemory issue when running a large query. I think explicitly closing the ColumnValueReaders could help with running large queries. For now, I set `deriveNumDocsPerChunkForRawIndex=true` for the large JSON column, and the uncompressed chunk size will be smaller, allowing for smaller DirectByteBuffers to be allocated. Currently, everything is running smoothly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geekboood closed issue #10960: Large off-heap memory usage?
Posted by "geekboood (via GitHub)" <gi...@apache.org>.
geekboood closed issue #10960: Large off-heap memory usage?
URL: https://github.com/apache/pinot/issues/10960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10960: Large off-heap memory usage?
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10960:
URL: https://github.com/apache/pinot/issues/10960#issuecomment-1641553188
@geekboood My suggestion would be setting a smaller heap size. If the pod has 32GB memory limit, then 12GB heap might be a good starting point. It will use up to 12GB heap + 12GB direct memory. We do need to preserve some memory for direct memory because they are also used in netty when sending the responses
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geekboood commented on issue #10960: Large off-heap memory usage?
Posted by "geekboood (via GitHub)" <gi...@apache.org>.
geekboood commented on issue #10960:
URL: https://github.com/apache/pinot/issues/10960#issuecomment-1632644587
> When configuring xmx16G, java will use up to 16G heap and up to 16G direct memory, mapped memory is not tracked though. The mapped memory can go over the limit (OS should reclaim mapped memory when it is short on memory, but that might be too late). One best practice is to not put limit for the memory assuming pinot server is the only pod running on the node
I think this is the reason. Cgroup won't calculate mmap committed memory seperately, so the oom killer will kick in before kernel reclaim these memory.
However, I found another mysterious problem. When I run a same query repeatly (I only have offline table so the data should be consistent), RES memory keeps increasing. I think this might not be right...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10960: Large off-heap memory usage?
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10960:
URL: https://github.com/apache/pinot/issues/10960#issuecomment-1634923286
@geekboood Good findings! Currently we don't explicitly close the `ColumnValueReader`s (there is a TODO in `DataFetcher`). We rely on the `ByteBuffer` finalizer to reclaim the memory.
Can you try triggering a garbage collection and see if these buffers getting released?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org