You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/11/09 02:08:52 UTC
[GitHub] [hudi] BalaMahesh opened a new issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
BalaMahesh opened a new issue #2236:
URL: https://github.com/apache/hudi/issues/2236
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
We are running HUDI for data ingestion. While starting the application initially we gave the executor and driver heap memory a limit of 2 GB and 2 GB respectively. But after running for few hours, application exited with OOM error. We have bumped up the limit to 3GB and 3GB after this , it only helped to get some more extra time before the error and eventually exited with same error. After this we have configured to 6GB and 6 GB , since then there is no error till this point but the memory usage is increasing linearly with the time . Now it is using 19 GB after running for two days.
**To Reproduce**
Steps to reproduce the behavior:
1. Run Hoodie Delta streamer with JSON KAFKA SOURCE in continuous mode.
**Expected behavior**
Constant memory utilization based on ingestion pattern.
**Environment Description**
* Hudi version : 0.6.1
* Spark version : 2.4.5
* Hive version : 1.2
* Hadoop version : 2.8
* Storage (HDFS/S3/GCS..) : s3a
* Running on Docker? (yes/no) : no
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BalaMahesh commented on issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
Posted by GitBox <gi...@apache.org>.
BalaMahesh commented on issue #2236:
URL: https://github.com/apache/hudi/issues/2236#issuecomment-900464347
Please use the GC as mentioned in the HUDI tuning guide. That solved issue for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] GitSpree23 commented on issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
Posted by GitBox <gi...@apache.org>.
GitSpree23 commented on issue #2236:
URL: https://github.com/apache/hudi/issues/2236#issuecomment-900461103
Hey @BalaMahesh ... I'm facing a similar issue ... The initial load takes way too long & suffers heap space run-outs due to clogged Kafka pipeline. Did the above config work for you?
I'm also looking to consume events from Kafka in batches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BalaMahesh commented on issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
Posted by GitBox <gi...@apache.org>.
BalaMahesh commented on issue #2236:
URL: https://github.com/apache/hudi/issues/2236#issuecomment-900464347
Please use the GC as mentioned in the HUDI tuning guide. That solved issue for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BalaMahesh closed issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
Posted by GitBox <gi...@apache.org>.
BalaMahesh closed issue #2236:
URL: https://github.com/apache/hudi/issues/2236
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] GitSpree23 edited a comment on issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
Posted by GitBox <gi...@apache.org>.
GitSpree23 edited a comment on issue #2236:
URL: https://github.com/apache/hudi/issues/2236#issuecomment-900468799
Ok, thanks. Adding this for reference: [Tuning Guide](https://cwiki.apache.org/confluence/display/HUDI/Tuning+Guide).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #2236:
URL: https://github.com/apache/hudi/issues/2236#issuecomment-724431515
No worries!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] GitSpree23 commented on issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
Posted by GitBox <gi...@apache.org>.
GitSpree23 commented on issue #2236:
URL: https://github.com/apache/hudi/issues/2236#issuecomment-900468799
Ok, thanks. Adding this for reference [Tuning Guide](https://cwiki.apache.org/confluence/display/HUDI/Tuning+Guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BalaMahesh commented on issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
Posted by GitBox <gi...@apache.org>.
BalaMahesh commented on issue #2236:
URL: https://github.com/apache/hudi/issues/2236#issuecomment-723866337
some misunderstanding
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] GitSpree23 edited a comment on issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
Posted by GitBox <gi...@apache.org>.
GitSpree23 edited a comment on issue #2236:
URL: https://github.com/apache/hudi/issues/2236#issuecomment-900468799
Ok, thanks. Adding this for reference: [Tuning Guide](https://cwiki.apache.org/confluence/display/HUDI/Tuning+Guide).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] GitSpree23 commented on issue #2236: [SUPPORT] HUDI throwing GC limit exceed : Out of Memory Error with 4 GB Executor and Drive Memory. Without a limit, Heap memory consumption is growing continuously.
Posted by GitBox <gi...@apache.org>.
GitSpree23 commented on issue #2236:
URL: https://github.com/apache/hudi/issues/2236#issuecomment-900461103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org