You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ashishdeole (via GitHub)" <gi...@apache.org> on 2023/02/16 11:33:09 UTC
[GitHub] [beam] ashishdeole opened a new issue, #25510: [Bug]: beam (even word count example) on flink ( session mode job submission in flink) results in memory leak when submitting job multiple times
ashishdeole opened a new issue, #25510:
URL: https://github.com/apache/beam/issues/25510
### What happened?
### Pre-requisites
We should have an independent flink cluster running. (either standalone or using kubernetes resource provider). We used flink 1.14.5.
And requirement is to you session mode of flink job submission.
### Steps to reproduce
1. Use WordCount example from beam ( latest stable and even with 2.38.0).
2. Run the wordcount pipeline on the flink cluster.
3. Submit the same multiple times. Based on JVM configuration, after some jobs we get Outofmemoryerror - metaspace.
4. Even if we dont wait for outofmemory and get the heapdump - it can be observed that ChildFirstClassLoader is not getting garbage collected after each job is finished. Leading to memory leakage.
### Analysis
1. When jobs are submitted to flink in session mode, flink uses jobmanager-io-thread for each submitted job using thread pool.
2. Beam PipelineOptionsFactory has threadLocal DefaultDeserializationContext.Impl.
3. DefaultDeserializationContext.Impl is loaded from childfirstclassloader and after job completion as the job manager thread goes back to pool this threadLocal is not removed and leads to memory leak.
PFA heapdump produced from Outofmemory - metaspace after submitting wordcount job 4-5 times ( jobmanager.memory.jvm-metaspace.size: 100mb was deliberately kept to reproduce).
### References
1. https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks
2. https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
3. deserializationcontext was made threadlocal using this [PR ](https://github.com/apache/beam/pull/16680) for a bug BEAM-13782.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: beam (even word count example) on flink ( session mode job submission in flink) results in memory leak when submitting job multiple times [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #25510: [Bug]: beam (even word count example) on flink ( session mode job submission in flink) results in memory leak when submitting job multiple times
URL: https://github.com/apache/beam/issues/25510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ashishdeole commented on issue #25510: [Bug]: beam (even word count example) on flink ( session mode job submission in flink) results in memory leak when submitting job multiple times
Posted by "ashishdeole (via GitHub)" <gi...@apache.org>.
ashishdeole commented on issue #25510:
URL: https://github.com/apache/beam/issues/25510#issuecomment-1442310536
https://issues.apache.org/jira/browse/FLINK-28248 confirms the same observations as above and asserts that it should be fixed in beam java sdk code ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: beam (even word count example) on flink ( session mode job submission in flink) results in memory leak when submitting job multiple times [beam]
Posted by "jinyangzhen (via GitHub)" <gi...@apache.org>.
jinyangzhen commented on issue #25510:
URL: https://github.com/apache/beam/issues/25510#issuecomment-1880535903
Hi beam team, it has been almost one year this issue is still existing. It will impact Flink cluster stability. Could u prioritize the fix to this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] gabrywu commented on issue #25510: [Bug]: beam (even word count example) on flink ( session mode job submission in flink) results in memory leak when submitting job multiple times
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on issue #25510:
URL: https://github.com/apache/beam/issues/25510#issuecomment-1722887430
any updates here? our job always throw OOM: Metaspace, don't find RCA yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org