You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "suxiaogang223 (via GitHub)" <gi...@apache.org> on 2025/03/05 16:19:21 UTC
[I] [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference [hudi]
suxiaogang223 opened a new issue, #12918:
URL: https://github.com/apache/hudi/issues/12918
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
While debugging a memory leak issue, I noticed a steady increase in the number of BitCaskDiskMap instances through jmap logs. Using VisualVM, I found that a memory leak may exist in BitCaskDiskMap due to a circular reference.
From my investigation, it appears that the memory leak might be caused by a function closure where this refers to BitCaskDiskMap itself. Additionally, since BitCaskDiskMap holds a reference to shutdownThread, this could be preventing proper garbage collection.
I’m not entirely sure whether this implementation indeed leads to a memory leak, but based on my findings, it seems like a possible cause. Could you help confirm if this is an issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #12918: [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference
URL: https://github.com/apache/hudi/issues/12918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12918:
URL: https://github.com/apache/hudi/issues/12918#issuecomment-2707798839
@suxiaogang223 Can you fire a fix for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference [hudi]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on issue #12918:
URL: https://github.com/apache/hudi/issues/12918#issuecomment-2713598056
@danny0405 Hi, can you please review this pr👀? https://github.com/apache/hudi/pull/12956
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12918:
URL: https://github.com/apache/hudi/issues/12918#issuecomment-2702478345
The shutdown hook triggers the `#cleanUp` method where there is no `BitCaskDiskMap` reference.
```java
private void cleanup(boolean isTriggeredFromShutdownHook) {
try {
FileIOUtils.deleteDirectory(diskMapPathFile);
} catch (IOException exception) {
LOG.warn("Error while deleting the disk map directory=" + diskMapPath, exception);
}
if (!isTriggeredFromShutdownHook && shutdownThread != null) {
Runtime.getRuntime().removeShutdownHook(shutdownThread);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference [hudi]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on issue #12918:
URL: https://github.com/apache/hudi/issues/12918#issuecomment-2716873270
> [@suxiaogang223](https://github.com/suxiaogang223) Does doris reuse the merged log reader? For engine like Spark and Flink, no issue around this was reported before.
Doris just call `getAllPartitionPaths()` to get partitions of table. When the hudi table has partition metadata information, hudi will create a merged log reader to read the partition information, and a BitCaskMap will be generated in the process.
https://github.com/apache/doris/blob/5e47a50e2643dd25efd055a46920301df4ad255f/fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiPartitionProcessor.java#L55
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference [hudi]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on issue #12918:
URL: https://github.com/apache/hudi/issues/12918#issuecomment-2706168126
@danny0405 I believe the key issue is that even when we explicitly call the close() function of BitCaskDiskMap, the JVM still won't garbage collect the BitCaskDiskMap class itself due to circular references, which is what we observed.
```java
/**
* Close and cleanup the Map.
*/
public void close() {
cleanup(false);
}
/**
* Cleanup all resources, files and folders.
*/
private void cleanup(boolean isTriggeredFromShutdownHook) {
try {
FileIOUtils.deleteDirectory(diskMapPathFile);
} catch (IOException exception) {
LOG.warn("Error while deleting the disk map directory=" + diskMapPath, exception);
}
if (!isTriggeredFromShutdownHook && shutdownThread != null) {
Runtime.getRuntime().removeShutdownHook(shutdownThread);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference [hudi]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on issue #12918:
URL: https://github.com/apache/hudi/issues/12918#issuecomment-2702656666
- The cleanup method is an **instance method** that implicitly depends on the this pointer
- When cleanup is called from shutdownThread, it holds a reference to this
- Meanwhile, the DiskMap class holds a reference to shutdownThread
This creates a circular reference:
```text
DiskMap (this) -> shutdownThread -> cleanup -> DiskMap (this)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference [hudi]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on issue #12918:
URL: https://github.com/apache/hudi/issues/12918#issuecomment-2708292802
I will try to fix this next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Potential Memory Leak in BitCaskDiskMap Due to Circular Reference [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12918:
URL: https://github.com/apache/hudi/issues/12918#issuecomment-2716636280
@suxiaogang223 Does doris reuse the merged log reader? For engine like Spark and Flink, no issue around this was reported before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org