You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2024/09/03 07:28:10 UTC

[PR] [core] FileBasedBloomFilter should be pass private member to cache manager. [paimon]

leaves12138 opened a new pull request, #4118:
URL: https://github.com/apache/paimon/pull/4118

   
   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Cache manager cause out of memory while lookup merge
   
   <!-- What is the purpose of the change -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] FileBasedBloomFilter should not be pass private member to cache manager. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4118:
URL: https://github.com/apache/paimon/pull/4118


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] FileBasedBloomFilter should not be pass private member to cache manager. [paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on PR #4118:
URL: https://github.com/apache/paimon/pull/4118#issuecomment-2451546075

   > Hi @leaves12138 , why this will lead to out of memory ?
   
   If we use lambda, the member "filter" will be indexed. The instance which contains the "filter" will also be referred, cannot gc


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] FileBasedBloomFilter should not be pass private member to cache manager. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4118:
URL: https://github.com/apache/paimon/pull/4118#discussion_r1741585181


##########
paimon-common/src/main/java/org/apache/paimon/utils/FileBasedBloomFilter.java:
##########
@@ -58,12 +58,13 @@ public boolean testHash(int hash) {
         // we should refresh cache in LRU, but we cannot refresh everytime, it is costly.
         // so we introduce a refresh count to reduce refresh
         if (accessCount == REFRESH_COUNT || filter.getMemorySegment() == null) {
+            final BloomFilter bloomFilter = filter;
             MemorySegment segment =
                     cacheManager.getPage(
                             CacheKey.forPosition(input.file(), readOffset, readLength),
                             key -> input.readPosition(readOffset, readLength),
-                            key -> filter.unsetMemorySegment());
-            filter.setMemorySegment(segment, 0);
+                            key -> bloomFilter.unsetMemorySegment());

Review Comment:
   add a static inner class to do this unset.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] FileBasedBloomFilter should not be pass private member to cache manager. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4118:
URL: https://github.com/apache/paimon/pull/4118#issuecomment-2325911605

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core] FileBasedBloomFilter should not be pass private member to cache manager. [paimon]

Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on PR #4118:
URL: https://github.com/apache/paimon/pull/4118#issuecomment-2444452915

   Hi @leaves12138 , why this will lead to out of memory ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org