You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "smiklosovic (via GitHub)" <gi...@apache.org> on 2025/12/15 12:19:31 UTC
[PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
smiklosovic opened a new pull request, #4521:
URL: https://github.com/apache/cassandra/pull/4521
Thanks for sending a pull request! Here are some tips if you're new here:
* Ensure you have added or run the [appropriate tests](https://cassandra.apache.org/_/development/testing.html) for your PR.
* Be sure to keep the PR description updated to reflect all changes.
* Write your PR title to summarize what this PR proposes.
* If possible, provide a concise example to reproduce the issue for a faster review.
* Read our [contributor guidelines](https://cassandra.apache.org/_/development/index.html)
* If you're making a documentation change, see our [guide to documentation contribution](https://cassandra.apache.org/_/development/documentation.html)
Commit messages should follow the following format:
```
<One sentence description, usually Jira title or CHANGES.txt summary>
<Optional lengthier description (context on patch)>
patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####
Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>
```
The [Cassandra Jira](https://issues.apache.org/jira/projects/CASSANDRA/issues/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619556710
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -102,7 +102,12 @@ public void add(@Nullable CompressionDictionary compressionDictionary)
// Only update cache if not already in the cache
DictId newDictId = compressionDictionary.dictId();
- cache.get(newDictId, id -> compressionDictionary);
+ CompressionDictionary cached = cache.get(newDictId, id -> compressionDictionary);
+ if (cached != compressionDictionary)
+ {
+ compressionDictionary.close();
+ return;
+ }
Review Comment:
this is exercised when doing `refreshDictionaryFromSystemTable` repeatedly (every `compression_dictionary_refresh_interval`) it is just that in tests, this will be run also after `compression_dictionary_refresh_initial_delay` which is 10s so if tests take longer than this then `refreshDictionaryFromSystemTable` will kick in and it gets here.
My suspicion is that we retrieve last from disk, then we go to cache it but it is already there so we dont do that, then the rest of the `cache.add` method finishes and we end up with `CompressionDictionary` from disk nobody is interested in anymore (because it is cached under a different instance already) and it is not closed either. So once it is going to be GCed it will leak - saying that nobody released it yet it goes to be GCed - that's because nobody called `close()` on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619556710
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -102,7 +102,12 @@ public void add(@Nullable CompressionDictionary compressionDictionary)
// Only update cache if not already in the cache
DictId newDictId = compressionDictionary.dictId();
- cache.get(newDictId, id -> compressionDictionary);
+ CompressionDictionary cached = cache.get(newDictId, id -> compressionDictionary);
+ if (cached != compressionDictionary)
+ {
+ compressionDictionary.close();
+ return;
+ }
Review Comment:
this is exercised when doing `refreshDictionaryFromSystemTable` repeatedly (every `compression_dictionary_refresh_interval`) it is just that in tests, this will be run also after `compression_dictionary_refresh_initial_delay` which is 10s so if tests take longer than this then `refreshDictionaryFromSystemTable` will kick in and it gets here.
My suspicion is that we retrieve the last dict from disk, then we go to cache it but it is already there so we dont do that, then the rest of the `cache.add` method finishes and we end up with `CompressionDictionary` from disk nobody is interested in anymore (because it is cached under a different instance already) and it is not closed either. So once it is going to be GCed it will leak - saying that nobody released it yet it goes to be GCed - that's because nobody called `close()` on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#issuecomment-3661011632
closing this as we will treat the problem differently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619556710
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -102,7 +102,12 @@ public void add(@Nullable CompressionDictionary compressionDictionary)
// Only update cache if not already in the cache
DictId newDictId = compressionDictionary.dictId();
- cache.get(newDictId, id -> compressionDictionary);
+ CompressionDictionary cached = cache.get(newDictId, id -> compressionDictionary);
+ if (cached != compressionDictionary)
+ {
+ compressionDictionary.close();
+ return;
+ }
Review Comment:
this is exercised when doing `refreshDictionaryFromSystemTable` repeatedly (every `compression_dictionary_refresh_interval`) it is just that in tests, this will be run also after `compression_dictionary_refresh_initial_delay` which is 10s so if tests take longer than this then `refreshDictionaryFromSystemTable` will kick in and it gets here. My suspicion is that we retrieve last from disk, then we go to cache it but it is already there so we dont do that, then the rest of the `cache.add` method finishes and we end up with `CompressionDictionary` from disk nobody is interested in anymore (because it is cached under a different instance already) and it is not closed either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic closed pull request #4521: CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes
URL: https://github.com/apache/cassandra/pull/4521
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619488064
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -283,7 +283,14 @@ public CompositeData getCompressionDictionary()
if (compressionDictionary == null)
return null;
- return CompressionDictionaryDetailsTabularData.fromCompressionDictionary(keyspaceName, tableName, compressionDictionary);
+ try
+ {
+ return CompressionDictionaryDetailsTabularData.fromCompressionDictionary(keyspaceName, tableName, compressionDictionary);
+ }
+ finally
+ {
+ compressionDictionary.close();
+ }
Review Comment:
without doing it like this we are not closing anywhere so we would keep dictionary we just retrieved from disk dangling, not releasing it anywhere, then once that gets GCed it would leak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619556710
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -102,7 +102,12 @@ public void add(@Nullable CompressionDictionary compressionDictionary)
// Only update cache if not already in the cache
DictId newDictId = compressionDictionary.dictId();
- cache.get(newDictId, id -> compressionDictionary);
+ CompressionDictionary cached = cache.get(newDictId, id -> compressionDictionary);
+ if (cached != compressionDictionary)
+ {
+ compressionDictionary.close();
+ return;
+ }
Review Comment:
this is exercised when doing `refreshDictionaryFromSystemTable` repeatedly (every `compression_dictionary_refresh_interval`) it is just that in tests, this will be run also after `compression_dictionary_refresh_initial_delay` which is 10s so if tests take longer than this then `refreshDictionaryFromSystemTable` will kick in and it gets here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619432298
##########
test/unit/org/apache/cassandra/tools/nodetool/ExportImportListCompressionDictionaryTest.java:
##########
@@ -53,6 +53,7 @@ public static void setup() throws Throwable
public void testExportImportListCompressionDictionary() throws Throwable
{
// create table, train dictionary for it, export it to json file
+
Review Comment:
will be removed on commit, i just need to have some change in this file so circleci scripting we have in place will evaluate this test to be included in a multiplexer
##########
test/unit/org/apache/cassandra/tools/nodetool/ExportImportListCompressionDictionaryTest.java:
##########
@@ -53,6 +53,7 @@ public static void setup() throws Throwable
public void testExportImportListCompressionDictionary() throws Throwable
{
// create table, train dictionary for it, export it to json file
+
Review Comment:
will be removed on commit, i just need to have some change in this file so circleci scripting we have in place will evaluate this test to be included in a multiplexer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619556710
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -102,7 +102,12 @@ public void add(@Nullable CompressionDictionary compressionDictionary)
// Only update cache if not already in the cache
DictId newDictId = compressionDictionary.dictId();
- cache.get(newDictId, id -> compressionDictionary);
+ CompressionDictionary cached = cache.get(newDictId, id -> compressionDictionary);
+ if (cached != compressionDictionary)
+ {
+ compressionDictionary.close();
+ return;
+ }
Review Comment:
this is exercised when doing `refreshDictionaryFromSystemTable` repeatedly (every `compression_dictionary_refresh_interval`) it is just that in tests, this will be run also after `compression_dictionary_refresh_initial_delay` which is 10s so if tests take longer than this then `refreshDictionaryFromSystemTable` will kick in and it gets here. Same error would be logged (as in ticket comments) if we just waited one more hour.
My suspicion is that we retrieve the last dict from disk, then we go to cache it but it is already there so we dont do that, then the rest of the `cache.add` method finishes and we end up with `CompressionDictionary` from disk nobody is interested in anymore (because it is cached under a different instance already) and it is not closed either. So once it is going to be GCed it will leak - saying that nobody released it - that's because nobody called `close()` on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619556710
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -102,7 +102,12 @@ public void add(@Nullable CompressionDictionary compressionDictionary)
// Only update cache if not already in the cache
DictId newDictId = compressionDictionary.dictId();
- cache.get(newDictId, id -> compressionDictionary);
+ CompressionDictionary cached = cache.get(newDictId, id -> compressionDictionary);
+ if (cached != compressionDictionary)
+ {
+ compressionDictionary.close();
+ return;
+ }
Review Comment:
this is exercised when doing `refreshDictionaryFromSystemTable` repeatedly (every `compression_dictionary_refresh_interval`) it is just that in tests, this will be run also after `compression_dictionary_refresh_initial_delay` which is 10s so if tests take longer than this then `refreshDictionaryFromSystemTable` will kick in and it gets here.
My suspicion is that we retrieve the last dict from disk, then we go to cache it but it is already there so we dont do that, then the rest of the `cache.add` method finishes and we end up with `CompressionDictionary` from disk nobody is interested in anymore (because it is cached under a different instance already) and it is not closed either. So once it is going to be GCed it will leak - saying that nobody released it - that's because nobody called `close()` on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "jyothsnakonisa (via GitHub)" <gi...@apache.org>.
jyothsnakonisa commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2621119246
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -102,7 +102,12 @@ public void add(@Nullable CompressionDictionary compressionDictionary)
// Only update cache if not already in the cache
DictId newDictId = compressionDictionary.dictId();
- cache.get(newDictId, id -> compressionDictionary);
+ CompressionDictionary cached = cache.get(newDictId, id -> compressionDictionary);
+ if (cached != compressionDictionary)
+ {
+ compressionDictionary.close();
+ return;
+ }
Review Comment:
@smiklosovic The `add()` method is closing dictionaries that this class didn't create. Callers may attempt to use dictionaries that have been silently closed. have you considered other alternatives? may be like
1. Moving `SystemDistributedKeyspace.retrieveCompressionDictionary` call to `add()` method and handling the close in the same method
2. `add ()` method indicating the caller that the dictionary has to be closed and the caller takes action based on the result
3. Any other options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619483976
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -102,7 +102,12 @@ public void add(@Nullable CompressionDictionary compressionDictionary)
// Only update cache if not already in the cache
DictId newDictId = compressionDictionary.dictId();
- cache.get(newDictId, id -> compressionDictionary);
+ CompressionDictionary cached = cache.get(newDictId, id -> compressionDictionary);
+ if (cached != compressionDictionary)
+ {
+ compressionDictionary.close();
+ return;
+ }
Review Comment:
the logic here is that if the returned `cached` instance is different from `compressionDictionary`, then there is already cached dictionary under same dictionary id, so it is presumably same dictionary, so we do not need to do anything except closing it and returning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619490603
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -370,11 +391,6 @@ private CompressionDictionaryTrainingConfig createTrainingConfig()
private void storeDictionary(CompressionDictionary dictionary)
{
- if (!isEnabled)
Review Comment:
if we do not add to cache then who is actually releasing it? ... right ... nobody!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21074 close compression dict after retrieval, before returning for e.g. export purposes [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4521:
URL: https://github.com/apache/cassandra/pull/4521#discussion_r2619556710
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -102,7 +102,12 @@ public void add(@Nullable CompressionDictionary compressionDictionary)
// Only update cache if not already in the cache
DictId newDictId = compressionDictionary.dictId();
- cache.get(newDictId, id -> compressionDictionary);
+ CompressionDictionary cached = cache.get(newDictId, id -> compressionDictionary);
+ if (cached != compressionDictionary)
+ {
+ compressionDictionary.close();
+ return;
+ }
Review Comment:
this is exercised when doing `refreshDictionaryFromSystemTable` repeatedly (every `compression_dictionary_refresh_interval`) it is just that in tests, this will be run also after `compression_dictionary_refresh_initial_delay` which is 10s so if tests take longer than this then `refreshDictionaryFromSystemTable` will kick in and it gets here. My suspicion is that we retrieve last from disk, then we go to cache it but it is already there so we dont do that, then the rest of the `cache.add` method finishes and we end up with `CompressionDictionary` from disk nobody is interested in anymore (because it is cached) and it is not closed either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org