You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "yifan-c (via GitHub)" <gi...@apache.org> on 2025/09/26 22:09:59 UTC
[PR] Support ZSTD dictionary compression [cassandra]
yifan-c opened a new pull request, #4399:
URL: https://github.com/apache/cassandra/pull/4399
Add ZstdDictionaryCompressor and wire to the read and the write path
Resolve a correct dictionary compressor on the read path
lazily init ZstdDictCompress and ZstdDictDecompress
checksumming
Added compression dictionary table and accessors
Wire to System table access to CompressionDictionaryManager
Integrate dictionary trainer
Add nodetool to train dictionary
Cleanup
break CompressionDictionaryManager up
properly release dictionary with reference counting
Add ZstdDictionaryCompressorTest
Simplify trainer interface
Bump SSTable version
Add CompressionDictionaryCacheTest
Add ZstdDictionaryTrainerTest
update to use assertJ
Add CompressionDictionaryTrainingConfigTest
Update comments in CompressionDictionaryEventHandler and update exception handling in MessagingService
Add CompressionDictionaryEventHandlerTest
Add CompressionDictionarySchedulerTest
Minor update
Add CompressionDictionaryManagerTest
Add TrainCompressionDictionaryTest (nodetool)
Add CompressionDictionaryIntegrationTest
Tidy up
Add ZstdCompressionDictionaryTest
Add SystemDistributedKeyspaceCompressionDictionaryTest
Add --sampling-rate for the nodetool
Update consistency level from QUORUM to ONE for best effort
Add the missing setCompressionDictionaryManager to sstable writer builders
Add benchmark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "rustyrazorblade (via GitHub)" <gi...@apache.org>.
rustyrazorblade commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414974162
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
Review Comment:
I think the eviction code is broken, see my full comment in JIRA. When compacting a table that I left overnight (assuming the dictionary was evicted) I get this:
```
WARN [CompactionExecutor:869] 2025-10-08T17:16:05,033 SSTableWriter.java:157 - Failed to open /Users/jhaddad/dev/cassandra/bin/../data/data/chunk4_2/keyvalue-1b255f4def2540a600000000000000f6/pa-2-big for writing
java.lang.IllegalStateException: Dictionary is being closed
at org.apache.cassandra.io.compress.ZstdDictionaryCompressor.lambda$getOrCreate$1(ZstdDictionaryCompressor.java:100)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410744562
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+public interface ICompressionDictionaryEventHandler
Review Comment:
As suggested elsewhere we might use `DictId` here instead for both methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415140355
##########
src/java/org/apache/cassandra/db/compaction/unified/ShardedMultiWriter.java:
##########
@@ -118,6 +118,7 @@ private SSTableWriter createWriter(Descriptor descriptor)
.setSerializationHeader(header)
.addDefaultComponents(indexGroups)
.setSecondaryIndexGroups(indexGroups)
+ .setCompressionDictionaryManager(cfs.compressionDictionaryManager())
Review Comment:
It is closed by CFS (the owner) when `invalidate` (e.g. drop table)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414970331
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
Review Comment:
I am open to 24 hours. Was just placing some default values in the initial patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410973646
##########
src/java/org/apache/cassandra/db/compression/ZstdDictionaryTrainer.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.luben.zstd.Zstd;
+import com.github.luben.zstd.ZstdDictTrainer;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.CompressionDictionary.DictId;
+import org.apache.cassandra.db.compression.CompressionDictionary.Kind;
+import org.apache.cassandra.io.compress.IDictionaryCompressor;
+import org.apache.cassandra.io.compress.ZstdDictionaryCompressor;
+import org.apache.cassandra.schema.CompressionParams;
+
+/**
+ * Zstd implementation of dictionary trainer with lifecycle management.
+ */
+public class ZstdDictionaryTrainer implements ICompressionDictionaryTrainer
+{
+ private static final Logger logger = LoggerFactory.getLogger(ZstdDictionaryTrainer.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final CompressionDictionaryTrainingConfig config;
+ private final AtomicLong totalSampleSize;
+ private final AtomicLong sampleCount;
+ private final int compressionLevel; // optimal if using the same level for training as when compressing.
+
+ // Sampling rate can be updated during training
+ private volatile int samplingRate;
+
+ // Minimum number of samples required by ZSTD library
+ private static final int MIN_SAMPLES_REQUIRED = 10;
+
+ private volatile Consumer<CompressionDictionary> dictionaryTrainedListener;
+ // TODO: manage the samples in this class for auto-train (follow-up). The ZstdDictTrainer cannot be re-used for multiple training runs.
+ private ZstdDictTrainer zstdTrainer;
+ private volatile boolean closed = false;
+ private volatile TrainingStatus currentTrainingStatus;
+
+ public ZstdDictionaryTrainer(String keyspaceName, String tableName,
+ CompressionDictionaryTrainingConfig config,
+ int compressionLevel)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.config = config;
+ this.totalSampleSize = new AtomicLong(0);
+ this.sampleCount = new AtomicLong(0);
+ this.compressionLevel = compressionLevel;
+ this.samplingRate = config.samplingRate;
+ this.currentTrainingStatus = TrainingStatus.NOT_STARTED;
+ }
+
+ @Override
+ public boolean shouldSample()
+ {
+ return zstdTrainer != null && ThreadLocalRandom.current().nextInt(samplingRate) == 0;
+ }
+
+ @Override
+ public void addSample(ByteBuffer sample)
+ {
+ if (closed || sample == null || !sample.hasRemaining() || zstdTrainer == null)
+ return;
+
+ byte[] sampleBytes = new byte[sample.remaining()];
+ sample.duplicate().get(sampleBytes);
+
+ if (zstdTrainer.addSample(sampleBytes))
+ {
+ // Update the totalSampleSize and sampleCount if the sample is added
+ totalSampleSize.addAndGet(sampleBytes.length);
+ sampleCount.incrementAndGet();
+ }
+ }
+
+ @Override
+ public CompressionDictionary trainDictionary(boolean force)
+ {
+ boolean isReady = isReady();
+ if (!force && !isReady)
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ throw new IllegalStateException("Trainer is not ready");
+ }
+
+ long currentSampleCount = sampleCount.get();
+ if (currentSampleCount < MIN_SAMPLES_REQUIRED) // minimum samples should be required even if force training
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ String errorMsg = String.format("Insufficient samples for training: %d (minimum required: %d)",
+ currentSampleCount, MIN_SAMPLES_REQUIRED);
+ throw new IllegalStateException(errorMsg);
+ }
+
+ currentTrainingStatus = TrainingStatus.TRAINING;
+ try
+ {
+ logger.debug("Training with sample count: {}, sample size: {}, isReady: {}",
+ currentSampleCount, totalSampleSize.get(), isReady);
+ byte[] dictBytes = zstdTrainer.trainSamples();
+ long zstdDictId = Zstd.getDictIdFromDict(dictBytes);
+ DictId dictId = new DictId(Kind.ZSTD, makeDictionaryId(System.currentTimeMillis(), zstdDictId));
+ currentTrainingStatus = TrainingStatus.COMPLETED;
+ logger.debug("New dictionary is trained with {}", dictId);
+ CompressionDictionary dictionary = new ZstdCompressionDictionary(dictId, dictBytes);
+ notifyDictionaryTrainedListener(dictionary);
+ return dictionary;
+ }
+ catch (Exception e)
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ throw new RuntimeException("Failed to train Zstd dictionary", e);
+ }
+ }
+
+ @Override
+ public boolean isReady()
+ {
+ return currentTrainingStatus != TrainingStatus.TRAINING
+ && !closed
+ && zstdTrainer != null
+ && totalSampleSize.get() >= config.acceptableTotalSampleSize
+ && sampleCount.get() > MIN_SAMPLES_REQUIRED;
+ }
+
+ @Override
+ public TrainingStatus getTrainingStatus()
+ {
+ return currentTrainingStatus;
+ }
+
+ @Override
+ public boolean start(boolean manualTraining)
+ {
+ if (closed || !(manualTraining || shouldAutoStartTraining()))
+ return false;
+
+ try
+ {
+ // reset on starting; a new zstdTrainer instance is created during reset
+ reset();
+ logger.info("Started dictionary training for {}.{}", keyspaceName, tableName);
+ currentTrainingStatus = TrainingStatus.SAMPLING;
+ return true;
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to create ZstdDictTrainer for {}.{}", keyspaceName, tableName, e);
+ currentTrainingStatus = TrainingStatus.FAILED;
+ }
+ return false;
+ }
+
+ /**
+ * Determines if training should auto-start based on configuration.
+ */
+ private boolean shouldAutoStartTraining()
+ {
+ return DatabaseDescriptor.getCompressionDictionaryTrainingAutoTrainEnabled();
+ }
+
+ @Override
+ public void reset()
+ {
+ if (closed)
+ {
+ return;
+ }
+
+ currentTrainingStatus = TrainingStatus.NOT_STARTED;
+ synchronized (this)
+ {
+ totalSampleSize.set(0);
+ sampleCount.set(0);
+ zstdTrainer = new ZstdDictTrainer(config.maxTotalSampleSize, config.maxDictionarySize, compressionLevel);
+ }
+ }
+
+ @Override
+ public Kind kind()
+ {
+ return Kind.ZSTD;
+ }
+
+ @Override
+ public void setDictionaryTrainedListener(Consumer<CompressionDictionary> listener)
+ {
+ this.dictionaryTrainedListener = listener;
+ }
+
+ @Override
+ public void updateSamplingRate(int newSamplingRate)
+ {
+ if (newSamplingRate <= 0)
+ {
+ throw new IllegalArgumentException("Sampling rate must be positive, got: " + newSamplingRate);
+ }
+ this.samplingRate = newSamplingRate;
+ logger.debug("Updated sampling rate to {} for {}.{}", newSamplingRate, keyspaceName, tableName);
+ }
+
+ /**
+ * Notifies the registered listener that a dictionary has been trained.
+ *
+ * @param dictionary the newly trained dictionary
+ */
+ private void notifyDictionaryTrainedListener(CompressionDictionary dictionary)
+ {
+ Consumer<CompressionDictionary> listener = this.dictionaryTrainedListener;
+ if (listener != null)
+ {
+ try
+ {
+ listener.accept(dictionary);
+ }
+ catch (Exception e)
+ {
+ logger.warn("Error notifying dictionary trained listener for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+ }
+
+ @Override
+ public boolean isCompatibleWith(CompressionParams newParams)
+ {
+ if (!newParams.isDictionaryCompressionEnabled())
+ {
+ return false;
+ }
+
+ IDictionaryCompressor newCompressor = (IDictionaryCompressor) newParams.getSstableCompressor();
+
+ // Check if the compressor type is compatible with this trainer
+ if (newCompressor.acceptableDictionaryKind() != Kind.ZSTD)
+ {
+ return false;
+ }
+
+ ZstdDictionaryCompressor zstdDictionaryCompressor = (ZstdDictionaryCompressor) newCompressor;
+ // For Zstd compressors, check if compression level matches
+ return this.compressionLevel == zstdDictionaryCompressor.compressionLevel();
+ }
+
+ @Override
+ public void close()
+ {
+ if (closed)
+ return;
+
+ closed = true;
+ currentTrainingStatus = TrainingStatus.NOT_STARTED;
+
+ synchronized (this)
+ {
+ // Permanent shutdown: clear all state and prevent restart
+ totalSampleSize.set(0);
+ sampleCount.set(0);
+ zstdTrainer = null;
+ }
+
+ logger.info("Permanently closed dictionary trainer for {}.{}", keyspaceName, tableName);
+ }
+
+ /**
+ * Creates a monotonically increasing dictionary ID by combining timestamp and Zstd dictionary ID.
+ *
+ * The resulting dictionary ID has the following structure:
+ * - Upper 32 bits: timestamp in minutes (signed int)
+ * - Lower 32 bits: Zstd dictionary ID (unsigned int, passed as long due to Java limitations)
+ *
+ * This ensures dictionary IDs are monotonically increasing over time, which helps to identify
+ * the latest dictionary.
+ *
+ * The implementation assumes that dictionary training frequency is significantly larger than
+ * every minute, which a healthy system should do. In the scenario when multiple dictionaries
+ * are trained in the same minute (only possible using manual training), there should not be
+ * correctness concerns since the dictionary is attached to the SSTables, but leads to performance
+ * hit from having too many dictionary. Therefore, such scenario should be avoided at the best.
+ *
+ * @param currentTimeMillis the current time in milliseconds
+ * @param zstdDictId Zstd dictionary ID (unsigned 32-bit value represented as long)
+ * @return combined dictionary ID that is monotonically increasing over time
+ */
+ static long makeDictionaryId(long currentTimeMillis, long zstdDictId)
Review Comment:
is this really ZSTD specific? seems like moving this to DictId or similar would be better so it can be used for other compressors too. I just dont see why is this ZSTD specific only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2424784647
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
@yifan-c well with a big gun there comes a lot of responsibility :) I would also introduce number of samples, if we are worried about that never ending, I would add duration _as additional boundary_.
For example, max samples 100 000 OR duration of 6 hours. Whatever happens first will end it.
@rustyrazorblade what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "rustyrazorblade (via GitHub)" <gi...@apache.org>.
rustyrazorblade commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2424796868
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
I think the priority should be reducing the number of operator headaches. If it runs for 6 hours, would most people want it to keep running or fail? I'd rather it keep going. What is the benefit of failing, ever? What is the problem if it takes a full week?
> I acknowledge that having a failed training after 10 minutes is frustrating. It is up to operators to configure the tool properly;
I think we have a responsibility not to ship foot-guns. Defaults should work well. Expecting people to change every config under the guise of "operator responsibility" is a mindset that caused Cassandra to ship some pretty awful defaults for years - GC being a notable example. It's done a lot of damage to the project, and I see it as an abdication of an important responsibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#issuecomment-3403720807
Pushed changes and trigger the CI https://pre-ci.cassandra.apache.org/job/cassandra/123/pipeline-overview/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#issuecomment-3408273512
A new commit to further simplify/enhance the nodetool command. And a new CI run https://pre-ci.cassandra.apache.org/job/cassandra/124/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414979062
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415153854
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.time.Duration;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import org.apache.cassandra.config.DatabaseDescriptor;
+
+/**
+ * Manages caching and current dictionary state for compression dictionaries.
+ * <p>
+ * This class handles:
+ * - Local caching of compression dictionaries with automatic cleanup
+ * - Managing the current active dictionary for write operations
+ * - Thread-safe access to cached dictionaries
+ */
+public class CompressionDictionaryCache implements ICompressionDictionaryCache
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryCache.class);
+
+ private final Cache<CompressionDictionary.DictId, CompressionDictionary> cache;
+ private final AtomicReference<CompressionDictionary> currentDictionary = new AtomicReference<>();
+
+ public CompressionDictionaryCache()
+ {
+ Duration expiryTime = Duration.ofSeconds(DatabaseDescriptor.getCompressionDictionaryCacheExpireSeconds());
Review Comment:
Originally it was inline. But the line is too long. I am fine with inline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410953126
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
+ {
+ throw new IllegalArgumentException("Table " + keyspaceName + '.' + tableName + " does not support dictionary compression");
+ }
+
+ if (trainer == null)
+ {
+ throw new IllegalStateException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+
+ // Parse and validate training options
+ ManualTrainingOptions trainingOptions = ManualTrainingOptions.fromStringMap(options);
+
+ trainer.start(true);
+ scheduler.scheduleManualTraining(trainingOptions, trainer);
Review Comment:
we would not actually _scheduled_ anything in that method if trainer is no-op instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415148228
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
+ if (kind == Kind.ZSTD)
+ {
+ dictionary = new ZstdCompressionDictionary(dictId, dict);
+ }
+
+ if (dictionary == null)
+ {
+ throw new IOException(kind + " compression dictionary is not created");
+ }
+
+ // update the dictionary manager if it exists
+ if (manager != null)
+ {
+ manager.add(dictionary);
+ }
+ return dictionary;
+ }
+
+ static CompressionDictionary createFromRow(UntypedResultSet.Row row)
+ {
+ String kindStr = row.getString("kind");
+ long id = row.getLong("dict_id");
+ byte[] dict = row.getByteArray("dict");
Review Comment:
Good point. I saw that you have added the change. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410269478
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryScheduler.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+
+/**
+ * Manages scheduled tasks for compression dictionary operations.
+ * <p>
+ * This class handles:
+ * - Periodic refresh of dictionaries from system tables
+ * - Manual training task scheduling and monitoring
+ * - Cleanup of scheduled tasks
+ */
+public class CompressionDictionaryScheduler implements ICompressionDictionaryScheduler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryScheduler.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ private volatile ScheduledFuture<?> scheduledRefreshTask;
+ private volatile ScheduledFuture<?> scheduledManualTrainingTask;
+ private volatile boolean isEnabled;
+
+ public CompressionDictionaryScheduler(String keyspaceName,
+ String tableName,
+ ICompressionDictionaryCache cache,
+ boolean isEnabled)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.cache = cache;
+ this.isEnabled = isEnabled;
+ }
+
+ /**
+ * Schedules the periodic dictionary refresh task if not already scheduled.
+ */
+ public void scheduleRefreshTask()
+ {
+ if (scheduledRefreshTask != null)
+ return;
+
+ this.scheduledRefreshTask = ScheduledExecutors.scheduledTasks.scheduleWithFixedDelay(
+ this::refreshDictionaryFromSystemTable,
+ DatabaseDescriptor.getCompressionDictionaryRefreshInitialDelaySeconds(),
+ DatabaseDescriptor.getCompressionDictionaryRefreshIntervalSeconds(),
+ TimeUnit.SECONDS
+ );
+ }
+
+ @Override
+ public void scheduleManualTraining(ManualTrainingOptions options, ICompressionDictionaryTrainer trainer)
+ {
+ if (scheduledManualTrainingTask != null)
+ {
+ throw new IllegalStateException("Training already in progress for table " + keyspaceName + '.' + tableName);
+ }
+
+ int maxSamplingDurationSeconds = options.getMaxSamplingDurationSeconds();
+
+ logger.info("Starting manual dictionary training for {}.{} with max sampling duration: {} seconds",
+ keyspaceName, tableName, maxSamplingDurationSeconds);
+
+ long deadlineMillis = System.currentTimeMillis() + TimeUnit.SECONDS.toMillis(maxSamplingDurationSeconds);
Review Comment:
Avoid System for time, should use `org.apache.cassandra.utils.Clock.Global` or `org.apache.cassandra.utils.Clock` interface [blockSystemClock]
##########
src/java/org/apache/cassandra/io/compress/CompressionMetadata.java:
##########
@@ -139,11 +165,54 @@ private CompressionMetadata(CompressionMetadata copy)
this.compressedFileLength = copy.compressedFileLength;
this.chunkOffsets = copy.chunkOffsets;
this.chunkOffsetsSize = copy.chunkOffsetsSize;
+ this.compressionDictionary = copy.compressionDictionary;
}
public ICompressor compressor()
{
- return parameters.getSstableCompressor();
+ // classic double-checked locking to call resolveCompressor method just once per CompressionMetadata object
+ ICompressor result = resolvedCompressor;
+ if (result != null)
+ return result;
+
+ synchronized (this)
+ {
+ result = resolvedCompressor;
+ if (result == null)
+ {
+ result = resolveCompressor(parameters.getSstableCompressor(), compressionDictionary);
+ resolvedCompressor = result;
+ }
+ return result;
+ }
+ }
+
+ static ICompressor resolveCompressor(ICompressor compressor, CompressionDictionary dictionary)
+ {
+ if (dictionary == null)
+ return compressor;
+
+ // When the attached dictionary can be consumed by the current dictionary compressor
+ if (compressor instanceof IDictionaryCompressor)
+ {
+ IDictionaryCompressor dictionaryCompressor = (IDictionaryCompressor) compressor;
+ if (dictionaryCompressor.canConsumeDictionary(dictionary))
+ return dictionaryCompressor.getOrCopyWithDictionary(dictionary);
+ }
+
+ // When the current compressor is not compatible with the dictionary. It could happen in the read path when:
+ // 1. The current compressor is not a dictionary compressor, but there is dictionary attached
+ // 2. The current dictionary compressor is a different type, e.g. table schema is changed
+ // In those cases, we should get the compatible dictionary compressor based on the dictionary
+ if (dictionary.kind() == ZSTD)
Review Comment:
We are again too "ZSTD-centric" here. I would expect this does not contain anything ZSTD specific so we do not need to touch this anymore. The fact that dictionary is not empty again pretty much guarantees that we can create a compressor from it, no? Why would we have a dictionary, of `CompressionDictionary` type, if we were not able to create a compressor from it? That would be a pretty useless dictionary to me.
So maybe doing something like
`return dictionary.kind().getCompressor(dictionary);`
and calling `ZstdDictionaryCompressor.create(zstdDict)` in there would be way more simpler and at one place.
You would need to return `ICompressor` from `getCompressor` as `resolveCompressor` returns it too. If `IDictionaryCompressor` extended `ICompressor` as already suggested elsewhere (as you need to instantiate `ZstdDictionaryCompressor` there) it would be a piece of cake.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
Review Comment:
checkstyle (`ant realclean && ant artifacts`) evaluated this as illegal import, see checkstyle.xml and grep `CompletableFuture` there to see. I am not completely sure yet but I think it should be replaced by something in `org.apache.cassandra.utils.concurrent` package.
##########
src/java/org/apache/cassandra/io/compress/CompressionMetadata.java:
##########
@@ -139,11 +165,54 @@ private CompressionMetadata(CompressionMetadata copy)
this.compressedFileLength = copy.compressedFileLength;
this.chunkOffsets = copy.chunkOffsets;
this.chunkOffsetsSize = copy.chunkOffsetsSize;
+ this.compressionDictionary = copy.compressionDictionary;
}
public ICompressor compressor()
{
- return parameters.getSstableCompressor();
+ // classic double-checked locking to call resolveCompressor method just once per CompressionMetadata object
+ ICompressor result = resolvedCompressor;
+ if (result != null)
+ return result;
+
+ synchronized (this)
+ {
+ result = resolvedCompressor;
+ if (result == null)
+ {
+ result = resolveCompressor(parameters.getSstableCompressor(), compressionDictionary);
+ resolvedCompressor = result;
+ }
+ return result;
+ }
+ }
+
+ static ICompressor resolveCompressor(ICompressor compressor, CompressionDictionary dictionary)
+ {
+ if (dictionary == null)
+ return compressor;
+
+ // When the attached dictionary can be consumed by the current dictionary compressor
+ if (compressor instanceof IDictionaryCompressor)
+ {
+ IDictionaryCompressor dictionaryCompressor = (IDictionaryCompressor) compressor;
+ if (dictionaryCompressor.canConsumeDictionary(dictionary))
+ return dictionaryCompressor.getOrCopyWithDictionary(dictionary);
+ }
+
+ // When the current compressor is not compatible with the dictionary. It could happen in the read path when:
+ // 1. The current compressor is not a dictionary compressor, but there is dictionary attached
+ // 2. The current dictionary compressor is a different type, e.g. table schema is changed
+ // In those cases, we should get the compatible dictionary compressor based on the dictionary
+ if (dictionary.kind() == ZSTD)
Review Comment:
or even better
return dictionary.getCompressor();
which would be like
default IDictionaryCompressor getCompressor()
{
return kind().getCompressor(this);
}
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
Review Comment:
We most probably need to copy all of this to cassandra_latest.yaml as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410959195
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
+ {
+ throw new IllegalArgumentException("Table " + keyspaceName + '.' + tableName + " does not support dictionary compression");
+ }
+
+ if (trainer == null)
+ {
+ throw new IllegalStateException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+
+ // Parse and validate training options
+ ManualTrainingOptions trainingOptions = ManualTrainingOptions.fromStringMap(options);
+
+ trainer.start(true);
+ scheduler.scheduleManualTraining(trainingOptions, trainer);
+ }
+
+ @Override
+ public String getTrainingStatus()
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer == null)
+ {
+ return TrainingStatus.NOT_STARTED.toString();
+ }
+ return dictionaryTrainer.getTrainingStatus().toString();
+ }
+
+ @Override
+ public void updateSamplingRate(int samplingRate)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer == null)
+ {
+ throw new IllegalArgumentException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+ dictionaryTrainer.updateSamplingRate(samplingRate);
+ }
+
+ /**
+ * Close all the resources. The method can be called multiple times.
+ */
+ @Override
+ public synchronized void close()
+ {
+ unregisterMbean();
+ if (trainer != null)
Review Comment:
closing no-op trainer would do nothing and when using concrete trainer we would just do `trainer = NO_OP` instead of `trainer = null`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2427256985
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
that's a good question. What do you do when there is 10 SSTables already (or if somebody is changing compression algorithm)? You take the last one (newest one) or the first one? (oldest one?) I would take the newest one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "rustyrazorblade (via GitHub)" <gi...@apache.org>.
rustyrazorblade commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2427269109
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
I think if there's no SSTables on disk, run flush. Otherwise, train on what's there.
@yifan-c thanks for explaining the sampling, that's what I had in mind too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "rustyrazorblade (via GitHub)" <gi...@apache.org>.
rustyrazorblade commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414968688
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
Review Comment:
What is the overhead of this? This seems like a huge bikeshed over something that has extremely low overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2437718619
##########
src/java/org/apache/cassandra/utils/StorageCompatibilityMode.java:
##########
@@ -35,6 +35,13 @@ public enum StorageCompatibilityMode
*/
CASSANDRA_4(4),
+ /**
+ * Similar to CASSANDRA_4.
+ * The new features in 6.0 are
+ * - ZSTD dictionary-based compression. Once SSTables are compressed with dictionary, they cannot be rolled back.
+ */
+ CASSANDRA_5(5),
+
Review Comment:
Is it necessary? `StorageCompatibilityMode` is to check the sstable format compatibility, which is not changed in this patch. Also, the dictionary feature is opt-in. I'd incline removing the code. It seems unnecessary.
cc: @smiklosovic , @rustyrazorblade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2425782938
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
SSTable-based sampling - that one which also solves Jon's usecase by training existing data? Yeah I think that is more robust. My instinct is that by write-based training we are trying to be too smart, while it works more "real-time-ish" and more suitable for automatic scenarios, it has also unpredictable behaviour if not enough writes or similar when used manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410570729
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
Review Comment:
I think it is `long` because that is used as something like a timestamp in `makeDictionaryId`
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
Review Comment:
if I understand this correctly, it basically assumes that additions of new dictionary kinds will be only ever appended at the end of the `Kind` enum. Basically we can not put anything in the middle because ordinals would be shifted. I see you put the comment about this in Kind enum which is important, thank you for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410382840
##########
src/java/org/apache/cassandra/db/compaction/CompactionManager.java:
##########
@@ -1796,6 +1796,7 @@ public static SSTableWriter createWriter(ColumnFamilyStore cfs,
.setSerializationHeader(sstable.header)
.addDefaultComponents(cfs.indexManager.listIndexGroups())
.setSecondaryIndexGroups(cfs.indexManager.listIndexGroups())
+ .setCompressionDictionaryManager(cfs.compressionDictionaryManager())
Review Comment:
closing anywhere?
##########
src/java/org/apache/cassandra/db/compaction/unified/ShardedMultiWriter.java:
##########
@@ -118,6 +118,7 @@ private SSTableWriter createWriter(Descriptor descriptor)
.setSerializationHeader(header)
.addDefaultComponents(indexGroups)
.setSecondaryIndexGroups(indexGroups)
+ .setCompressionDictionaryManager(cfs.compressionDictionaryManager())
Review Comment:
who is "closing" this manager? Not so far in the review to figure it out on my own ... This is highlighted in IDEA as something we do not use try-with-resources on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410953126
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
+ {
+ throw new IllegalArgumentException("Table " + keyspaceName + '.' + tableName + " does not support dictionary compression");
+ }
+
+ if (trainer == null)
+ {
+ throw new IllegalStateException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+
+ // Parse and validate training options
+ ManualTrainingOptions trainingOptions = ManualTrainingOptions.fromStringMap(options);
+
+ trainer.start(true);
+ scheduler.scheduleManualTraining(trainingOptions, trainer);
Review Comment:
we would not actually _scheduled_ anything in that method if trainer is no-op instance.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
+ {
+ throw new IllegalArgumentException("Table " + keyspaceName + '.' + tableName + " does not support dictionary compression");
+ }
+
+ if (trainer == null)
+ {
+ throw new IllegalStateException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+
+ // Parse and validate training options
+ ManualTrainingOptions trainingOptions = ManualTrainingOptions.fromStringMap(options);
+
+ trainer.start(true);
+ scheduler.scheduleManualTraining(trainingOptions, trainer);
+ }
+
+ @Override
+ public String getTrainingStatus()
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer == null)
+ {
+ return TrainingStatus.NOT_STARTED.toString();
+ }
+ return dictionaryTrainer.getTrainingStatus().toString();
Review Comment:
`getTrainingStatus` on no-op trainer would always return `NOT_STARTED`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410421907
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
Review Comment:
nit: braces not necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410519082
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
Review Comment:
This is also too complicated I think. Similarly as below:
int length = input.readInt();
byte[] dict = new byte[length];
input.readFully(dict);
int checksum = input.readInt();
int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
if (checksum != calculatedChecksum)
throw new IOException("Compression dictionary checksum does not match");
// here we are guaranteed to have a dictionary, no?
CompressionDictionary dictionary = kind.getDictionary(dictId, dict);
// update the dictionary manager if it exists
if (manager != null)
manager.add(dictionary);
return dictionary;
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
+
+# Dictionary training configuration (advanced settings)
+# These settings control how compression dictionaries are trained from sample data.
+
+# Maximum size of a trained compression dictionary in bytes.
+# Larger dictionaries may provide better compression but use more memory.
+# Min unit: B
+compression_dictionary_training_max_dictionary_size: 65536
+
+# Maximum total size of sample data to collect for dictionary training.
+# More sample data generally produces better dictionaries but takes longer to train.
+# The recommended sample size is 100x the dictionary size.
+# Min unit: B
+compression_dictionary_training_max_total_sample_size: 10485760
+
+# Enable automatic dictionary training based on sampling of write operations.
+# When enabled, the system will automatically collect samples and train new dictionaries.
+# Manual training via nodetool is always available regardless of this setting.
+compression_dictionary_training_auto_train_enabled: false
+
+# Sampling rate for automatic dictionary training (1-10000).
+# Value of 100 means 1% of writes are sampled. Lower values reduce overhead but may
+# result in less representative sample data for dictionary training.
+compression_dictionary_training_sampling_rate: 100
Review Comment:
Does it really matter to be so granular to express 0.5% as "50"? The "problem" with this is that a reader has to manually re-compute the number in their head. Like 2.35% would be 235 and 0.8% would be 80.
It would be way more comfortable to put these figures in percentage already.
There is `data_disk_usage_percentage_warn_threshold` which operates with percentage conceptually. We can convert the percentage to whatever data type internally afterwards. So where it would be `1` instead of `100`, we would just multiply by 100 internally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415141369
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
Review Comment:
yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415159262
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Handles compression dictionary events including training completion and cluster notifications.
+ * <p>
+ * This class handles:
+ * - Broadcasting dictionary updates to cluster nodes
+ * - Retrieving new dictionaries when notified by other nodes
+ * - Managing dictionary cache updates
+ */
+public class CompressionDictionaryEventHandler implements ICompressionDictionaryEventHandler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryEventHandler.class);
+
+ private final ColumnFamilyStore cfs;
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ public CompressionDictionaryEventHandler(ColumnFamilyStore cfs, ICompressionDictionaryCache cache)
+ {
+ this.cfs = cfs;
+ this.keyspaceName = cfs.keyspace.getName();
+ this.tableName = cfs.getTableName();
+ this.cache = cache;
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
Review Comment:
You have addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415182762
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
Review Comment:
Thanks for calling it out. Looks like the code base is not consistent on using snake case or camel case. For example, `SnapshotOptions` uses camel case and `clearSnapshot` takes options using snake case. I am fine with either way.
The content of the map is not customer facing. It is purely for communication between tooling and server (jmx).
I will create a string constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "rustyrazorblade (via GitHub)" <gi...@apache.org>.
rustyrazorblade commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2421475890
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
Review Comment:
Sorry for the confusion. I'm not advocating for randomized delay. The point I was trying to make is that this operation appears to be so lightweight that debating if it should be run once a day or an hour seems like a bikeshed. We're talking about an incredibly trivial amount of data and a very lightweight operation. Doing it once a day vs an hour or even a minute is like the difference between one and three grain of sand on an entire beach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410967130
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryTrainer.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.io.compress.ICompressor;
+import org.apache.cassandra.io.compress.IDictionaryCompressor;
+import org.apache.cassandra.io.compress.ZstdDictionaryCompressor;
+import org.apache.cassandra.schema.CompressionParams;
+
+/**
+ * Interface for training compression dictionaries from sample data.
+ * <p>
+ * Implementations handle:
+ * - Sample collection and management
+ * - Dictionary training lifecycle
+ * - Asynchronous training execution
+ * - Training status tracking
+ */
+public interface ICompressionDictionaryTrainer extends AutoCloseable
+{
+ /**
+ * Starts the trainer for collecting samples.
+ *
+ * @param manualTraining true if this is manual training, false for automatic
+ * @return true if the trainer is started; otherwise false. The trainer is started
+ * in any of those conditions: 1. trainer closed; 2. not requested for
+ * either manual or auto training; 3. failed to start
+ */
+ boolean start(boolean manualTraining);
+
+ /**
+ * @return true if the trainer is ready to take a new sample; otherwise, false
+ */
+ boolean shouldSample();
+
+ /**
+ * Adds a sample to the training dataset.
+ *
+ * @param sample the sample data to add for training
+ */
+ void addSample(ByteBuffer sample);
+
+ /**
+ * Trains and produces a compression dictionary from collected samples synchronously.
+ *
+ * @param force force the dictionary training even if there are not enough samples;
+ * otherwise, dictionary training won't start if the trainer is not ready
+ * @return the trained compression dictionary
+ */
+ CompressionDictionary trainDictionary(boolean force);
+
+ /**
+ * Trains and produces a compression dictionary from collected samples asynchronously.
+ *
+ * @param force force the dictionary training even if there are not enough samples
+ * @return CompletableFuture that completes when training is done
+ */
+ default CompletableFuture<CompressionDictionary> trainDictionaryAsync(boolean force)
+ {
+ return CompletableFuture.supplyAsync(() -> trainDictionary(force), ScheduledExecutors.nonPeriodicTasks);
+ }
+
+ /**
+ * @return true if enough samples have been collected for training
+ */
+ boolean isReady();
+
+ /**
+ * Clears all collected samples and resets trainer state.
+ */
+ void reset();
+
+ /**
+ * @return the current training status
+ */
+ TrainingStatus getTrainingStatus();
+
+ /**
+ * @return the compression algorithm kind this trainer supports
+ */
+ CompressionDictionary.Kind kind();
+
+ /**
+ * Determines if this trainer is compatible with the given compression parameters.
+ * This method allows the trainer to decide whether it can continue operating
+ * with new compression parameters or if a new trainer instance is needed.
+ *
+ * @param newParams the new compression parameters to check compatibility against
+ * @return true if this trainer is compatible with the new parameters, false otherwise
+ */
+ boolean isCompatibleWith(CompressionParams newParams);
+
+ /**
+ * Sets the listener for dictionary training events.
+ *
+ * @param listener the listener to be notified when dictionaries are trained, null to remove listener
+ */
+ void setDictionaryTrainedListener(Consumer<CompressionDictionary> listener);
+
+ /**
+ * Updates the sampling rate for this trainer.
+ *
+ * @param newSamplingRate the new sampling rate. For exmaple, 1 = sample every time (100%),
+ * 2 = expect sample 1/2 of data (50%), n = expect sample 1/n of data
+ */
+ void updateSamplingRate(int newSamplingRate);
+
+ /**
+ * Factory method to create appropriate trainer based on compression parameters.
+ *
+ * @param keyspaceName the keyspace name for logging
+ * @param tableName the table name for logging
+ * @param params the compression parameters
+ * @param config the training configuration
+ * @return a dictionary trainer for the specified compression algorithm
+ * @throws IllegalArgumentException if no dictionary trainer is available for the compression algorithm
+ */
+ static ICompressionDictionaryTrainer create(String keyspaceName,
+ String tableName,
+ CompressionParams params,
+ CompressionDictionaryTrainingConfig config)
+ {
+ ICompressor compressor = params.getSstableCompressor();
+ if (!(compressor instanceof IDictionaryCompressor))
+ {
+ throw new IllegalArgumentException("Compressor does not support dictionary training: " + params.getSstableCompressor());
+ }
+
+ IDictionaryCompressor dictionaryCompressor = (IDictionaryCompressor) compressor;
+ if (dictionaryCompressor.acceptableDictionaryKind() == CompressionDictionary.Kind.ZSTD)
Review Comment:
we are zstd specific here, not sure that is really needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410270440
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryScheduler.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+
+/**
+ * Manages scheduled tasks for compression dictionary operations.
+ * <p>
+ * This class handles:
+ * - Periodic refresh of dictionaries from system tables
+ * - Manual training task scheduling and monitoring
+ * - Cleanup of scheduled tasks
+ */
+public class CompressionDictionaryScheduler implements ICompressionDictionaryScheduler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryScheduler.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ private volatile ScheduledFuture<?> scheduledRefreshTask;
+ private volatile ScheduledFuture<?> scheduledManualTrainingTask;
+ private volatile boolean isEnabled;
+
+ public CompressionDictionaryScheduler(String keyspaceName,
+ String tableName,
+ ICompressionDictionaryCache cache,
+ boolean isEnabled)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.cache = cache;
+ this.isEnabled = isEnabled;
+ }
+
+ /**
+ * Schedules the periodic dictionary refresh task if not already scheduled.
+ */
+ public void scheduleRefreshTask()
+ {
+ if (scheduledRefreshTask != null)
+ return;
+
+ this.scheduledRefreshTask = ScheduledExecutors.scheduledTasks.scheduleWithFixedDelay(
+ this::refreshDictionaryFromSystemTable,
+ DatabaseDescriptor.getCompressionDictionaryRefreshInitialDelaySeconds(),
+ DatabaseDescriptor.getCompressionDictionaryRefreshIntervalSeconds(),
+ TimeUnit.SECONDS
+ );
+ }
+
+ @Override
+ public void scheduleManualTraining(ManualTrainingOptions options, ICompressionDictionaryTrainer trainer)
+ {
+ if (scheduledManualTrainingTask != null)
+ {
+ throw new IllegalStateException("Training already in progress for table " + keyspaceName + '.' + tableName);
+ }
+
+ int maxSamplingDurationSeconds = options.getMaxSamplingDurationSeconds();
+
+ logger.info("Starting manual dictionary training for {}.{} with max sampling duration: {} seconds",
+ keyspaceName, tableName, maxSamplingDurationSeconds);
+
+ long deadlineMillis = System.currentTimeMillis() + TimeUnit.SECONDS.toMillis(maxSamplingDurationSeconds);
+
+ ManualTrainingTask task = new ManualTrainingTask(deadlineMillis, trainer);
+
+ // Check every second whether it gets enough samples and completes training
+ scheduledManualTrainingTask = ScheduledExecutors.scheduledTasks
+ .scheduleWithFixedDelay(task, 1, 1, TimeUnit.SECONDS);
+ }
+
+ @Override
+ public void cancelManualTraining()
+ {
+ ScheduledFuture<?> future = scheduledManualTrainingTask;
+ if (future != null)
+ {
+ future.cancel(false);
+ }
+ scheduledManualTrainingTask = null;
+ }
+
+ /**
+ * Sets the enabled state of the scheduler. When disabled, refresh tasks will not execute.
+ *
+ * @param enabled whether the scheduler should be enabled
+ */
+ @Override
+ public void setEnabled(boolean enabled)
+ {
+ this.isEnabled = enabled;
+ }
+
+ /**
+ * Refreshes dictionary from system table and updates the cache.
+ * This method is called periodically by the scheduled refresh task.
+ */
+ private void refreshDictionaryFromSystemTable()
+ {
+ try
+ {
+ if (!isEnabled)
+ {
+ return;
+ }
+
+ CompressionDictionary dictionary = SystemDistributedKeyspace.retrieveLatestCompressionDictionary(keyspaceName, tableName);
+ cache.setCurrentIfNewer(dictionary);
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to refresh compression dictionary for {}.{}",
+ keyspaceName, tableName, e);
+ }
+ }
+
+ @Override
+ public void close()
+ {
+ if (scheduledRefreshTask != null)
+ {
+ scheduledRefreshTask.cancel(false);
+ scheduledRefreshTask = null;
+ }
+
+ if (scheduledManualTrainingTask != null)
+ {
+ scheduledManualTrainingTask.cancel(false);
+ scheduledManualTrainingTask = null;
+ }
+ }
+
+ private class ManualTrainingTask implements Runnable
+ {
+ private final long deadlineMillis;
+ private final ICompressionDictionaryTrainer trainer;
+ private boolean isTraining = false;
+
+ private ManualTrainingTask(long deadlineMillis, ICompressionDictionaryTrainer trainer)
+ {
+ this.deadlineMillis = deadlineMillis;
+ this.trainer = trainer;
+ }
+
+ @Override
+ public void run()
+ {
+ if (trainer.getTrainingStatus() == TrainingStatus.NOT_STARTED)
+ {
+ logger.warn("Trainer is not started. Stop training dictionary for table {}.{}", keyspaceName, tableName);
+ cancelManualTraining();
+ return;
+ }
+
+ long now = System.currentTimeMillis();
Review Comment:
same as above
##########
src/java/org/apache/cassandra/tools/nodetool/TrainCompressionDictionary.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.tools.nodetool;
+
+import java.io.PrintStream;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.util.concurrent.Uninterruptibles;
+
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.tools.NodeProbe;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+import picocli.CommandLine.Parameters;
+
+@Command(name = "traincompressiondictionary",
+description = "Manually trigger compression dictionary training for a table")
+public class TrainCompressionDictionary extends AbstractCommand
+{
+ @Parameters(index = "0", description = "The keyspace name", arity = "1")
+ private String keyspace;
+
+ @Parameters(index = "1", description = "The table name", arity = "1")
+ private String table;
+
+ @Option(names = {"-d", "--max-sampling-duration"},
+ description = "Maximum time to collect samples before training dictionary (default: 600 seconds)")
+ private int maxSamplingDurationSeconds = 600;
+
+ @Option(names = {"-r", "--sampling-rate"},
+ description = "Sampling rate as a double value in range (0, 1]. 1.0 means sample all data, 0.5 means sample 50%% of data")
+ private Double samplingRate;
+
+ @Option(names = {"-a", "--async"},
+ description = "Run training asynchronously without waiting for completion")
+ private boolean async = false;
+
+ @Option(names = {"-s", "--status"},
+ description = "Show current training status instead of starting new training")
+ private boolean showStatus = false;
+
+ @Override
+ public void execute(NodeProbe probe)
+ {
+ if (showStatus)
+ {
+ showTrainingStatus(probe);
+ return;
+ }
+
+ PrintStream out = probe.output().out;
+ PrintStream err = probe.output().err;
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ err.printf("Invalid value for max-sampling-duration: %s%n", maxSamplingDurationSeconds);
+ System.exit(1);
+ }
+
+ if (samplingRate != null && (samplingRate <= 0.0 || samplingRate > 1.0))
+ {
+ err.printf("Invalid value for sampling-rate: %s. Must be in range (0, 1]%n", samplingRate);
+ System.exit(1);
+ }
+ try
+ {
+ out.printf("Starting compression dictionary training for %s.%s...%n", keyspace, table);
+ out.printf("Will collect samples for up to %d seconds before training%n", maxSamplingDurationSeconds);
+ if (samplingRate != null)
+ {
+ out.printf("Using sampling rate: %.2f (%.1f%%)%n", samplingRate, samplingRate * 100);
+ }
+
+ // Build options map
+ Map<String, String> options = new HashMap<>();
+ options.put("maxSamplingDurationSeconds", String.valueOf(maxSamplingDurationSeconds));
+
+ probe.trainCompressionDictionary(keyspace, table, options);
+
+ // Update sampling rate if provided (after training has started)
+ if (samplingRate != null)
+ {
+ // Convert from double (0, 1] to integer format (1/rate)
+ // Examples: 1.0 -> 1 (sample every time), 0.5 -> 2 (roughly sample every 2nd), 0.1 -> 10 (roughly sample every 10th)
+ int integerSamplingRate = (int) Math.round(1.0 / samplingRate);
+ probe.updateCompressionDictionaryTrainingSamplingRate(keyspace, table, integerSamplingRate);
+ }
+
+ if (async)
+ {
+ out.printf("Training started asynchronously for %s.%s%n", keyspace, table);
+ out.printf("Use 'nodetool traincompressiondictionary --status %s %s' to check progress.%n",
+ keyspace, table);
+ return;
+ }
+
+ // Wait for completion (training will start automatically after sampling period)
+ out.println("Collecting samples and training. (Since the trainer samples chunk data on " +
+ "writing to new SSTable, you might consider running nodetool 'flush' along " +
+ "with this command to have chunk available for sampling)");
+ long maxWaitMillis = TimeUnit.SECONDS.toMillis(maxSamplingDurationSeconds + 300); // Add 5 minutes for training
+ long startTime = System.currentTimeMillis();
+
+ while (System.currentTimeMillis() - startTime < maxWaitMillis)
Review Comment:
same
##########
src/java/org/apache/cassandra/tools/nodetool/TrainCompressionDictionary.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.tools.nodetool;
+
+import java.io.PrintStream;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.util.concurrent.Uninterruptibles;
+
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.tools.NodeProbe;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+import picocli.CommandLine.Parameters;
+
+@Command(name = "traincompressiondictionary",
+description = "Manually trigger compression dictionary training for a table")
+public class TrainCompressionDictionary extends AbstractCommand
+{
+ @Parameters(index = "0", description = "The keyspace name", arity = "1")
+ private String keyspace;
+
+ @Parameters(index = "1", description = "The table name", arity = "1")
+ private String table;
+
+ @Option(names = {"-d", "--max-sampling-duration"},
+ description = "Maximum time to collect samples before training dictionary (default: 600 seconds)")
+ private int maxSamplingDurationSeconds = 600;
+
+ @Option(names = {"-r", "--sampling-rate"},
+ description = "Sampling rate as a double value in range (0, 1]. 1.0 means sample all data, 0.5 means sample 50%% of data")
+ private Double samplingRate;
+
+ @Option(names = {"-a", "--async"},
+ description = "Run training asynchronously without waiting for completion")
+ private boolean async = false;
+
+ @Option(names = {"-s", "--status"},
+ description = "Show current training status instead of starting new training")
+ private boolean showStatus = false;
+
+ @Override
+ public void execute(NodeProbe probe)
+ {
+ if (showStatus)
+ {
+ showTrainingStatus(probe);
+ return;
+ }
+
+ PrintStream out = probe.output().out;
+ PrintStream err = probe.output().err;
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ err.printf("Invalid value for max-sampling-duration: %s%n", maxSamplingDurationSeconds);
+ System.exit(1);
+ }
+
+ if (samplingRate != null && (samplingRate <= 0.0 || samplingRate > 1.0))
+ {
+ err.printf("Invalid value for sampling-rate: %s. Must be in range (0, 1]%n", samplingRate);
+ System.exit(1);
+ }
+ try
+ {
+ out.printf("Starting compression dictionary training for %s.%s...%n", keyspace, table);
+ out.printf("Will collect samples for up to %d seconds before training%n", maxSamplingDurationSeconds);
+ if (samplingRate != null)
+ {
+ out.printf("Using sampling rate: %.2f (%.1f%%)%n", samplingRate, samplingRate * 100);
+ }
+
+ // Build options map
+ Map<String, String> options = new HashMap<>();
+ options.put("maxSamplingDurationSeconds", String.valueOf(maxSamplingDurationSeconds));
+
+ probe.trainCompressionDictionary(keyspace, table, options);
+
+ // Update sampling rate if provided (after training has started)
+ if (samplingRate != null)
+ {
+ // Convert from double (0, 1] to integer format (1/rate)
+ // Examples: 1.0 -> 1 (sample every time), 0.5 -> 2 (roughly sample every 2nd), 0.1 -> 10 (roughly sample every 10th)
+ int integerSamplingRate = (int) Math.round(1.0 / samplingRate);
+ probe.updateCompressionDictionaryTrainingSamplingRate(keyspace, table, integerSamplingRate);
+ }
+
+ if (async)
+ {
+ out.printf("Training started asynchronously for %s.%s%n", keyspace, table);
+ out.printf("Use 'nodetool traincompressiondictionary --status %s %s' to check progress.%n",
+ keyspace, table);
+ return;
+ }
+
+ // Wait for completion (training will start automatically after sampling period)
+ out.println("Collecting samples and training. (Since the trainer samples chunk data on " +
+ "writing to new SSTable, you might consider running nodetool 'flush' along " +
+ "with this command to have chunk available for sampling)");
+ long maxWaitMillis = TimeUnit.SECONDS.toMillis(maxSamplingDurationSeconds + 300); // Add 5 minutes for training
+ long startTime = System.currentTimeMillis();
Review Comment:
Avoid System for time, should use `org.apache.cassandra.utils.Clock.Global` or `org.apache.cassandra.utils.Clock` interface [blockSystemClock]
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
+
+# Dictionary training configuration (advanced settings)
+# These settings control how compression dictionaries are trained from sample data.
+
+# Maximum size of a trained compression dictionary in bytes.
+# Larger dictionaries may provide better compression but use more memory.
+# Min unit: B
+compression_dictionary_training_max_dictionary_size: 65536
+
+# Maximum total size of sample data to collect for dictionary training.
+# More sample data generally produces better dictionaries but takes longer to train.
+# The recommended sample size is 100x the dictionary size.
+# Min unit: B
+compression_dictionary_training_max_total_sample_size: 10485760
+
+# Enable automatic dictionary training based on sampling of write operations.
+# When enabled, the system will automatically collect samples and train new dictionaries.
+# Manual training via nodetool is always available regardless of this setting.
+compression_dictionary_training_auto_train_enabled: false
+
+# Sampling rate for automatic dictionary training (1-10000).
+# Value of 100 means 1% of writes are sampled. Lower values reduce overhead but may
+# result in less representative sample data for dictionary training.
+compression_dictionary_training_sampling_rate: 100
Review Comment:
Does it really matter to be so granular to express 0.5% as "50"? The "problem" with this is that a reader has to manually re-compute the number in their head. Like 2.35% would be 235 and 0.8% would be 80.
It would be way more comfortable to put these figures in percentage already.
There is `data_disk_usage_percentage_warn_threshold` which operates with percentage conceptually. We can convert the percentage to whatever data type internally afterwards. So where it would be `1` instead of `100`, we would just multiply by 100 internally.
##########
src/java/org/apache/cassandra/db/compaction/unified/ShardedMultiWriter.java:
##########
@@ -118,6 +118,7 @@ private SSTableWriter createWriter(Descriptor descriptor)
.setSerializationHeader(header)
.addDefaultComponents(indexGroups)
.setSecondaryIndexGroups(indexGroups)
+ .setCompressionDictionaryManager(cfs.compressionDictionaryManager())
Review Comment:
who is "closing" this manager? Not so far in the review to figure it out on my own ... This is highlighted in IDEA as something we do not use try-with-resources on.
##########
src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java:
##########
@@ -329,6 +329,7 @@ protected long getExpectedWriteSize()
.setRepairedAt(minRepairedAt)
.setPendingRepair(pendingRepair)
.setSecondaryIndexGroups(cfs.indexManager.listIndexGroups())
- .addDefaultComponents(cfs.indexManager.listIndexGroups());
+ .addDefaultComponents(cfs.indexManager.listIndexGroups())
+ .setCompressionDictionaryManager(cfs.compressionDictionaryManager());
Review Comment:
same, who is closing this?
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
+ if (kind == Kind.ZSTD)
+ {
+ dictionary = new ZstdCompressionDictionary(dictId, dict);
+ }
+
+ if (dictionary == null)
+ {
+ throw new IOException(kind + " compression dictionary is not created");
+ }
+
+ // update the dictionary manager if it exists
+ if (manager != null)
+ {
+ manager.add(dictionary);
+ }
+ return dictionary;
+ }
+
+ static CompressionDictionary createFromRow(UntypedResultSet.Row row)
+ {
+ String kindStr = row.getString("kind");
+ long id = row.getLong("dict_id");
+ byte[] dict = row.getByteArray("dict");
+ CompressionDictionary.DictId dictId = new CompressionDictionary.DictId(CompressionDictionary.Kind.valueOf(kindStr), id);
Review Comment:
I think this whole logic is too complex. I would do this for Kind:
enum Kind
{
// Order matters: the enum ordinal is serialized
ZSTD
{
public CompressionDictionary getDictionary(DictId dictId, byte[] dict)
{
return new ZstdCompressionDictionary(dictId, dict);
}
};
public abstract CompressionDictionary getDictionary(CompressionDictionary.DictId dictId, byte[] dict);
}
And then
static CompressionDictionary createFromRow(UntypedResultSet.Row row)
{
String kindStr = row.getString("kind");
Kind kind;
try
{
kind = CompressionDictionary.Kind.valueOf(kindStr);
}
catch (IllegalArgumentException ex)
{
throw new IllegalStateException(kindStr + " compression dictionary is not created");
}
return kind.getDictionary(new DictId(kind, row.getLong("dict_id")), row.getByteArray("dict"));
}
The fact that we have successfully materialized `Kind` from `kind` string from `Row` pretty much guarantees that we can create a dictionary from it.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
+ if (kind == Kind.ZSTD)
+ {
+ dictionary = new ZstdCompressionDictionary(dictId, dict);
+ }
+
+ if (dictionary == null)
+ {
+ throw new IOException(kind + " compression dictionary is not created");
+ }
+
+ // update the dictionary manager if it exists
+ if (manager != null)
+ {
+ manager.add(dictionary);
+ }
+ return dictionary;
+ }
+
+ static CompressionDictionary createFromRow(UntypedResultSet.Row row)
+ {
+ String kindStr = row.getString("kind");
+ long id = row.getLong("dict_id");
+ byte[] dict = row.getByteArray("dict");
Review Comment:
We don't need this until `dictId.kind` is `ZSTD` so if it is not we at least do not materialize whole dictionary via means of `getByteArray` which creates `new byte[buf.remaining()]` and so on just to throw it away when we throw afterwards.
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryCache.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import javax.annotation.Nullable;
+
+/**
+ * Interface for managing compression dictionary caching and current dictionary state.
+ * <p>
+ * Implementations handle:
+ * - Local caching of compression dictionaries with automatic cleanup
+ * - Managing the current active dictionary for write operations
+ * - Thread-safe access to cached dictionaries
+ */
+public interface ICompressionDictionaryCache extends AutoCloseable
+{
+ /**
+ * Gets the current active compression dictionary.
+ *
+ * @return the current compression dictionary, or null if no dictionary is available
+ */
+ @Nullable
+ CompressionDictionary getCurrent();
Review Comment:
It would be better style to work with `Optional<CompressionDictionary>` than to work with `Nullable` annotations. Feels just more Java idiomatic than this. Applicable for `get` method too.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.time.Duration;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import org.apache.cassandra.config.DatabaseDescriptor;
+
+/**
+ * Manages caching and current dictionary state for compression dictionaries.
+ * <p>
+ * This class handles:
+ * - Local caching of compression dictionaries with automatic cleanup
+ * - Managing the current active dictionary for write operations
+ * - Thread-safe access to cached dictionaries
+ */
+public class CompressionDictionaryCache implements ICompressionDictionaryCache
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryCache.class);
+
+ private final Cache<CompressionDictionary.DictId, CompressionDictionary> cache;
+ private final AtomicReference<CompressionDictionary> currentDictionary = new AtomicReference<>();
+
+ public CompressionDictionaryCache()
+ {
+ Duration expiryTime = Duration.ofSeconds(DatabaseDescriptor.getCompressionDictionaryCacheExpireSeconds());
+ this.cache = Caffeine.newBuilder()
+ .maximumSize(DatabaseDescriptor.getCompressionDictionaryCacheSize())
+ .expireAfterAccess(expiryTime)
+ .removalListener((CompressionDictionary.DictId dictId,
+ CompressionDictionary dictionary,
+ RemovalCause cause) -> {
+ // Close dictionary when evicted from cache to free native resources
+ // SelfRefCounted ensures dictionary won't be actually closed if still referenced by compressors
+ if (dictionary != null)
+ {
+ try
+ {
+ dictionary.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close compression dictionary {}", dictId, e);
+ }
+ }
+ })
+ .build();
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return currentDictionary.get();
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.getIfPresent(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.put(compressionDictionary.identifier(), compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ if (dictionary == null)
+ return;
+
+ add(dictionary);
+ // Only update the current dictionary if we don't have one or the new one has a higher ID (newer)
+ CompressionDictionary current = currentDictionary.get();
+ while ((current == null || dictionary.identifier().id > current.identifier().id)
+ && !currentDictionary.compareAndSet(current, dictionary))
+ {
+ current = currentDictionary.get();
+ }
+ }
+
+ @Override
+ public synchronized void close()
Review Comment:
not sure if it is possible but what happens if somebody adds to this cache while this method is being executed. I think the fact this is synchronized just means that two threads can not execute `close` method at the same time but it does not mean that other thread could not call other methods in this cache.
Would not it be better to do the logic around closing `currentDictionary` in `removalListener` directly? Then we would just call `cache.invalidateAll();` and were done with it.
Also, what happens when we go to remove a dictionary by the means of `removalListener` while that dictionary is set in `currentDictionary`?
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
Review Comment:
I feel like this would be simplified if we did not put this logic around "isEnabled" here but we would pass isEnabled into `ICompressionDictionaryTrainer.create` and there, if it is `false`, we would return a dummy / no-op `ICompressionDictionaryTrainer` doing nothing. Then it would not matter that we would call `trainer.start(false)` on a dummy trainer and similar.
`this::handleNewDictionary` can be also part of `create` method and set internally, that's purely optional.
##########
src/java/org/apache/cassandra/io/compress/ZstdDictionaryCompressor.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.compress;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import com.github.luben.zstd.Zstd;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.ZstdCompressionDictionary;
+import org.apache.cassandra.db.compression.CompressionDictionary.Kind;
+import org.apache.cassandra.utils.concurrent.Ref;
+
+import javax.annotation.Nullable;
+
+public class ZstdDictionaryCompressor extends ZstdCompressorBase implements ICompressor, IDictionaryCompressor<ZstdCompressionDictionary>
Review Comment:
if `IDictionaryCompressor` extended `ICompressor` as suggested in `IDictionaryCompressor` interface, then `ZstdDictionaryCompressor` would implement `IDictionaryCompressor` only. `ZstdCompressorBase` implements `ICompressor` already so it is redundant here anyway.
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
Review Comment:
That means that all nodes would need to be started at the very same moment? Is this planned to be different per node in a real scenario? Because if all nodes have 10s then ... all of them would have same delay set, hence we would have the "simultaneous" situation again?
Maybe having _random initial delay_ in some _range_ (e.g. 5 to 20 seconds) would be better?
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
Review Comment:
checkstyle (`ant realclean && ant artifacts`) evaluated this as illegal import, see checkstyle.xml and grep `CompletableFuture` there to see. I am not completely sure yet but I think it should be replaced by something in `org.apache.cassandra.utils.concurrent` package.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryScheduler.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+
+/**
+ * Manages scheduled tasks for compression dictionary operations.
+ * <p>
+ * This class handles:
+ * - Periodic refresh of dictionaries from system tables
+ * - Manual training task scheduling and monitoring
+ * - Cleanup of scheduled tasks
+ */
+public class CompressionDictionaryScheduler implements ICompressionDictionaryScheduler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryScheduler.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ private volatile ScheduledFuture<?> scheduledRefreshTask;
+ private volatile ScheduledFuture<?> scheduledManualTrainingTask;
+ private volatile boolean isEnabled;
+
+ public CompressionDictionaryScheduler(String keyspaceName,
+ String tableName,
+ ICompressionDictionaryCache cache,
+ boolean isEnabled)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.cache = cache;
+ this.isEnabled = isEnabled;
+ }
+
+ /**
+ * Schedules the periodic dictionary refresh task if not already scheduled.
+ */
+ public void scheduleRefreshTask()
+ {
+ if (scheduledRefreshTask != null)
+ return;
+
+ this.scheduledRefreshTask = ScheduledExecutors.scheduledTasks.scheduleWithFixedDelay(
+ this::refreshDictionaryFromSystemTable,
+ DatabaseDescriptor.getCompressionDictionaryRefreshInitialDelaySeconds(),
+ DatabaseDescriptor.getCompressionDictionaryRefreshIntervalSeconds(),
+ TimeUnit.SECONDS
+ );
+ }
+
+ @Override
+ public void scheduleManualTraining(ManualTrainingOptions options, ICompressionDictionaryTrainer trainer)
+ {
+ if (scheduledManualTrainingTask != null)
+ {
+ throw new IllegalStateException("Training already in progress for table " + keyspaceName + '.' + tableName);
+ }
+
+ int maxSamplingDurationSeconds = options.getMaxSamplingDurationSeconds();
+
+ logger.info("Starting manual dictionary training for {}.{} with max sampling duration: {} seconds",
+ keyspaceName, tableName, maxSamplingDurationSeconds);
+
+ long deadlineMillis = System.currentTimeMillis() + TimeUnit.SECONDS.toMillis(maxSamplingDurationSeconds);
Review Comment:
Avoid System for time, should use `org.apache.cassandra.utils.Clock.Global` or `org.apache.cassandra.utils.Clock` interface [blockSystemClock]
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryTrainer.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.CompletableFuture;
Review Comment:
illegal import
##########
src/java/org/apache/cassandra/db/compression/ZstdDictionaryTrainer.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.luben.zstd.Zstd;
+import com.github.luben.zstd.ZstdDictTrainer;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.CompressionDictionary.DictId;
+import org.apache.cassandra.db.compression.CompressionDictionary.Kind;
+import org.apache.cassandra.io.compress.IDictionaryCompressor;
+import org.apache.cassandra.io.compress.ZstdDictionaryCompressor;
+import org.apache.cassandra.schema.CompressionParams;
+
+/**
+ * Zstd implementation of dictionary trainer with lifecycle management.
+ */
+public class ZstdDictionaryTrainer implements ICompressionDictionaryTrainer
+{
+ private static final Logger logger = LoggerFactory.getLogger(ZstdDictionaryTrainer.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final CompressionDictionaryTrainingConfig config;
+ private final AtomicLong totalSampleSize;
+ private final AtomicLong sampleCount;
+ private final int compressionLevel; // optimal if using the same level for training as when compressing.
+
+ // Sampling rate can be updated during training
+ private volatile int samplingRate;
+
+ // Minimum number of samples required by ZSTD library
+ private static final int MIN_SAMPLES_REQUIRED = 10;
+
+ private volatile Consumer<CompressionDictionary> dictionaryTrainedListener;
+ // TODO: manage the samples in this class for auto-train (follow-up). The ZstdDictTrainer cannot be re-used for multiple training runs.
+ private ZstdDictTrainer zstdTrainer;
+ private volatile boolean closed = false;
+ private volatile TrainingStatus currentTrainingStatus;
+
+ public ZstdDictionaryTrainer(String keyspaceName, String tableName,
+ CompressionDictionaryTrainingConfig config,
+ int compressionLevel)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.config = config;
+ this.totalSampleSize = new AtomicLong(0);
+ this.sampleCount = new AtomicLong(0);
+ this.compressionLevel = compressionLevel;
+ this.samplingRate = config.samplingRate;
+ this.currentTrainingStatus = TrainingStatus.NOT_STARTED;
+ }
+
+ @Override
+ public boolean shouldSample()
+ {
+ return zstdTrainer != null && ThreadLocalRandom.current().nextInt(samplingRate) == 0;
+ }
+
+ @Override
+ public void addSample(ByteBuffer sample)
+ {
+ if (closed || sample == null || !sample.hasRemaining() || zstdTrainer == null)
+ return;
+
+ byte[] sampleBytes = new byte[sample.remaining()];
+ sample.duplicate().get(sampleBytes);
+
+ if (zstdTrainer.addSample(sampleBytes))
+ {
+ // Update the totalSampleSize and sampleCount if the sample is added
+ totalSampleSize.addAndGet(sampleBytes.length);
+ sampleCount.incrementAndGet();
+ }
+ }
+
+ @Override
+ public CompressionDictionary trainDictionary(boolean force)
+ {
+ boolean isReady = isReady();
+ if (!force && !isReady)
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ throw new IllegalStateException("Trainer is not ready");
+ }
+
+ long currentSampleCount = sampleCount.get();
+ if (currentSampleCount < MIN_SAMPLES_REQUIRED) // minimum samples should be required even if force training
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ String errorMsg = String.format("Insufficient samples for training: %d (minimum required: %d)",
+ currentSampleCount, MIN_SAMPLES_REQUIRED);
+ throw new IllegalStateException(errorMsg);
+ }
+
+ currentTrainingStatus = TrainingStatus.TRAINING;
+ try
+ {
+ logger.debug("Training with sample count: {}, sample size: {}, isReady: {}",
+ currentSampleCount, totalSampleSize.get(), isReady);
+ byte[] dictBytes = zstdTrainer.trainSamples();
+ long zstdDictId = Zstd.getDictIdFromDict(dictBytes);
+ DictId dictId = new DictId(Kind.ZSTD, makeDictionaryId(System.currentTimeMillis(), zstdDictId));
Review Comment:
Avoid System for time, should use `org.apache.cassandra.utils.Clock.Globa`l or `org.apache.cassandra.utils.Clock` interface [blockSystemClock]
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+public interface ICompressionDictionaryEventHandler
Review Comment:
As suggested elsewhere we might use `DictId` here instead for both methods.
##########
src/java/org/apache/cassandra/db/compression/ZstdCompressionDictionary.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.luben.zstd.ZstdDictCompress;
+import com.github.luben.zstd.ZstdDictDecompress;
+import org.apache.cassandra.io.compress.ZstdCompressorBase;
+import org.apache.cassandra.utils.concurrent.Ref;
+import org.apache.cassandra.utils.concurrent.RefCounted;
+import org.apache.cassandra.utils.concurrent.SelfRefCounted;
+
+public class ZstdCompressionDictionary implements CompressionDictionary, SelfRefCounted<ZstdCompressionDictionary>
+{
+ private static final Logger logger = LoggerFactory.getLogger(ZstdCompressionDictionary.class);
+
+ private final DictId dictId;
+ private final byte[] rawDictionary;
+ // One ZstdDictDecompress and multiple ZstdDictCompress (per level) can be derived from the same raw dictionary content
+ private final ConcurrentHashMap<Integer, ZstdDictCompress> zstdDictCompressPerLevel = new ConcurrentHashMap<>();
+ private volatile ZstdDictDecompress dictDecompress;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Ref<ZstdCompressionDictionary> selfRef;
+
+ public ZstdCompressionDictionary(DictId dictId, byte[] rawDictionary)
+ {
+ this.dictId = dictId;
+ this.rawDictionary = rawDictionary;
+ this.selfRef = new Ref<>(this, new Tidy(zstdDictCompressPerLevel, dictDecompress));
+ }
+
+ @Override
+ public DictId identifier()
+ {
+ return dictId;
+ }
+
+ @Override
+ public Kind kind()
+ {
+ return Kind.ZSTD;
+ }
+
+ @Override
+ public byte[] rawDictionary()
+ {
+ return rawDictionary;
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (!(o instanceof ZstdCompressionDictionary)) return false;
+ ZstdCompressionDictionary that = (ZstdCompressionDictionary) o;
+ return Objects.equals(dictId, that.dictId);
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return dictId.hashCode();
+ }
+
+ /**
+ * Get a pre-processed compression tables that is optimized for compression.
+ * It is derived/computed from dictionary bytes.
+ * The internal data structure is different from the tables for decompression.
+ *
+ * @param compressionLevel compression level to create the compression table
+ * @return ZstdDictCompress
+ */
+ public ZstdDictCompress dictionaryForCompression(int compressionLevel)
+ {
+ if (closed.get())
+ throw new IllegalStateException("Dictionary has been closed");
+
+ ZstdCompressorBase.validateCompressionLevel(compressionLevel);
+
+ return zstdDictCompressPerLevel.computeIfAbsent(compressionLevel, level -> {
+ if (closed.get())
+ throw new IllegalStateException("Dictionary has been closed");
+ return new ZstdDictCompress(rawDictionary, level);
+ });
+ }
+
+ /**
+ * Get a pre-processed decompression tables that is optimized for decompression.
+ * It is derived/computed from dictionary bytes.
+ * The internal data structure is different from the tables for compression.
+ *
+ * @return ZstdDictDecompress
+ */
+ public ZstdDictDecompress dictionaryForDecompression()
+ {
+ if (closed.get())
+ throw new IllegalStateException("Dictionary has been closed");
+
+ ZstdDictDecompress result = dictDecompress;
+ if (result != null)
+ return result;
+
+ synchronized (this)
+ {
+ if (closed.get())
+ throw new IllegalStateException("Dictionary has been closed");
+
+ result = dictDecompress;
+ if (result == null)
+ {
+ result = new ZstdDictDecompress(rawDictionary);
+ dictDecompress = result;
+ }
+ return result;
+ }
+ }
+
+ @Override
+ public Ref<ZstdCompressionDictionary> tryRef()
+ {
+ return selfRef.tryRef();
+ }
+
+ @Override
+ public Ref<ZstdCompressionDictionary> selfRef()
+ {
+ return selfRef;
+ }
+
+ @Override
+ public Ref<ZstdCompressionDictionary> ref()
+ {
+ return selfRef.ref();
+ }
+
+ @Override
+ public void close()
+ {
+ if (closed.compareAndSet(false, true))
+ {
+ selfRef.release();
+ }
+ }
+
+ private static class Tidy implements RefCounted.Tidy
+ {
+ private final ConcurrentHashMap<Integer, ZstdDictCompress> zstdDictCompressPerLevel;
+ private volatile ZstdDictDecompress dictDecompress;
+
+ Tidy(ConcurrentHashMap<Integer, ZstdDictCompress> zstdDictCompressPerLevel, ZstdDictDecompress dictDecompress)
+ {
+ this.zstdDictCompressPerLevel = zstdDictCompressPerLevel;
+ this.dictDecompress = dictDecompress;
+ }
+
+ @Override
+ public void tidy()
+ {
+ // Close all compression dictionaries
+ for (ZstdDictCompress compressDict : zstdDictCompressPerLevel.values())
+ {
+ try
+ {
+ compressDict.close();
+ }
+ catch (Exception e)
+ {
+ // Log but don't fail - continue closing other resources
+ logger.warn("Failed to close ZstdDictCompress", e);
+ }
+ }
+ zstdDictCompressPerLevel.clear();
+
+ // Close decompression dictionary
+ ZstdDictDecompress decompressDict = dictDecompress;
+ if (decompressDict != null)
+ {
+ try
+ {
+ decompressDict.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close ZstdDictDecompress", e);
+ }
+ dictDecompress = null;
+ }
+ }
+
+ @Override
+ public String name()
+ {
+ return "ZstdCompressionDictionary";
Review Comment:
`ZstdCompressionDictionary.class.getSimpleName()`.
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
Review Comment:
I am also not completely sure this is the right section of the yaml to put all of this in. Wondering based on what it was put here. Not saying it is wrong, just saying that this is probably the opportunity to be more thoughtful and intentional about where we put it. Given this is probably experimental feature we might also consider to put it among experimental ones at the end of the yaml? Just shooting thoughts.
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
+
+# Dictionary training configuration (advanced settings)
+# These settings control how compression dictionaries are trained from sample data.
+
+# Maximum size of a trained compression dictionary in bytes.
+# Larger dictionaries may provide better compression but use more memory.
+# Min unit: B
+compression_dictionary_training_max_dictionary_size: 65536
Review Comment:
could this be converted to `64KiB`? I think that is more expressive then just a number. That would lead to `DataStorageSpec.IntKibibytesBound` type of compression_dictionary_training_max_dictionary_size. By doing it like this, people are really forced to put there just integers and nothing else. This is in general not a style which is used through this yaml where data / capacity related configuration parameters are descendants of `DataStorageSpec`.
Same way you used `DurationSpec.IntSecondsBound` for properties above, we should use `DataStorageSpec` for these.
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
Review Comment:
are we really going to use camel case? not snake case? not sure what is better or what we use consistently. Also we might probably extract this string to a standalone constant in this class. Easy to mess it up by typing it constantly.
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
Review Comment:
probably safe just saying the fact that this is same (3600s) as `compression_dictionary_refresh_interval` feels like there might be some kind of a problem when both periods ends at the same time.
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
Review Comment:
consider `Preconditions.checkArgument` here
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
Review Comment:
We most probably need to copy all of this to cassandra_latest.yaml as well.
##########
src/java/org/apache/cassandra/io/compress/CompressionMetadata.java:
##########
@@ -139,11 +165,54 @@ private CompressionMetadata(CompressionMetadata copy)
this.compressedFileLength = copy.compressedFileLength;
this.chunkOffsets = copy.chunkOffsets;
this.chunkOffsetsSize = copy.chunkOffsetsSize;
+ this.compressionDictionary = copy.compressionDictionary;
}
public ICompressor compressor()
{
- return parameters.getSstableCompressor();
+ // classic double-checked locking to call resolveCompressor method just once per CompressionMetadata object
+ ICompressor result = resolvedCompressor;
+ if (result != null)
+ return result;
+
+ synchronized (this)
+ {
+ result = resolvedCompressor;
+ if (result == null)
+ {
+ result = resolveCompressor(parameters.getSstableCompressor(), compressionDictionary);
+ resolvedCompressor = result;
+ }
+ return result;
+ }
+ }
+
+ static ICompressor resolveCompressor(ICompressor compressor, CompressionDictionary dictionary)
+ {
+ if (dictionary == null)
+ return compressor;
+
+ // When the attached dictionary can be consumed by the current dictionary compressor
+ if (compressor instanceof IDictionaryCompressor)
+ {
+ IDictionaryCompressor dictionaryCompressor = (IDictionaryCompressor) compressor;
+ if (dictionaryCompressor.canConsumeDictionary(dictionary))
+ return dictionaryCompressor.getOrCopyWithDictionary(dictionary);
+ }
+
+ // When the current compressor is not compatible with the dictionary. It could happen in the read path when:
+ // 1. The current compressor is not a dictionary compressor, but there is dictionary attached
+ // 2. The current dictionary compressor is a different type, e.g. table schema is changed
+ // In those cases, we should get the compatible dictionary compressor based on the dictionary
+ if (dictionary.kind() == ZSTD)
Review Comment:
We are again too "ZSTD-centric" here. I would expect this does not contain anything ZSTD specific so we do not need to touch this anymore. The fact that dictionary is not empty again pretty much guarantees that we can create a compressor from it, no? Why would we have a dictionary, of `CompressionDictionary` type, if we were not able to create a compressor from it? That would be a pretty useless dictionary to me.
So maybe doing something like
`return dictionary.kind().getCompressor(dictionary);`
and calling `ZstdDictionaryCompressor.create(zstdDict)` in there would be way more simpler and at one place.
You would need to return `ICompressor` from `getCompressor` as `resolveCompressor` returns it too. If `IDictionaryCompressor` extended `ICompressor` as already suggested elsewhere (as you need to instantiate `ZstdDictionaryCompressor` there) it would be a piece of cake.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
Review Comment:
The problem I see with the current logic is that we are too "aware" of ZSTD. If this is designed to be "future proof" when other dictionary compressors can be added, then the logic should not be "wired" around ZSTD there. It should not _matter_ what `Kind` we got. Why would it ... I just want to add a new Kind and overload its "getDictionary". I do not need to touch this code anymore.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
Review Comment:
I think it is `long` because that is used as something like a timestamp?
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
Review Comment:
nit: braces not necessary
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
Review Comment:
Do we really need `long` for this? Hard to imagine that `DictId.id` would be bigger than int which is 2,147,483,647 max.
##########
src/java/org/apache/cassandra/io/compress/IDictionaryCompressor.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.compress;
+
+import org.apache.cassandra.db.compression.CompressionDictionary;
+
+/**
+ * Interface for compressors that support dictionary-based compression.
+ * <br>
+ * Dictionary compressors can use pre-trained compression dictionaries to achieve
+ * better compression ratios, especially for small data chunks that are similar
+ * to the training data used to create the dictionary.
+ *
+ * @param <T> the specific type of compression dictionary this compressor supports
+ */
+public interface IDictionaryCompressor<T extends CompressionDictionary>
Review Comment:
https://github.com/apache/cassandra/pull/4399/files#r2410732332
##########
src/java/org/apache/cassandra/db/compaction/CompactionManager.java:
##########
@@ -1796,6 +1796,7 @@ public static SSTableWriter createWriter(ColumnFamilyStore cfs,
.setSerializationHeader(sstable.header)
.addDefaultComponents(cfs.indexManager.listIndexGroups())
.setSecondaryIndexGroups(cfs.indexManager.listIndexGroups())
+ .setCompressionDictionaryManager(cfs.compressionDictionaryManager())
Review Comment:
closing anywhere?
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
Review Comment:
if I understand this correctly, it basically assumes that additions of new dictionary kinds will be only ever appended at the end of the `Kind` enum. Basically we can not put anything in the middle because ordinals would be shifted. I see you put the comment about this in Kind enum which is important, thank you for that.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
Review Comment:
nit: braces not necessary and we might include all possible Kind.values() (ordinals) in this message as well
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
Review Comment:
nit: braces not necessary
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
Review Comment:
include both values of `checksum` and `calculatedChecksum` in the message perhaps. A reader of such a message happens to wonder what was computed and what was serialized.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.time.Duration;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import org.apache.cassandra.config.DatabaseDescriptor;
+
+/**
+ * Manages caching and current dictionary state for compression dictionaries.
+ * <p>
+ * This class handles:
+ * - Local caching of compression dictionaries with automatic cleanup
+ * - Managing the current active dictionary for write operations
+ * - Thread-safe access to cached dictionaries
+ */
+public class CompressionDictionaryCache implements ICompressionDictionaryCache
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryCache.class);
+
+ private final Cache<CompressionDictionary.DictId, CompressionDictionary> cache;
+ private final AtomicReference<CompressionDictionary> currentDictionary = new AtomicReference<>();
+
+ public CompressionDictionaryCache()
+ {
+ Duration expiryTime = Duration.ofSeconds(DatabaseDescriptor.getCompressionDictionaryCacheExpireSeconds());
Review Comment:
can be just inlined into `expireAfterAccess`?
.expireAfterAccess(Duration.ofSeconds(DatabaseDescriptor.getCompressionDictionaryCacheExpireSeconds()))
We are not making special variable for `DatabaseDescriptor.getCompressionDictionaryCacheSize()` for `maximumSize` method.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
Review Comment:
This is also too complicated I think. Similarly as below:
int length = input.readInt();
byte[] dict = new byte[length];
input.readFully(dict);
int checksum = input.readInt();
int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
if (checksum != calculatedChecksum)
throw new IOException("Compression dictionary checksum does not match");
// here we are guaranteed to have a dictionary, no?
CompressionDictionary dictionary = kind.getDictionary(dictId, dict);
// update the dictionary manager if it exists
if (manager != null)
manager.add(dictionary);
return dictionary;
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Handles compression dictionary events including training completion and cluster notifications.
+ * <p>
+ * This class handles:
+ * - Broadcasting dictionary updates to cluster nodes
+ * - Retrieving new dictionaries when notified by other nodes
+ * - Managing dictionary cache updates
+ */
+public class CompressionDictionaryEventHandler implements ICompressionDictionaryEventHandler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryEventHandler.class);
+
+ private final ColumnFamilyStore cfs;
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ public CompressionDictionaryEventHandler(ColumnFamilyStore cfs, ICompressionDictionaryCache cache)
+ {
+ this.cfs = cfs;
+ this.keyspaceName = cfs.keyspace.getName();
+ this.tableName = cfs.getTableName();
+ this.cache = cache;
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ logger.info("Notifying cluster about dictionary update for {}.{} with dictionaryId {}",
+ keyspaceName, tableName, dictionaryId);
+
+ CompressionDictionaryUpdateMessage message = new CompressionDictionaryUpdateMessage(cfs.metadata().id, dictionaryId);
+ Collection<InetAddressAndPort> allNodes = ClusterMetadata.current().directory.allJoinedEndpoints();
+ // Broadcast notification using the fire-and-forget fashion
+ for (InetAddressAndPort node : allNodes)
+ {
+ if (node.equals(FBUtilities.getBroadcastAddressAndPort())) // skip ourself
+ continue;
+ sendNotification(node, message);
+ }
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictId)
+ {
+ // Best effort to retrieve the dictionary; otherwise, the periodic task should retrieve the dictionary later
+ CompletableFuture.runAsync(() -> {
+ try
+ {
+ if (!cfs.metadata().params.compression.isDictionaryCompressionEnabled())
+ {
Review Comment:
braces not necessary
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManagerMBean.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+public interface CompressionDictionaryManagerMBean
+{
+ /**
+ * Starts sampling and training for this table.
+ *
+ * @param options options for the training process (currently unused, reserved for future extensions)
+ * @throws IllegalArgumentException if table doesn't support dictionary compression
Review Comment:
Throw `UnsupportedOperationException` then?
##########
src/java/org/apache/cassandra/io/compress/IDictionaryCompressor.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.compress;
+
+import org.apache.cassandra.db.compression.CompressionDictionary;
+
+/**
+ * Interface for compressors that support dictionary-based compression.
+ * <br>
+ * Dictionary compressors can use pre-trained compression dictionaries to achieve
+ * better compression ratios, especially for small data chunks that are similar
+ * to the training data used to create the dictionary.
+ *
+ * @param <T> the specific type of compression dictionary this compressor supports
+ */
+public interface IDictionaryCompressor<T extends CompressionDictionary>
Review Comment:
interesting, should not be `IDictionaryCompressor` extending `ICompressor`? Because in `CompressionParams` there is
private final ICompressor sstableCompressor;
and
public boolean isDictionaryCompressionEnabled()
{
return isEnabled() && sstableCompressor instanceof IDictionaryCompressor;
}
Which is a little bit confusing to me.
I think that `IDictionaryCompressor` extending `ICompressor` is way more conceptually logical.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Handles compression dictionary events including training completion and cluster notifications.
+ * <p>
+ * This class handles:
+ * - Broadcasting dictionary updates to cluster nodes
+ * - Retrieving new dictionaries when notified by other nodes
+ * - Managing dictionary cache updates
+ */
+public class CompressionDictionaryEventHandler implements ICompressionDictionaryEventHandler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryEventHandler.class);
+
+ private final ColumnFamilyStore cfs;
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ public CompressionDictionaryEventHandler(ColumnFamilyStore cfs, ICompressionDictionaryCache cache)
+ {
+ this.cfs = cfs;
+ this.keyspaceName = cfs.keyspace.getName();
+ this.tableName = cfs.getTableName();
+ this.cache = cache;
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ logger.info("Notifying cluster about dictionary update for {}.{} with dictionaryId {}",
+ keyspaceName, tableName, dictionaryId);
+
+ CompressionDictionaryUpdateMessage message = new CompressionDictionaryUpdateMessage(cfs.metadata().id, dictionaryId);
+ Collection<InetAddressAndPort> allNodes = ClusterMetadata.current().directory.allJoinedEndpoints();
+ // Broadcast notification using the fire-and-forget fashion
+ for (InetAddressAndPort node : allNodes)
+ {
+ if (node.equals(FBUtilities.getBroadcastAddressAndPort())) // skip ourself
+ continue;
+ sendNotification(node, message);
+ }
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictId)
+ {
+ // Best effort to retrieve the dictionary; otherwise, the periodic task should retrieve the dictionary later
+ CompletableFuture.runAsync(() -> {
+ try
+ {
+ if (!cfs.metadata().params.compression.isDictionaryCompressionEnabled())
+ {
+ return;
+ }
+
+ CompressionDictionary dictionary = SystemDistributedKeyspace.retrieveCompressionDictionary(keyspaceName, tableName, dictId);
+ cache.setCurrentIfNewer(dictionary);
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to retrieve compression dictionary for {}.{}. dictionaryId={}",
+ keyspaceName, tableName, dictId, e);
+ }
+ }, ScheduledExecutors.nonPeriodicTasks);
+ }
+
+ // Best effort to notify the peer regarding the new dictionary being available to pull.
+ // If the request fails, each peer has periodic task scheduled to pull.
+ private void sendNotification(InetAddressAndPort target, CompressionDictionaryUpdateMessage message)
+ {
+ logger.debug("Sending dictionary update notification to {}", target);
Review Comment:
Could we be more verbose? Dictionary of what kind and id? Here `DictId` would come handy. Same for all other debug messages, like in the listener below.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Handles compression dictionary events including training completion and cluster notifications.
+ * <p>
+ * This class handles:
+ * - Broadcasting dictionary updates to cluster nodes
+ * - Retrieving new dictionaries when notified by other nodes
+ * - Managing dictionary cache updates
+ */
+public class CompressionDictionaryEventHandler implements ICompressionDictionaryEventHandler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryEventHandler.class);
+
+ private final ColumnFamilyStore cfs;
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ public CompressionDictionaryEventHandler(ColumnFamilyStore cfs, ICompressionDictionaryCache cache)
+ {
+ this.cfs = cfs;
+ this.keyspaceName = cfs.keyspace.getName();
+ this.tableName = cfs.getTableName();
+ this.cache = cache;
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
Review Comment:
dont we need to include `DictId`, actually? That also happens to contain `Kind`. I think that is way more descriptive and robust.
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
Can not that be some default when missing? Seems like a hassle to be force to include it all the time.
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
Review Comment:
nit: braces not necessary at some places
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
Review Comment:
this happens to be extracted into a constant like
public static final String MBEAN_NAME = "org.apache.cassandra.hints:type=HintsService";
for other MBeans. I would also move that constant to MBean interface itself and just referenced here.
##########
src/java/org/apache/cassandra/io/compress/ZstdDictionaryCompressor.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.compress;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import com.github.luben.zstd.Zstd;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.ZstdCompressionDictionary;
+import org.apache.cassandra.db.compression.CompressionDictionary.Kind;
+import org.apache.cassandra.utils.concurrent.Ref;
+
+import javax.annotation.Nullable;
+
+public class ZstdDictionaryCompressor extends ZstdCompressorBase implements ICompressor, IDictionaryCompressor<ZstdCompressionDictionary>
+{
+ private static final ConcurrentHashMap<Integer, ZstdDictionaryCompressor> instancesPerLevel = new ConcurrentHashMap<>();
+ private static final Cache<ZstdCompressionDictionary, ZstdDictionaryCompressor> instancePerDict =
+ Caffeine.newBuilder()
+ .maximumSize(DatabaseDescriptor.getCompressionDictionaryCacheSize())
+ .expireAfterAccess(Duration.ofSeconds(DatabaseDescriptor.getCompressionDictionaryCacheExpireSeconds()))
+ .removalListener((ZstdCompressionDictionary dictionary,
+ ZstdDictionaryCompressor compressor,
+ RemovalCause cause) -> {
+ // Release dictionary reference when compressor is evicted from cache
+ if (compressor != null && compressor.dictionaryRef != null)
+ {
+ compressor.dictionaryRef.release();
+ }
+ })
+ .build();
+
+ // dictioanry and its ref are null, when they are absent.
Review Comment:
typo in `dictioanry`.
##########
src/java/org/apache/cassandra/db/compression/ZstdCompressionDictionary.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.luben.zstd.ZstdDictCompress;
+import com.github.luben.zstd.ZstdDictDecompress;
+import org.apache.cassandra.io.compress.ZstdCompressorBase;
+import org.apache.cassandra.utils.concurrent.Ref;
+import org.apache.cassandra.utils.concurrent.RefCounted;
+import org.apache.cassandra.utils.concurrent.SelfRefCounted;
+
+public class ZstdCompressionDictionary implements CompressionDictionary, SelfRefCounted<ZstdCompressionDictionary>
+{
+ private static final Logger logger = LoggerFactory.getLogger(ZstdCompressionDictionary.class);
+
+ private final DictId dictId;
+ private final byte[] rawDictionary;
+ // One ZstdDictDecompress and multiple ZstdDictCompress (per level) can be derived from the same raw dictionary content
+ private final ConcurrentHashMap<Integer, ZstdDictCompress> zstdDictCompressPerLevel = new ConcurrentHashMap<>();
+ private volatile ZstdDictDecompress dictDecompress;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Ref<ZstdCompressionDictionary> selfRef;
+
+ public ZstdCompressionDictionary(DictId dictId, byte[] rawDictionary)
+ {
+ this.dictId = dictId;
+ this.rawDictionary = rawDictionary;
+ this.selfRef = new Ref<>(this, new Tidy(zstdDictCompressPerLevel, dictDecompress));
+ }
+
+ @Override
+ public DictId identifier()
+ {
+ return dictId;
+ }
+
+ @Override
+ public Kind kind()
+ {
+ return Kind.ZSTD;
+ }
+
+ @Override
+ public byte[] rawDictionary()
+ {
+ return rawDictionary;
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (!(o instanceof ZstdCompressionDictionary)) return false;
+ ZstdCompressionDictionary that = (ZstdCompressionDictionary) o;
+ return Objects.equals(dictId, that.dictId);
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return dictId.hashCode();
+ }
+
+ /**
+ * Get a pre-processed compression tables that is optimized for compression.
+ * It is derived/computed from dictionary bytes.
+ * The internal data structure is different from the tables for decompression.
+ *
+ * @param compressionLevel compression level to create the compression table
+ * @return ZstdDictCompress
+ */
+ public ZstdDictCompress dictionaryForCompression(int compressionLevel)
+ {
+ if (closed.get())
+ throw new IllegalStateException("Dictionary has been closed");
Review Comment:
maybe being more descriptive in this message (and all other same messages) by including dict id with kind and id would be nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414917768
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryScheduler.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+
+/**
+ * Manages scheduled tasks for compression dictionary operations.
+ * <p>
+ * This class handles:
+ * - Periodic refresh of dictionaries from system tables
+ * - Manual training task scheduling and monitoring
+ * - Cleanup of scheduled tasks
+ */
+public class CompressionDictionaryScheduler implements ICompressionDictionaryScheduler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryScheduler.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ private volatile ScheduledFuture<?> scheduledRefreshTask;
+ private volatile ScheduledFuture<?> scheduledManualTrainingTask;
+ private volatile boolean isEnabled;
+
+ public CompressionDictionaryScheduler(String keyspaceName,
+ String tableName,
+ ICompressionDictionaryCache cache,
+ boolean isEnabled)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.cache = cache;
+ this.isEnabled = isEnabled;
+ }
+
+ /**
+ * Schedules the periodic dictionary refresh task if not already scheduled.
+ */
+ public void scheduleRefreshTask()
+ {
+ if (scheduledRefreshTask != null)
+ return;
+
+ this.scheduledRefreshTask = ScheduledExecutors.scheduledTasks.scheduleWithFixedDelay(
+ this::refreshDictionaryFromSystemTable,
+ DatabaseDescriptor.getCompressionDictionaryRefreshInitialDelaySeconds(),
+ DatabaseDescriptor.getCompressionDictionaryRefreshIntervalSeconds(),
+ TimeUnit.SECONDS
+ );
+ }
+
+ @Override
+ public void scheduleManualTraining(ManualTrainingOptions options, ICompressionDictionaryTrainer trainer)
+ {
+ if (scheduledManualTrainingTask != null)
+ {
+ throw new IllegalStateException("Training already in progress for table " + keyspaceName + '.' + tableName);
+ }
+
+ int maxSamplingDurationSeconds = options.getMaxSamplingDurationSeconds();
+
+ logger.info("Starting manual dictionary training for {}.{} with max sampling duration: {} seconds",
+ keyspaceName, tableName, maxSamplingDurationSeconds);
+
+ long deadlineMillis = System.currentTimeMillis() + TimeUnit.SECONDS.toMillis(maxSamplingDurationSeconds);
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414966140
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
Review Comment:
`compression_dictionary_cache_expire` and `compression_dictionary_refresh_interval` are not strongly connected, since they apply to different components. Cache entry expiry is renewed on every access. Meanwhile, `compression_dictionary_refresh_interval` only defines how frequent to read from system table.
If there is no access to the cache entry for 1 hour, it is evicted. The chance is low that the dictionary is wanted right after eviction. Even in such scenario, the cache entry is again populated from compressionInfo or system table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410701055
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
Review Comment:
I feel like this would be simplified if we did not put this logic around "isEnabled" here but we would pass isEnabled into `ICompressionDictionaryTrainer.create` and there, if it is `false`, we would return a dummy / no-op (singleton) `ICompressionDictionaryTrainer` doing nothing. Then it would not matter that we would call `trainer.start(false)` on a dummy trainer and similar, we would not need to check if a trainer is null or not further in the code.
`this::handleNewDictionary` can be also part of `create` method and set internally, that's purely optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410570729
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
Review Comment:
I think it is `long` because that is used as something like a timestamp in `makeDictionaryId`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2411740844
##########
src/java/org/apache/cassandra/io/compress/CompressionMetadata.java:
##########
@@ -139,11 +165,54 @@ private CompressionMetadata(CompressionMetadata copy)
this.compressedFileLength = copy.compressedFileLength;
this.chunkOffsets = copy.chunkOffsets;
this.chunkOffsetsSize = copy.chunkOffsetsSize;
+ this.compressionDictionary = copy.compressionDictionary;
}
public ICompressor compressor()
{
- return parameters.getSstableCompressor();
+ // classic double-checked locking to call resolveCompressor method just once per CompressionMetadata object
+ ICompressor result = resolvedCompressor;
+ if (result != null)
+ return result;
+
+ synchronized (this)
+ {
+ result = resolvedCompressor;
+ if (result == null)
+ {
+ result = resolveCompressor(parameters.getSstableCompressor(), compressionDictionary);
+ resolvedCompressor = result;
+ }
+ return result;
+ }
+ }
+
+ static ICompressor resolveCompressor(ICompressor compressor, CompressionDictionary dictionary)
+ {
+ if (dictionary == null)
+ return compressor;
+
+ // When the attached dictionary can be consumed by the current dictionary compressor
+ if (compressor instanceof IDictionaryCompressor)
+ {
+ IDictionaryCompressor dictionaryCompressor = (IDictionaryCompressor) compressor;
+ if (dictionaryCompressor.canConsumeDictionary(dictionary))
+ return dictionaryCompressor.getOrCopyWithDictionary(dictionary);
+ }
+
+ // When the current compressor is not compatible with the dictionary. It could happen in the read path when:
+ // 1. The current compressor is not a dictionary compressor, but there is dictionary attached
+ // 2. The current dictionary compressor is a different type, e.g. table schema is changed
+ // In those cases, we should get the compatible dictionary compressor based on the dictionary
+ if (dictionary.kind() == ZSTD)
Review Comment:
or even better
return dictionary.getCompressor()
which would be like
public IDictionaryCompressor getCompressor()
{
return kind().getCompressor(this);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2421352645
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
Review Comment:
@rustyrazorblade, I am not sure if I understand your comment. I would like to keep the settings as they are for now and not introduce randomized delay unless we see an actual need. If the default interval turns out to be sub-optimal, we can adjust that directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410402574
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
Review Comment:
nit: braces not necessary and we might include all possible Kind.values() (ordinals) in this message as well
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
Review Comment:
nit: braces not necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410508310
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
+ if (kind == Kind.ZSTD)
+ {
+ dictionary = new ZstdCompressionDictionary(dictId, dict);
+ }
+
+ if (dictionary == null)
+ {
+ throw new IOException(kind + " compression dictionary is not created");
+ }
+
+ // update the dictionary manager if it exists
+ if (manager != null)
+ {
+ manager.add(dictionary);
+ }
+ return dictionary;
+ }
+
+ static CompressionDictionary createFromRow(UntypedResultSet.Row row)
+ {
+ String kindStr = row.getString("kind");
+ long id = row.getLong("dict_id");
+ byte[] dict = row.getByteArray("dict");
+ CompressionDictionary.DictId dictId = new CompressionDictionary.DictId(CompressionDictionary.Kind.valueOf(kindStr), id);
Review Comment:
I think this whole logic is too complex. I would do this for Kind:
enum Kind
{
// Order matters: the enum ordinal is serialized
ZSTD
{
public CompressionDictionary getDictionary(DictId dictId, byte[] dict)
{
return new ZstdCompressionDictionary(dictId, dict);
}
};
public abstract CompressionDictionary getDictionary(CompressionDictionary.DictId dictId, byte[] dict);
}
And then
static CompressionDictionary createFromRow(UntypedResultSet.Row row)
{
String kindStr = row.getString("kind");
Kind kind;
try
{
kind = CompressionDictionary.Kind.valueOf(kindStr);
}
catch (IllegalArgumentException ex)
{
throw new IllegalStateException(kindStr + " compression dictionary is not created");
}
return kind.getDictionary(new DictId(kind, row.getLong("dict_id")), row.getByteArray("dict"));
}
The fact that we have successfully materialized `Kind` from `kind` string from `Row` pretty much guarantees that we can create a dictionary from it.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
Review Comment:
this happens to be extracted into a constant like
public static final String MBEAN_NAME = "org.apache.cassandra.hints:type=HintsService";
for other MBeans. I would also move that constant to MBean interface itself and just referenced here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410787864
##########
src/java/org/apache/cassandra/io/compress/ZstdDictionaryCompressor.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.compress;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import com.github.luben.zstd.Zstd;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.ZstdCompressionDictionary;
+import org.apache.cassandra.db.compression.CompressionDictionary.Kind;
+import org.apache.cassandra.utils.concurrent.Ref;
+
+import javax.annotation.Nullable;
+
+public class ZstdDictionaryCompressor extends ZstdCompressorBase implements ICompressor, IDictionaryCompressor<ZstdCompressionDictionary>
+{
+ private static final ConcurrentHashMap<Integer, ZstdDictionaryCompressor> instancesPerLevel = new ConcurrentHashMap<>();
+ private static final Cache<ZstdCompressionDictionary, ZstdDictionaryCompressor> instancePerDict =
+ Caffeine.newBuilder()
+ .maximumSize(DatabaseDescriptor.getCompressionDictionaryCacheSize())
+ .expireAfterAccess(Duration.ofSeconds(DatabaseDescriptor.getCompressionDictionaryCacheExpireSeconds()))
+ .removalListener((ZstdCompressionDictionary dictionary,
+ ZstdDictionaryCompressor compressor,
+ RemovalCause cause) -> {
+ // Release dictionary reference when compressor is evicted from cache
+ if (compressor != null && compressor.dictionaryRef != null)
+ {
+ compressor.dictionaryRef.release();
+ }
+ })
+ .build();
+
+ // dictioanry and its ref are null, when they are absent.
Review Comment:
typo in `dictioanry`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415191161
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
Review Comment:
changed to throw `UnsupportedOperationException` as suggested in another comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2427248918
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
Sounds good. I am open to remove the write-based sampling from the manual training tool invocation.
I would like to clarify on whether we want to run `flush` on the table regardless or only when there is no live sstable. I think the former is probably fine as the training tool is used very sparsely, possibly only once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2429967193
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
@yifan-c thanks, I had to overlook that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "rustyrazorblade (via GitHub)" <gi...@apache.org>.
rustyrazorblade commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2427157376
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
I think if starting training first triggers a flush, meaning we'll almost certainly have data on disk (unless there are zero writes ever, which we can't do much about), then removing training from the write path seems perfectly reasonable to me. I can't think of a reason to keep the write based sampling if we flush first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410981315
##########
src/java/org/apache/cassandra/tools/NodeProbe.java:
##########
@@ -2682,6 +2683,61 @@ public void setMixedMajorVersionRepairEnabled(boolean enabled)
{
autoRepairProxy.setMixedMajorVersionRepairEnabled(enabled);
}
+
+ /**
+ * Triggers compression dictionary training for the specified table.
+ *
+ * @param keyspace the keyspace name
+ * @param table the table name
+ * @param options options for the training process (currently unused, reserved for future extensions)
+ * @throws IOException if there's an error accessing the MBean
+ * @throws IllegalArgumentException if table doesn't support dictionary compression
+ */
+ public void trainCompressionDictionary(String keyspace, String table, Map<String, String> options) throws IOException
+ {
+ getDictionaryManagerProxy(keyspace, table).train(options);
+ }
+
+ /**
+ * Gets the compression dictionary training status for the specified table.
+ *
+ * @param keyspace the keyspace name
+ * @param table the table name
+ * @return the training status as string
+ * @throws IOException if there's an error accessing the MBean
+ */
+ public String getCompressionDictionaryTrainingStatus(String keyspace, String table) throws IOException
+ {
+ return getDictionaryManagerProxy(keyspace, table).getTrainingStatus();
+ }
+
+ /**
+ * Updates the sampling rate for compression dictionary training.
+ *
+ * @param keyspace the keyspace name
+ * @param table the table name
+ * @param samplingRate the new sampling rate (1 = sample every time, 2 = sample every 2nd time, etc.)
+ * @throws IOException if there's an error accessing the MBean
+ */
+ public void updateCompressionDictionaryTrainingSamplingRate(String keyspace, String table, int samplingRate) throws IOException
+ {
+ getDictionaryManagerProxy(keyspace, table).updateSamplingRate(samplingRate);
+ }
+
+ private CompressionDictionaryManagerMBean getDictionaryManagerProxy(String keyspace, String table) throws IOException
+ {
+ // Construct table-specific MBean name
+ String mbeanName = "org.apache.cassandra.db.compression:type=CompressionDictionaryManager,keyspace=" + keyspace + ",table=" + table;
Review Comment:
replace with constant once introduced
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415140355
##########
src/java/org/apache/cassandra/db/compaction/unified/ShardedMultiWriter.java:
##########
@@ -118,6 +118,7 @@ private SSTableWriter createWriter(Descriptor descriptor)
.setSerializationHeader(header)
.addDefaultComponents(indexGroups)
.setSecondaryIndexGroups(indexGroups)
+ .setCompressionDictionaryManager(cfs.compressionDictionaryManager())
Review Comment:
It is closed by CFS (the owner) when `invalidate` (e.g. drop table)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2417342419
##########
src/java/org/apache/cassandra/db/compression/ZstdDictionaryTrainer.java:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.luben.zstd.Zstd;
+import com.github.luben.zstd.ZstdDictTrainer;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.CompressionDictionary.DictId;
+import org.apache.cassandra.db.compression.CompressionDictionary.Kind;
+import org.apache.cassandra.io.compress.IDictionaryCompressor;
+import org.apache.cassandra.io.compress.ZstdDictionaryCompressor;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.utils.Clock;
+
+/**
+ * Zstd implementation of dictionary trainer with lifecycle management.
+ */
+public class ZstdDictionaryTrainer implements ICompressionDictionaryTrainer
+{
+ private static final Logger logger = LoggerFactory.getLogger(ZstdDictionaryTrainer.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final CompressionDictionaryTrainingConfig config;
+ private final AtomicLong totalSampleSize;
+ private final AtomicLong sampleCount;
+ private final int compressionLevel; // optimal if using the same level for training as when compressing.
+
+ // Sampling rate can be updated during training
+ private volatile int samplingRate;
+
+ // Minimum number of samples required by ZSTD library
+ private static final int MIN_SAMPLES_REQUIRED = 10;
+
+ private volatile Consumer<CompressionDictionary> dictionaryTrainedListener;
+ // TODO: manage the samples in this class for auto-train (follow-up). The ZstdDictTrainer cannot be re-used for multiple training runs.
+ private ZstdDictTrainer zstdTrainer;
+ private volatile boolean closed = false;
+ private volatile TrainingStatus currentTrainingStatus;
+
+ public ZstdDictionaryTrainer(String keyspaceName, String tableName,
+ CompressionDictionaryTrainingConfig config,
+ int compressionLevel)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.config = config;
+ this.totalSampleSize = new AtomicLong(0);
+ this.sampleCount = new AtomicLong(0);
+ this.compressionLevel = compressionLevel;
+ this.samplingRate = config.samplingRate;
+ this.currentTrainingStatus = TrainingStatus.NOT_STARTED;
+ }
+
+ @Override
+ public boolean shouldSample()
+ {
+ return zstdTrainer != null && ThreadLocalRandom.current().nextInt(samplingRate) == 0;
+ }
+
+ @Override
+ public void addSample(ByteBuffer sample)
+ {
+ if (closed || sample == null || !sample.hasRemaining() || zstdTrainer == null)
+ return;
+
+ byte[] sampleBytes = new byte[sample.remaining()];
+ sample.duplicate().get(sampleBytes);
+
+ if (zstdTrainer.addSample(sampleBytes))
+ {
+ // Update the totalSampleSize and sampleCount if the sample is added
+ totalSampleSize.addAndGet(sampleBytes.length);
+ sampleCount.incrementAndGet();
+ }
+ }
+
+ @Override
+ public CompressionDictionary trainDictionary(boolean force)
+ {
+ boolean isReady = isReady();
+ if (!force && !isReady)
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ throw new IllegalStateException("Trainer is not ready");
+ }
+
+ long currentSampleCount = sampleCount.get();
+ if (currentSampleCount < MIN_SAMPLES_REQUIRED) // minimum samples should be required even if force training
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ String errorMsg = String.format("Insufficient samples for training: %d (minimum required: %d)",
+ currentSampleCount, MIN_SAMPLES_REQUIRED);
+ throw new IllegalStateException(errorMsg);
Review Comment:
I would change the result of this whole method to "TraningResult" where status would be only one component of that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2413973074
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
Review Comment:
I would rename to dictId();
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415149748
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
+ if (kind == Kind.ZSTD)
+ {
+ dictionary = new ZstdCompressionDictionary(dictId, dict);
+ }
+
+ if (dictionary == null)
+ {
+ throw new IOException(kind + " compression dictionary is not created");
+ }
+
+ // update the dictionary manager if it exists
+ if (manager != null)
+ {
+ manager.add(dictionary);
+ }
+ return dictionary;
+ }
+
+ static CompressionDictionary createFromRow(UntypedResultSet.Row row)
+ {
+ String kindStr = row.getString("kind");
+ long id = row.getLong("dict_id");
+ byte[] dict = row.getByteArray("dict");
+ CompressionDictionary.DictId dictId = new CompressionDictionary.DictId(CompressionDictionary.Kind.valueOf(kindStr), id);
Review Comment:
Again, credit goes to Stefan for the refactor.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Handles compression dictionary events including training completion and cluster notifications.
+ * <p>
+ * This class handles:
+ * - Broadcasting dictionary updates to cluster nodes
+ * - Retrieving new dictionaries when notified by other nodes
+ * - Managing dictionary cache updates
+ */
+public class CompressionDictionaryEventHandler implements ICompressionDictionaryEventHandler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryEventHandler.class);
+
+ private final ColumnFamilyStore cfs;
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ public CompressionDictionaryEventHandler(ColumnFamilyStore cfs, ICompressionDictionaryCache cache)
+ {
+ this.cfs = cfs;
+ this.keyspaceName = cfs.keyspace.getName();
+ this.tableName = cfs.getTableName();
+ this.cache = cache;
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
Review Comment:
You have addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410757796
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
Review Comment:
are we really going to use camel case? not snake case? not sure what is better or what we use consistently. Also we might probably extract this string to a standalone constant in this class. Easy to mess it up by typing it constantly.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManagerMBean.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+public interface CompressionDictionaryManagerMBean
+{
+ /**
+ * Starts sampling and training for this table.
+ *
+ * @param options options for the training process (currently unused, reserved for future extensions)
+ * @throws IllegalArgumentException if table doesn't support dictionary compression
Review Comment:
Throw `UnsupportedOperationException` then?
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryCache.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import javax.annotation.Nullable;
+
+/**
+ * Interface for managing compression dictionary caching and current dictionary state.
+ * <p>
+ * Implementations handle:
+ * - Local caching of compression dictionaries with automatic cleanup
+ * - Managing the current active dictionary for write operations
+ * - Thread-safe access to cached dictionaries
+ */
+public interface ICompressionDictionaryCache extends AutoCloseable
+{
+ /**
+ * Gets the current active compression dictionary.
+ *
+ * @return the current compression dictionary, or null if no dictionary is available
+ */
+ @Nullable
+ CompressionDictionary getCurrent();
Review Comment:
It would be better style to work with `Optional<CompressionDictionary>` than to work with `Nullable` annotations. Feels just more Java idiomatic than this. Applicable for `get` method too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410526070
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
Review Comment:
The problem I see with the current logic is that we are too "aware" of ZSTD. If this is designed to be "future proof" when other dictionary compressors can be added, then the logic should not be "wired" around ZSTD there. It should not _matter_ what `Kind` we got. Why would it ... I just want to add a new Kind and overload its "getDictionary". I do not need to touch this code anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410701055
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
Review Comment:
I feel like this would be simplified if we did not put this logic around "isEnabled" here but we would pass isEnabled into `ICompressionDictionaryTrainer.create` and there, if it is `false`, we would return a dummy / no-op (singleton) `ICompressionDictionaryTrainer` doing nothing. Then it would not matter that we would call `trainer.start(false)` on a dummy trainer and similar.
`this::handleNewDictionary` can be also part of `create` method and set internally, that's purely optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2425436401
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
The typical reason for training failure (after about 10 minutes) is that no `flush` is invoked with the `traincompressiondictionary` command. Otherwise, within 10 minutes, there should normally be enough samples collected.
To clarify the current implementation: the termination condition for training is already defined as either reaching the maximum number of samples or the maximum sampling duration. The maximum sample size (`compression_dictionary_training_max_total_sample_size`) is configured in `cassandra.yaml`. The trainer starts the dictionary training once either of these thresholds is reached.
I agree that we should avoid shipping foot-guns. Based on the feedback, I see problems with the write-based sampling approach. It is too complex since it requires running another command (`flush`) to ensure success. Frequent flushes can also lead to undesirable side effects, such as producing many small SSTables.
On the other hand, SSTable-based sampling works better for manual training. It provides a more deterministic and self-contained process, and ensures the command terminates without having to worry about the sampling duration or manual intervention.
In contrast, write-based sampling seems more suitable for automatic (auto-training) scenarios.
Given these points, should we remove write-based sampling from the manual `traincompressiondictionary` command? What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#issuecomment-3470684212
Thanks all. The patch is merged!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c closed pull request #4399: CASSANDRA-17021: Support ZSTD dictionary compression
URL: https://github.com/apache/cassandra/pull/4399
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2421370386
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
The reason of using duration is to have a time bound for sampling. If using the max # of samples, there is a chance that the tool runs too long, even indefinitely, if no flush and no compaction occurs.
Having a time bound at least will terminate the tool process in a controlled manner. I acknowledge that having a failed training after 10 minutes is frustrating. It is up to operators to configure the tool properly; also the SSTable-based sampling that you suggested and I just added should solve the issue better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410967130
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryTrainer.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.io.compress.ICompressor;
+import org.apache.cassandra.io.compress.IDictionaryCompressor;
+import org.apache.cassandra.io.compress.ZstdDictionaryCompressor;
+import org.apache.cassandra.schema.CompressionParams;
+
+/**
+ * Interface for training compression dictionaries from sample data.
+ * <p>
+ * Implementations handle:
+ * - Sample collection and management
+ * - Dictionary training lifecycle
+ * - Asynchronous training execution
+ * - Training status tracking
+ */
+public interface ICompressionDictionaryTrainer extends AutoCloseable
+{
+ /**
+ * Starts the trainer for collecting samples.
+ *
+ * @param manualTraining true if this is manual training, false for automatic
+ * @return true if the trainer is started; otherwise false. The trainer is started
+ * in any of those conditions: 1. trainer closed; 2. not requested for
+ * either manual or auto training; 3. failed to start
+ */
+ boolean start(boolean manualTraining);
+
+ /**
+ * @return true if the trainer is ready to take a new sample; otherwise, false
+ */
+ boolean shouldSample();
+
+ /**
+ * Adds a sample to the training dataset.
+ *
+ * @param sample the sample data to add for training
+ */
+ void addSample(ByteBuffer sample);
+
+ /**
+ * Trains and produces a compression dictionary from collected samples synchronously.
+ *
+ * @param force force the dictionary training even if there are not enough samples;
+ * otherwise, dictionary training won't start if the trainer is not ready
+ * @return the trained compression dictionary
+ */
+ CompressionDictionary trainDictionary(boolean force);
+
+ /**
+ * Trains and produces a compression dictionary from collected samples asynchronously.
+ *
+ * @param force force the dictionary training even if there are not enough samples
+ * @return CompletableFuture that completes when training is done
+ */
+ default CompletableFuture<CompressionDictionary> trainDictionaryAsync(boolean force)
+ {
+ return CompletableFuture.supplyAsync(() -> trainDictionary(force), ScheduledExecutors.nonPeriodicTasks);
+ }
+
+ /**
+ * @return true if enough samples have been collected for training
+ */
+ boolean isReady();
+
+ /**
+ * Clears all collected samples and resets trainer state.
+ */
+ void reset();
+
+ /**
+ * @return the current training status
+ */
+ TrainingStatus getTrainingStatus();
+
+ /**
+ * @return the compression algorithm kind this trainer supports
+ */
+ CompressionDictionary.Kind kind();
+
+ /**
+ * Determines if this trainer is compatible with the given compression parameters.
+ * This method allows the trainer to decide whether it can continue operating
+ * with new compression parameters or if a new trainer instance is needed.
+ *
+ * @param newParams the new compression parameters to check compatibility against
+ * @return true if this trainer is compatible with the new parameters, false otherwise
+ */
+ boolean isCompatibleWith(CompressionParams newParams);
+
+ /**
+ * Sets the listener for dictionary training events.
+ *
+ * @param listener the listener to be notified when dictionaries are trained, null to remove listener
+ */
+ void setDictionaryTrainedListener(Consumer<CompressionDictionary> listener);
+
+ /**
+ * Updates the sampling rate for this trainer.
+ *
+ * @param newSamplingRate the new sampling rate. For exmaple, 1 = sample every time (100%),
+ * 2 = expect sample 1/2 of data (50%), n = expect sample 1/n of data
+ */
+ void updateSamplingRate(int newSamplingRate);
+
+ /**
+ * Factory method to create appropriate trainer based on compression parameters.
+ *
+ * @param keyspaceName the keyspace name for logging
+ * @param tableName the table name for logging
+ * @param params the compression parameters
+ * @param config the training configuration
+ * @return a dictionary trainer for the specified compression algorithm
+ * @throws IllegalArgumentException if no dictionary trainer is available for the compression algorithm
+ */
+ static ICompressionDictionaryTrainer create(String keyspaceName,
+ String tableName,
+ CompressionParams params,
+ CompressionDictionaryTrainingConfig config)
+ {
+ ICompressor compressor = params.getSstableCompressor();
+ if (!(compressor instanceof IDictionaryCompressor))
+ {
+ throw new IllegalArgumentException("Compressor does not support dictionary training: " + params.getSstableCompressor());
+ }
+
+ IDictionaryCompressor dictionaryCompressor = (IDictionaryCompressor) compressor;
+ if (dictionaryCompressor.acceptableDictionaryKind() == CompressionDictionary.Kind.ZSTD)
Review Comment:
we are zstd specific here, not sure that is really needed.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Handles compression dictionary events including training completion and cluster notifications.
+ * <p>
+ * This class handles:
+ * - Broadcasting dictionary updates to cluster nodes
+ * - Retrieving new dictionaries when notified by other nodes
+ * - Managing dictionary cache updates
+ */
+public class CompressionDictionaryEventHandler implements ICompressionDictionaryEventHandler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryEventHandler.class);
+
+ private final ColumnFamilyStore cfs;
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ public CompressionDictionaryEventHandler(ColumnFamilyStore cfs, ICompressionDictionaryCache cache)
+ {
+ this.cfs = cfs;
+ this.keyspaceName = cfs.keyspace.getName();
+ this.tableName = cfs.getTableName();
+ this.cache = cache;
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ logger.info("Notifying cluster about dictionary update for {}.{} with dictionaryId {}",
+ keyspaceName, tableName, dictionaryId);
+
+ CompressionDictionaryUpdateMessage message = new CompressionDictionaryUpdateMessage(cfs.metadata().id, dictionaryId);
+ Collection<InetAddressAndPort> allNodes = ClusterMetadata.current().directory.allJoinedEndpoints();
+ // Broadcast notification using the fire-and-forget fashion
+ for (InetAddressAndPort node : allNodes)
+ {
+ if (node.equals(FBUtilities.getBroadcastAddressAndPort())) // skip ourself
+ continue;
+ sendNotification(node, message);
+ }
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictId)
+ {
+ // Best effort to retrieve the dictionary; otherwise, the periodic task should retrieve the dictionary later
+ CompletableFuture.runAsync(() -> {
+ try
+ {
+ if (!cfs.metadata().params.compression.isDictionaryCompressionEnabled())
+ {
+ return;
+ }
+
+ CompressionDictionary dictionary = SystemDistributedKeyspace.retrieveCompressionDictionary(keyspaceName, tableName, dictId);
+ cache.setCurrentIfNewer(dictionary);
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to retrieve compression dictionary for {}.{}. dictionaryId={}",
+ keyspaceName, tableName, dictId, e);
+ }
+ }, ScheduledExecutors.nonPeriodicTasks);
+ }
+
+ // Best effort to notify the peer regarding the new dictionary being available to pull.
+ // If the request fails, each peer has periodic task scheduled to pull.
+ private void sendNotification(InetAddressAndPort target, CompressionDictionaryUpdateMessage message)
+ {
+ logger.debug("Sending dictionary update notification to {}", target);
Review Comment:
Could we be more verbose? Dictionary of what kind and id? Here `DictId` would come handy. Same for all other debug messages, like in the listener below.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryScheduler.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+
+/**
+ * Manages scheduled tasks for compression dictionary operations.
+ * <p>
+ * This class handles:
+ * - Periodic refresh of dictionaries from system tables
+ * - Manual training task scheduling and monitoring
+ * - Cleanup of scheduled tasks
+ */
+public class CompressionDictionaryScheduler implements ICompressionDictionaryScheduler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryScheduler.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ private volatile ScheduledFuture<?> scheduledRefreshTask;
+ private volatile ScheduledFuture<?> scheduledManualTrainingTask;
+ private volatile boolean isEnabled;
+
+ public CompressionDictionaryScheduler(String keyspaceName,
+ String tableName,
+ ICompressionDictionaryCache cache,
+ boolean isEnabled)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.cache = cache;
+ this.isEnabled = isEnabled;
+ }
+
+ /**
+ * Schedules the periodic dictionary refresh task if not already scheduled.
+ */
+ public void scheduleRefreshTask()
+ {
+ if (scheduledRefreshTask != null)
+ return;
+
+ this.scheduledRefreshTask = ScheduledExecutors.scheduledTasks.scheduleWithFixedDelay(
+ this::refreshDictionaryFromSystemTable,
+ DatabaseDescriptor.getCompressionDictionaryRefreshInitialDelaySeconds(),
+ DatabaseDescriptor.getCompressionDictionaryRefreshIntervalSeconds(),
+ TimeUnit.SECONDS
+ );
+ }
+
+ @Override
+ public void scheduleManualTraining(ManualTrainingOptions options, ICompressionDictionaryTrainer trainer)
+ {
+ if (scheduledManualTrainingTask != null)
+ {
+ throw new IllegalStateException("Training already in progress for table " + keyspaceName + '.' + tableName);
+ }
+
+ int maxSamplingDurationSeconds = options.getMaxSamplingDurationSeconds();
+
+ logger.info("Starting manual dictionary training for {}.{} with max sampling duration: {} seconds",
+ keyspaceName, tableName, maxSamplingDurationSeconds);
+
+ long deadlineMillis = System.currentTimeMillis() + TimeUnit.SECONDS.toMillis(maxSamplingDurationSeconds);
+
+ ManualTrainingTask task = new ManualTrainingTask(deadlineMillis, trainer);
+
+ // Check every second whether it gets enough samples and completes training
+ scheduledManualTrainingTask = ScheduledExecutors.scheduledTasks
+ .scheduleWithFixedDelay(task, 1, 1, TimeUnit.SECONDS);
+ }
+
+ @Override
+ public void cancelManualTraining()
+ {
+ ScheduledFuture<?> future = scheduledManualTrainingTask;
+ if (future != null)
+ {
+ future.cancel(false);
+ }
+ scheduledManualTrainingTask = null;
+ }
+
+ /**
+ * Sets the enabled state of the scheduler. When disabled, refresh tasks will not execute.
+ *
+ * @param enabled whether the scheduler should be enabled
+ */
+ @Override
+ public void setEnabled(boolean enabled)
+ {
+ this.isEnabled = enabled;
+ }
+
+ /**
+ * Refreshes dictionary from system table and updates the cache.
+ * This method is called periodically by the scheduled refresh task.
+ */
+ private void refreshDictionaryFromSystemTable()
+ {
+ try
+ {
+ if (!isEnabled)
+ {
+ return;
+ }
+
+ CompressionDictionary dictionary = SystemDistributedKeyspace.retrieveLatestCompressionDictionary(keyspaceName, tableName);
+ cache.setCurrentIfNewer(dictionary);
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to refresh compression dictionary for {}.{}",
+ keyspaceName, tableName, e);
+ }
+ }
+
+ @Override
+ public void close()
+ {
+ if (scheduledRefreshTask != null)
+ {
+ scheduledRefreshTask.cancel(false);
+ scheduledRefreshTask = null;
+ }
+
+ if (scheduledManualTrainingTask != null)
+ {
+ scheduledManualTrainingTask.cancel(false);
+ scheduledManualTrainingTask = null;
+ }
+ }
+
+ private class ManualTrainingTask implements Runnable
+ {
+ private final long deadlineMillis;
+ private final ICompressionDictionaryTrainer trainer;
+ private boolean isTraining = false;
+
+ private ManualTrainingTask(long deadlineMillis, ICompressionDictionaryTrainer trainer)
+ {
+ this.deadlineMillis = deadlineMillis;
+ this.trainer = trainer;
+ }
+
+ @Override
+ public void run()
+ {
+ if (trainer.getTrainingStatus() == TrainingStatus.NOT_STARTED)
+ {
+ logger.warn("Trainer is not started. Stop training dictionary for table {}.{}", keyspaceName, tableName);
+ cancelManualTraining();
+ return;
+ }
+
+ long now = System.currentTimeMillis();
Review Comment:
same as above
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.time.Duration;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import org.apache.cassandra.config.DatabaseDescriptor;
+
+/**
+ * Manages caching and current dictionary state for compression dictionaries.
+ * <p>
+ * This class handles:
+ * - Local caching of compression dictionaries with automatic cleanup
+ * - Managing the current active dictionary for write operations
+ * - Thread-safe access to cached dictionaries
+ */
+public class CompressionDictionaryCache implements ICompressionDictionaryCache
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryCache.class);
+
+ private final Cache<CompressionDictionary.DictId, CompressionDictionary> cache;
+ private final AtomicReference<CompressionDictionary> currentDictionary = new AtomicReference<>();
+
+ public CompressionDictionaryCache()
+ {
+ Duration expiryTime = Duration.ofSeconds(DatabaseDescriptor.getCompressionDictionaryCacheExpireSeconds());
Review Comment:
can be just inlined into `expireAfterAccess`?
.expireAfterAccess(Duration.ofSeconds(DatabaseDescriptor.getCompressionDictionaryCacheExpireSeconds()))
We are not making special variable for `DatabaseDescriptor.getCompressionDictionaryCacheSize()` for `maximumSize` method.
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryTrainer.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.CompletableFuture;
Review Comment:
illegal import
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410329650
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
+
+# Dictionary training configuration (advanced settings)
+# These settings control how compression dictionaries are trained from sample data.
+
+# Maximum size of a trained compression dictionary in bytes.
+# Larger dictionaries may provide better compression but use more memory.
+# Min unit: B
+compression_dictionary_training_max_dictionary_size: 65536
Review Comment:
could this be converted to `64KiB`? I think that is more expressive then just a number. That would lead to `DataStorageSpec.IntKibibytesBound` type of compression_dictionary_training_max_dictionary_size. By doing it like this, people are really forced to put there just integers and nothing else. This is in general not a style which is used through this yaml where data / capacity related configuration parameters are descendants of `DataStorageSpec`.
Same way you used `DurationSpec.IntSecondsBound` for properties above, we should use `DataStorageSpec` for these.
##########
src/java/org/apache/cassandra/db/compression/ZstdDictionaryTrainer.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.luben.zstd.Zstd;
+import com.github.luben.zstd.ZstdDictTrainer;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.CompressionDictionary.DictId;
+import org.apache.cassandra.db.compression.CompressionDictionary.Kind;
+import org.apache.cassandra.io.compress.IDictionaryCompressor;
+import org.apache.cassandra.io.compress.ZstdDictionaryCompressor;
+import org.apache.cassandra.schema.CompressionParams;
+
+/**
+ * Zstd implementation of dictionary trainer with lifecycle management.
+ */
+public class ZstdDictionaryTrainer implements ICompressionDictionaryTrainer
+{
+ private static final Logger logger = LoggerFactory.getLogger(ZstdDictionaryTrainer.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final CompressionDictionaryTrainingConfig config;
+ private final AtomicLong totalSampleSize;
+ private final AtomicLong sampleCount;
+ private final int compressionLevel; // optimal if using the same level for training as when compressing.
+
+ // Sampling rate can be updated during training
+ private volatile int samplingRate;
+
+ // Minimum number of samples required by ZSTD library
+ private static final int MIN_SAMPLES_REQUIRED = 10;
+
+ private volatile Consumer<CompressionDictionary> dictionaryTrainedListener;
+ // TODO: manage the samples in this class for auto-train (follow-up). The ZstdDictTrainer cannot be re-used for multiple training runs.
+ private ZstdDictTrainer zstdTrainer;
+ private volatile boolean closed = false;
+ private volatile TrainingStatus currentTrainingStatus;
+
+ public ZstdDictionaryTrainer(String keyspaceName, String tableName,
+ CompressionDictionaryTrainingConfig config,
+ int compressionLevel)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.config = config;
+ this.totalSampleSize = new AtomicLong(0);
+ this.sampleCount = new AtomicLong(0);
+ this.compressionLevel = compressionLevel;
+ this.samplingRate = config.samplingRate;
+ this.currentTrainingStatus = TrainingStatus.NOT_STARTED;
+ }
+
+ @Override
+ public boolean shouldSample()
+ {
+ return zstdTrainer != null && ThreadLocalRandom.current().nextInt(samplingRate) == 0;
+ }
+
+ @Override
+ public void addSample(ByteBuffer sample)
+ {
+ if (closed || sample == null || !sample.hasRemaining() || zstdTrainer == null)
+ return;
+
+ byte[] sampleBytes = new byte[sample.remaining()];
+ sample.duplicate().get(sampleBytes);
+
+ if (zstdTrainer.addSample(sampleBytes))
+ {
+ // Update the totalSampleSize and sampleCount if the sample is added
+ totalSampleSize.addAndGet(sampleBytes.length);
+ sampleCount.incrementAndGet();
+ }
+ }
+
+ @Override
+ public CompressionDictionary trainDictionary(boolean force)
+ {
+ boolean isReady = isReady();
+ if (!force && !isReady)
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ throw new IllegalStateException("Trainer is not ready");
+ }
+
+ long currentSampleCount = sampleCount.get();
+ if (currentSampleCount < MIN_SAMPLES_REQUIRED) // minimum samples should be required even if force training
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ String errorMsg = String.format("Insufficient samples for training: %d (minimum required: %d)",
+ currentSampleCount, MIN_SAMPLES_REQUIRED);
+ throw new IllegalStateException(errorMsg);
+ }
+
+ currentTrainingStatus = TrainingStatus.TRAINING;
+ try
+ {
+ logger.debug("Training with sample count: {}, sample size: {}, isReady: {}",
+ currentSampleCount, totalSampleSize.get(), isReady);
+ byte[] dictBytes = zstdTrainer.trainSamples();
+ long zstdDictId = Zstd.getDictIdFromDict(dictBytes);
+ DictId dictId = new DictId(Kind.ZSTD, makeDictionaryId(System.currentTimeMillis(), zstdDictId));
+ currentTrainingStatus = TrainingStatus.COMPLETED;
+ logger.debug("New dictionary is trained with {}", dictId);
+ CompressionDictionary dictionary = new ZstdCompressionDictionary(dictId, dictBytes);
+ notifyDictionaryTrainedListener(dictionary);
+ return dictionary;
+ }
+ catch (Exception e)
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ throw new RuntimeException("Failed to train Zstd dictionary", e);
+ }
+ }
+
+ @Override
+ public boolean isReady()
+ {
+ return currentTrainingStatus != TrainingStatus.TRAINING
+ && !closed
+ && zstdTrainer != null
+ && totalSampleSize.get() >= config.acceptableTotalSampleSize
+ && sampleCount.get() > MIN_SAMPLES_REQUIRED;
+ }
+
+ @Override
+ public TrainingStatus getTrainingStatus()
+ {
+ return currentTrainingStatus;
+ }
+
+ @Override
+ public boolean start(boolean manualTraining)
+ {
+ if (closed || !(manualTraining || shouldAutoStartTraining()))
+ return false;
+
+ try
+ {
+ // reset on starting; a new zstdTrainer instance is created during reset
+ reset();
+ logger.info("Started dictionary training for {}.{}", keyspaceName, tableName);
+ currentTrainingStatus = TrainingStatus.SAMPLING;
+ return true;
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to create ZstdDictTrainer for {}.{}", keyspaceName, tableName, e);
+ currentTrainingStatus = TrainingStatus.FAILED;
+ }
+ return false;
+ }
+
+ /**
+ * Determines if training should auto-start based on configuration.
+ */
+ private boolean shouldAutoStartTraining()
+ {
+ return DatabaseDescriptor.getCompressionDictionaryTrainingAutoTrainEnabled();
+ }
+
+ @Override
+ public void reset()
+ {
+ if (closed)
+ {
+ return;
+ }
+
+ currentTrainingStatus = TrainingStatus.NOT_STARTED;
+ synchronized (this)
+ {
+ totalSampleSize.set(0);
+ sampleCount.set(0);
+ zstdTrainer = new ZstdDictTrainer(config.maxTotalSampleSize, config.maxDictionarySize, compressionLevel);
+ }
+ }
+
+ @Override
+ public Kind kind()
+ {
+ return Kind.ZSTD;
+ }
+
+ @Override
+ public void setDictionaryTrainedListener(Consumer<CompressionDictionary> listener)
+ {
+ this.dictionaryTrainedListener = listener;
+ }
+
+ @Override
+ public void updateSamplingRate(int newSamplingRate)
+ {
+ if (newSamplingRate <= 0)
+ {
+ throw new IllegalArgumentException("Sampling rate must be positive, got: " + newSamplingRate);
+ }
+ this.samplingRate = newSamplingRate;
+ logger.debug("Updated sampling rate to {} for {}.{}", newSamplingRate, keyspaceName, tableName);
+ }
+
+ /**
+ * Notifies the registered listener that a dictionary has been trained.
+ *
+ * @param dictionary the newly trained dictionary
+ */
+ private void notifyDictionaryTrainedListener(CompressionDictionary dictionary)
+ {
+ Consumer<CompressionDictionary> listener = this.dictionaryTrainedListener;
+ if (listener != null)
+ {
+ try
+ {
+ listener.accept(dictionary);
+ }
+ catch (Exception e)
+ {
+ logger.warn("Error notifying dictionary trained listener for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+ }
+
+ @Override
+ public boolean isCompatibleWith(CompressionParams newParams)
+ {
+ if (!newParams.isDictionaryCompressionEnabled())
+ {
+ return false;
+ }
+
+ IDictionaryCompressor newCompressor = (IDictionaryCompressor) newParams.getSstableCompressor();
+
+ // Check if the compressor type is compatible with this trainer
+ if (newCompressor.acceptableDictionaryKind() != Kind.ZSTD)
+ {
+ return false;
+ }
+
+ ZstdDictionaryCompressor zstdDictionaryCompressor = (ZstdDictionaryCompressor) newCompressor;
+ // For Zstd compressors, check if compression level matches
+ return this.compressionLevel == zstdDictionaryCompressor.compressionLevel();
+ }
+
+ @Override
+ public void close()
+ {
+ if (closed)
+ return;
+
+ closed = true;
+ currentTrainingStatus = TrainingStatus.NOT_STARTED;
+
+ synchronized (this)
+ {
+ // Permanent shutdown: clear all state and prevent restart
+ totalSampleSize.set(0);
+ sampleCount.set(0);
+ zstdTrainer = null;
+ }
+
+ logger.info("Permanently closed dictionary trainer for {}.{}", keyspaceName, tableName);
+ }
+
+ /**
+ * Creates a monotonically increasing dictionary ID by combining timestamp and Zstd dictionary ID.
+ *
+ * The resulting dictionary ID has the following structure:
+ * - Upper 32 bits: timestamp in minutes (signed int)
+ * - Lower 32 bits: Zstd dictionary ID (unsigned int, passed as long due to Java limitations)
+ *
+ * This ensures dictionary IDs are monotonically increasing over time, which helps to identify
+ * the latest dictionary.
+ *
+ * The implementation assumes that dictionary training frequency is significantly larger than
+ * every minute, which a healthy system should do. In the scenario when multiple dictionaries
+ * are trained in the same minute (only possible using manual training), there should not be
+ * correctness concerns since the dictionary is attached to the SSTables, but leads to performance
+ * hit from having too many dictionary. Therefore, such scenario should be avoided at the best.
+ *
+ * @param currentTimeMillis the current time in milliseconds
+ * @param zstdDictId Zstd dictionary ID (unsigned 32-bit value represented as long)
+ * @return combined dictionary ID that is monotonically increasing over time
+ */
+ static long makeDictionaryId(long currentTimeMillis, long zstdDictId)
Review Comment:
is this really ZSTD specific? seems like moving this to DictId or similar would be better so it can be used for other compressors too. I just dont see why is this ZSTD specific only.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Handles compression dictionary events including training completion and cluster notifications.
+ * <p>
+ * This class handles:
+ * - Broadcasting dictionary updates to cluster nodes
+ * - Retrieving new dictionaries when notified by other nodes
+ * - Managing dictionary cache updates
+ */
+public class CompressionDictionaryEventHandler implements ICompressionDictionaryEventHandler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryEventHandler.class);
+
+ private final ColumnFamilyStore cfs;
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ public CompressionDictionaryEventHandler(ColumnFamilyStore cfs, ICompressionDictionaryCache cache)
+ {
+ this.cfs = cfs;
+ this.keyspaceName = cfs.keyspace.getName();
+ this.tableName = cfs.getTableName();
+ this.cache = cache;
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
Review Comment:
dont we need to include `DictId`, actually? That also happens to contain `Kind`. I think that is way more descriptive and robust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410782291
##########
src/java/org/apache/cassandra/db/compression/ZstdCompressionDictionary.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.luben.zstd.ZstdDictCompress;
+import com.github.luben.zstd.ZstdDictDecompress;
+import org.apache.cassandra.io.compress.ZstdCompressorBase;
+import org.apache.cassandra.utils.concurrent.Ref;
+import org.apache.cassandra.utils.concurrent.RefCounted;
+import org.apache.cassandra.utils.concurrent.SelfRefCounted;
+
+public class ZstdCompressionDictionary implements CompressionDictionary, SelfRefCounted<ZstdCompressionDictionary>
+{
+ private static final Logger logger = LoggerFactory.getLogger(ZstdCompressionDictionary.class);
+
+ private final DictId dictId;
+ private final byte[] rawDictionary;
+ // One ZstdDictDecompress and multiple ZstdDictCompress (per level) can be derived from the same raw dictionary content
+ private final ConcurrentHashMap<Integer, ZstdDictCompress> zstdDictCompressPerLevel = new ConcurrentHashMap<>();
+ private volatile ZstdDictDecompress dictDecompress;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Ref<ZstdCompressionDictionary> selfRef;
+
+ public ZstdCompressionDictionary(DictId dictId, byte[] rawDictionary)
+ {
+ this.dictId = dictId;
+ this.rawDictionary = rawDictionary;
+ this.selfRef = new Ref<>(this, new Tidy(zstdDictCompressPerLevel, dictDecompress));
+ }
+
+ @Override
+ public DictId identifier()
+ {
+ return dictId;
+ }
+
+ @Override
+ public Kind kind()
+ {
+ return Kind.ZSTD;
+ }
+
+ @Override
+ public byte[] rawDictionary()
+ {
+ return rawDictionary;
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (!(o instanceof ZstdCompressionDictionary)) return false;
+ ZstdCompressionDictionary that = (ZstdCompressionDictionary) o;
+ return Objects.equals(dictId, that.dictId);
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return dictId.hashCode();
+ }
+
+ /**
+ * Get a pre-processed compression tables that is optimized for compression.
+ * It is derived/computed from dictionary bytes.
+ * The internal data structure is different from the tables for decompression.
+ *
+ * @param compressionLevel compression level to create the compression table
+ * @return ZstdDictCompress
+ */
+ public ZstdDictCompress dictionaryForCompression(int compressionLevel)
+ {
+ if (closed.get())
+ throw new IllegalStateException("Dictionary has been closed");
+
+ ZstdCompressorBase.validateCompressionLevel(compressionLevel);
+
+ return zstdDictCompressPerLevel.computeIfAbsent(compressionLevel, level -> {
+ if (closed.get())
+ throw new IllegalStateException("Dictionary has been closed");
+ return new ZstdDictCompress(rawDictionary, level);
+ });
+ }
+
+ /**
+ * Get a pre-processed decompression tables that is optimized for decompression.
+ * It is derived/computed from dictionary bytes.
+ * The internal data structure is different from the tables for compression.
+ *
+ * @return ZstdDictDecompress
+ */
+ public ZstdDictDecompress dictionaryForDecompression()
+ {
+ if (closed.get())
+ throw new IllegalStateException("Dictionary has been closed");
+
+ ZstdDictDecompress result = dictDecompress;
+ if (result != null)
+ return result;
+
+ synchronized (this)
+ {
+ if (closed.get())
+ throw new IllegalStateException("Dictionary has been closed");
+
+ result = dictDecompress;
+ if (result == null)
+ {
+ result = new ZstdDictDecompress(rawDictionary);
+ dictDecompress = result;
+ }
+ return result;
+ }
+ }
+
+ @Override
+ public Ref<ZstdCompressionDictionary> tryRef()
+ {
+ return selfRef.tryRef();
+ }
+
+ @Override
+ public Ref<ZstdCompressionDictionary> selfRef()
+ {
+ return selfRef;
+ }
+
+ @Override
+ public Ref<ZstdCompressionDictionary> ref()
+ {
+ return selfRef.ref();
+ }
+
+ @Override
+ public void close()
+ {
+ if (closed.compareAndSet(false, true))
+ {
+ selfRef.release();
+ }
+ }
+
+ private static class Tidy implements RefCounted.Tidy
+ {
+ private final ConcurrentHashMap<Integer, ZstdDictCompress> zstdDictCompressPerLevel;
+ private volatile ZstdDictDecompress dictDecompress;
+
+ Tidy(ConcurrentHashMap<Integer, ZstdDictCompress> zstdDictCompressPerLevel, ZstdDictDecompress dictDecompress)
+ {
+ this.zstdDictCompressPerLevel = zstdDictCompressPerLevel;
+ this.dictDecompress = dictDecompress;
+ }
+
+ @Override
+ public void tidy()
+ {
+ // Close all compression dictionaries
+ for (ZstdDictCompress compressDict : zstdDictCompressPerLevel.values())
+ {
+ try
+ {
+ compressDict.close();
+ }
+ catch (Exception e)
+ {
+ // Log but don't fail - continue closing other resources
+ logger.warn("Failed to close ZstdDictCompress", e);
+ }
+ }
+ zstdDictCompressPerLevel.clear();
+
+ // Close decompression dictionary
+ ZstdDictDecompress decompressDict = dictDecompress;
+ if (decompressDict != null)
+ {
+ try
+ {
+ decompressDict.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close ZstdDictDecompress", e);
+ }
+ dictDecompress = null;
+ }
+ }
+
+ @Override
+ public String name()
+ {
+ return "ZstdCompressionDictionary";
Review Comment:
`ZstdCompressionDictionary.class.getSimpleName()`.
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
Review Comment:
consider `Preconditions.checkArgument` here
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
+ if (kind == Kind.ZSTD)
+ {
+ dictionary = new ZstdCompressionDictionary(dictId, dict);
+ }
+
+ if (dictionary == null)
+ {
+ throw new IOException(kind + " compression dictionary is not created");
+ }
+
+ // update the dictionary manager if it exists
+ if (manager != null)
+ {
+ manager.add(dictionary);
+ }
+ return dictionary;
+ }
+
+ static CompressionDictionary createFromRow(UntypedResultSet.Row row)
+ {
+ String kindStr = row.getString("kind");
+ long id = row.getLong("dict_id");
+ byte[] dict = row.getByteArray("dict");
Review Comment:
We don't need this until `dictId.kind` is `ZSTD` so if it is not we at least do not materialize whole dictionary via means of `getByteArray` which creates `new byte[buf.remaining()]` and so on just to throw it away when we throw afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414917768
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryScheduler.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+
+/**
+ * Manages scheduled tasks for compression dictionary operations.
+ * <p>
+ * This class handles:
+ * - Periodic refresh of dictionaries from system tables
+ * - Manual training task scheduling and monitoring
+ * - Cleanup of scheduled tasks
+ */
+public class CompressionDictionaryScheduler implements ICompressionDictionaryScheduler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryScheduler.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ private volatile ScheduledFuture<?> scheduledRefreshTask;
+ private volatile ScheduledFuture<?> scheduledManualTrainingTask;
+ private volatile boolean isEnabled;
+
+ public CompressionDictionaryScheduler(String keyspaceName,
+ String tableName,
+ ICompressionDictionaryCache cache,
+ boolean isEnabled)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.cache = cache;
+ this.isEnabled = isEnabled;
+ }
+
+ /**
+ * Schedules the periodic dictionary refresh task if not already scheduled.
+ */
+ public void scheduleRefreshTask()
+ {
+ if (scheduledRefreshTask != null)
+ return;
+
+ this.scheduledRefreshTask = ScheduledExecutors.scheduledTasks.scheduleWithFixedDelay(
+ this::refreshDictionaryFromSystemTable,
+ DatabaseDescriptor.getCompressionDictionaryRefreshInitialDelaySeconds(),
+ DatabaseDescriptor.getCompressionDictionaryRefreshIntervalSeconds(),
+ TimeUnit.SECONDS
+ );
+ }
+
+ @Override
+ public void scheduleManualTraining(ManualTrainingOptions options, ICompressionDictionaryTrainer trainer)
+ {
+ if (scheduledManualTrainingTask != null)
+ {
+ throw new IllegalStateException("Training already in progress for table " + keyspaceName + '.' + tableName);
+ }
+
+ int maxSamplingDurationSeconds = options.getMaxSamplingDurationSeconds();
+
+ logger.info("Starting manual dictionary training for {}.{} with max sampling duration: {} seconds",
+ keyspaceName, tableName, maxSamplingDurationSeconds);
+
+ long deadlineMillis = System.currentTimeMillis() + TimeUnit.SECONDS.toMillis(maxSamplingDurationSeconds);
Review Comment:
Addressed
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
+ if (kind == Kind.ZSTD)
+ {
+ dictionary = new ZstdCompressionDictionary(dictId, dict);
+ }
+
+ if (dictionary == null)
+ {
+ throw new IOException(kind + " compression dictionary is not created");
+ }
+
+ // update the dictionary manager if it exists
+ if (manager != null)
+ {
+ manager.add(dictionary);
+ }
+ return dictionary;
+ }
+
+ static CompressionDictionary createFromRow(UntypedResultSet.Row row)
+ {
+ String kindStr = row.getString("kind");
+ long id = row.getLong("dict_id");
+ byte[] dict = row.getByteArray("dict");
Review Comment:
Good point. I saw that you have added the change. Thanks!
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415164121
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
Review Comment:
ok.
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
+
+# Dictionary training configuration (advanced settings)
+# These settings control how compression dictionaries are trained from sample data.
+
+# Maximum size of a trained compression dictionary in bytes.
+# Larger dictionaries may provide better compression but use more memory.
+# Min unit: B
+compression_dictionary_training_max_dictionary_size: 65536
Review Comment:
Agreed.
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
Review Comment:
Make sense. Moved the section to the bottom and replicated to the `cassandra_latest.yaml` too (at the end).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414946121
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
Review Comment:
I do not think starting all nodes at the same moment is a requirement. The refresh interval just defines how frequent each node should check the system table. When there is a new dictionary trained, the node will broadcast the message to all live nodes to notify the dictionary is available. This happens outside of the periodic polling.
Combining both, you can consider the interval as the upper bound of how dictionary is refreshed (when broadcasting fails at some nodes).
When some nodes are running with the prior dictionary or no dictionary, it is still not a huge issue. Eventually, all nodes will pick up the latest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410950069
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
Review Comment:
Preconditions.checkArgument ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415145181
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410959195
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
+ {
+ throw new IllegalArgumentException("Table " + keyspaceName + '.' + tableName + " does not support dictionary compression");
+ }
+
+ if (trainer == null)
+ {
+ throw new IllegalStateException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+
+ // Parse and validate training options
+ ManualTrainingOptions trainingOptions = ManualTrainingOptions.fromStringMap(options);
+
+ trainer.start(true);
+ scheduler.scheduleManualTraining(trainingOptions, trainer);
+ }
+
+ @Override
+ public String getTrainingStatus()
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer == null)
+ {
+ return TrainingStatus.NOT_STARTED.toString();
+ }
+ return dictionaryTrainer.getTrainingStatus().toString();
+ }
+
+ @Override
+ public void updateSamplingRate(int samplingRate)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer == null)
+ {
+ throw new IllegalArgumentException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+ dictionaryTrainer.updateSamplingRate(samplingRate);
+ }
+
+ /**
+ * Close all the resources. The method can be called multiple times.
+ */
+ @Override
+ public synchronized void close()
+ {
+ unregisterMbean();
+ if (trainer != null)
Review Comment:
closing no-op trainer would do nothing and when using concrete trainer we would just do `trainer = NO_OP` instead of `trainer = null`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415145181
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
Review Comment:
Addressed
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryCache.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import javax.annotation.Nullable;
+
+/**
+ * Interface for managing compression dictionary caching and current dictionary state.
+ * <p>
+ * Implementations handle:
+ * - Local caching of compression dictionaries with automatic cleanup
+ * - Managing the current active dictionary for write operations
+ * - Thread-safe access to cached dictionaries
+ */
+public interface ICompressionDictionaryCache extends AutoCloseable
+{
+ /**
+ * Gets the current active compression dictionary.
+ *
+ * @return the current compression dictionary, or null if no dictionary is available
+ */
+ @Nullable
+ CompressionDictionary getCurrent();
Review Comment:
Not a huge fan of `Optional`. Null exists anyway, and the callsite would see null in certain cases. If the language can fully eliminate null, then `Optional` makes sense.
```
CompressionDictionary compressionDictionary = compressionDictionaryManager == null ? null : compressionDictionaryManager.getCurrent();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410750400
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
Review Comment:
nit: braces not necessary at some places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410955678
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
+ {
+ throw new IllegalArgumentException("Table " + keyspaceName + '.' + tableName + " does not support dictionary compression");
+ }
+
+ if (trainer == null)
+ {
+ throw new IllegalStateException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+
+ // Parse and validate training options
+ ManualTrainingOptions trainingOptions = ManualTrainingOptions.fromStringMap(options);
+
+ trainer.start(true);
+ scheduler.scheduleManualTraining(trainingOptions, trainer);
+ }
+
+ @Override
+ public String getTrainingStatus()
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer == null)
+ {
+ return TrainingStatus.NOT_STARTED.toString();
+ }
+ return dictionaryTrainer.getTrainingStatus().toString();
Review Comment:
`getTrainingStatus` on no-op trainer would always returned `NOT_STARTED`.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
+ {
+ throw new IllegalArgumentException("Table " + keyspaceName + '.' + tableName + " does not support dictionary compression");
+ }
+
+ if (trainer == null)
+ {
+ throw new IllegalStateException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+
+ // Parse and validate training options
+ ManualTrainingOptions trainingOptions = ManualTrainingOptions.fromStringMap(options);
+
+ trainer.start(true);
+ scheduler.scheduleManualTraining(trainingOptions, trainer);
+ }
+
+ @Override
+ public String getTrainingStatus()
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer == null)
+ {
+ return TrainingStatus.NOT_STARTED.toString();
+ }
+ return dictionaryTrainer.getTrainingStatus().toString();
Review Comment:
`getTrainingStatus` on no-op trainer would always return `NOT_STARTED`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2411740844
##########
src/java/org/apache/cassandra/io/compress/CompressionMetadata.java:
##########
@@ -139,11 +165,54 @@ private CompressionMetadata(CompressionMetadata copy)
this.compressedFileLength = copy.compressedFileLength;
this.chunkOffsets = copy.chunkOffsets;
this.chunkOffsetsSize = copy.chunkOffsetsSize;
+ this.compressionDictionary = copy.compressionDictionary;
}
public ICompressor compressor()
{
- return parameters.getSstableCompressor();
+ // classic double-checked locking to call resolveCompressor method just once per CompressionMetadata object
+ ICompressor result = resolvedCompressor;
+ if (result != null)
+ return result;
+
+ synchronized (this)
+ {
+ result = resolvedCompressor;
+ if (result == null)
+ {
+ result = resolveCompressor(parameters.getSstableCompressor(), compressionDictionary);
+ resolvedCompressor = result;
+ }
+ return result;
+ }
+ }
+
+ static ICompressor resolveCompressor(ICompressor compressor, CompressionDictionary dictionary)
+ {
+ if (dictionary == null)
+ return compressor;
+
+ // When the attached dictionary can be consumed by the current dictionary compressor
+ if (compressor instanceof IDictionaryCompressor)
+ {
+ IDictionaryCompressor dictionaryCompressor = (IDictionaryCompressor) compressor;
+ if (dictionaryCompressor.canConsumeDictionary(dictionary))
+ return dictionaryCompressor.getOrCopyWithDictionary(dictionary);
+ }
+
+ // When the current compressor is not compatible with the dictionary. It could happen in the read path when:
+ // 1. The current compressor is not a dictionary compressor, but there is dictionary attached
+ // 2. The current dictionary compressor is a different type, e.g. table schema is changed
+ // In those cases, we should get the compatible dictionary compressor based on the dictionary
+ if (dictionary.kind() == ZSTD)
Review Comment:
or even better
return dictionary.getCompressor();
which would be like
default IDictionaryCompressor getCompressor()
{
return kind().getCompressor(this);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2411740844
##########
src/java/org/apache/cassandra/io/compress/CompressionMetadata.java:
##########
@@ -139,11 +165,54 @@ private CompressionMetadata(CompressionMetadata copy)
this.compressedFileLength = copy.compressedFileLength;
this.chunkOffsets = copy.chunkOffsets;
this.chunkOffsetsSize = copy.chunkOffsetsSize;
+ this.compressionDictionary = copy.compressionDictionary;
}
public ICompressor compressor()
{
- return parameters.getSstableCompressor();
+ // classic double-checked locking to call resolveCompressor method just once per CompressionMetadata object
+ ICompressor result = resolvedCompressor;
+ if (result != null)
+ return result;
+
+ synchronized (this)
+ {
+ result = resolvedCompressor;
+ if (result == null)
+ {
+ result = resolveCompressor(parameters.getSstableCompressor(), compressionDictionary);
+ resolvedCompressor = result;
+ }
+ return result;
+ }
+ }
+
+ static ICompressor resolveCompressor(ICompressor compressor, CompressionDictionary dictionary)
+ {
+ if (dictionary == null)
+ return compressor;
+
+ // When the attached dictionary can be consumed by the current dictionary compressor
+ if (compressor instanceof IDictionaryCompressor)
+ {
+ IDictionaryCompressor dictionaryCompressor = (IDictionaryCompressor) compressor;
+ if (dictionaryCompressor.canConsumeDictionary(dictionary))
+ return dictionaryCompressor.getOrCopyWithDictionary(dictionary);
+ }
+
+ // When the current compressor is not compatible with the dictionary. It could happen in the read path when:
+ // 1. The current compressor is not a dictionary compressor, but there is dictionary attached
+ // 2. The current dictionary compressor is a different type, e.g. table schema is changed
+ // In those cases, we should get the compatible dictionary compressor based on the dictionary
+ if (dictionary.kind() == ZSTD)
Review Comment:
or even better
return dictionary.getCompressor();
which would be like
public IDictionaryCompressor getCompressor()
{
return kind().getCompressor(this);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2413973074
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
Review Comment:
I would rename to dictId();
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415153854
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryCache.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.time.Duration;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import org.apache.cassandra.config.DatabaseDescriptor;
+
+/**
+ * Manages caching and current dictionary state for compression dictionaries.
+ * <p>
+ * This class handles:
+ * - Local caching of compression dictionaries with automatic cleanup
+ * - Managing the current active dictionary for write operations
+ * - Thread-safe access to cached dictionaries
+ */
+public class CompressionDictionaryCache implements ICompressionDictionaryCache
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryCache.class);
+
+ private final Cache<CompressionDictionary.DictId, CompressionDictionary> cache;
+ private final AtomicReference<CompressionDictionary> currentDictionary = new AtomicReference<>();
+
+ public CompressionDictionaryCache()
+ {
+ Duration expiryTime = Duration.ofSeconds(DatabaseDescriptor.getCompressionDictionaryCacheExpireSeconds());
Review Comment:
Originally it was inline. But the line is too long. I am fine with inline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414972088
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
+
+# Dictionary training configuration (advanced settings)
+# These settings control how compression dictionaries are trained from sample data.
+
+# Maximum size of a trained compression dictionary in bytes.
+# Larger dictionaries may provide better compression but use more memory.
+# Min unit: B
+compression_dictionary_training_max_dictionary_size: 65536
Review Comment:
Agreed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410623394
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Handles compression dictionary events including training completion and cluster notifications.
+ * <p>
+ * This class handles:
+ * - Broadcasting dictionary updates to cluster nodes
+ * - Retrieving new dictionaries when notified by other nodes
+ * - Managing dictionary cache updates
+ */
+public class CompressionDictionaryEventHandler implements ICompressionDictionaryEventHandler
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryEventHandler.class);
+
+ private final ColumnFamilyStore cfs;
+ private final String keyspaceName;
+ private final String tableName;
+ private final ICompressionDictionaryCache cache;
+
+ public CompressionDictionaryEventHandler(ColumnFamilyStore cfs, ICompressionDictionaryCache cache)
+ {
+ this.cfs = cfs;
+ this.keyspaceName = cfs.keyspace.getName();
+ this.tableName = cfs.getTableName();
+ this.cache = cache;
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ logger.info("Notifying cluster about dictionary update for {}.{} with dictionaryId {}",
+ keyspaceName, tableName, dictionaryId);
+
+ CompressionDictionaryUpdateMessage message = new CompressionDictionaryUpdateMessage(cfs.metadata().id, dictionaryId);
+ Collection<InetAddressAndPort> allNodes = ClusterMetadata.current().directory.allJoinedEndpoints();
+ // Broadcast notification using the fire-and-forget fashion
+ for (InetAddressAndPort node : allNodes)
+ {
+ if (node.equals(FBUtilities.getBroadcastAddressAndPort())) // skip ourself
+ continue;
+ sendNotification(node, message);
+ }
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictId)
+ {
+ // Best effort to retrieve the dictionary; otherwise, the periodic task should retrieve the dictionary later
+ CompletableFuture.runAsync(() -> {
+ try
+ {
+ if (!cfs.metadata().params.compression.isDictionaryCompressionEnabled())
+ {
Review Comment:
braces not necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415167879
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
Review Comment:
I am not sure whether it simplifies. I would like to have the null check on the trainer object for its existence too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415182762
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
Review Comment:
Thanks for calling it out. Looks like the code base is not consistent on using snake case or camel case. For example, `SnapshotOptions` uses camel case and `clearSnapshot` takes options using snake case. I am fine with either way.
The content of the map is not customer facing. It is purely for communication between tooling and server (jmx).
I will create a string constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2417128551
##########
src/java/org/apache/cassandra/db/compression/ZstdDictionaryTrainer.java:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.luben.zstd.Zstd;
+import com.github.luben.zstd.ZstdDictTrainer;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.CompressionDictionary.DictId;
+import org.apache.cassandra.db.compression.CompressionDictionary.Kind;
+import org.apache.cassandra.io.compress.IDictionaryCompressor;
+import org.apache.cassandra.io.compress.ZstdDictionaryCompressor;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.utils.Clock;
+
+/**
+ * Zstd implementation of dictionary trainer with lifecycle management.
+ */
+public class ZstdDictionaryTrainer implements ICompressionDictionaryTrainer
+{
+ private static final Logger logger = LoggerFactory.getLogger(ZstdDictionaryTrainer.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private final CompressionDictionaryTrainingConfig config;
+ private final AtomicLong totalSampleSize;
+ private final AtomicLong sampleCount;
+ private final int compressionLevel; // optimal if using the same level for training as when compressing.
+
+ // Sampling rate can be updated during training
+ private volatile int samplingRate;
+
+ // Minimum number of samples required by ZSTD library
+ private static final int MIN_SAMPLES_REQUIRED = 10;
+
+ private volatile Consumer<CompressionDictionary> dictionaryTrainedListener;
+ // TODO: manage the samples in this class for auto-train (follow-up). The ZstdDictTrainer cannot be re-used for multiple training runs.
+ private ZstdDictTrainer zstdTrainer;
+ private volatile boolean closed = false;
+ private volatile TrainingStatus currentTrainingStatus;
+
+ public ZstdDictionaryTrainer(String keyspaceName, String tableName,
+ CompressionDictionaryTrainingConfig config,
+ int compressionLevel)
+ {
+ this.keyspaceName = keyspaceName;
+ this.tableName = tableName;
+ this.config = config;
+ this.totalSampleSize = new AtomicLong(0);
+ this.sampleCount = new AtomicLong(0);
+ this.compressionLevel = compressionLevel;
+ this.samplingRate = config.samplingRate;
+ this.currentTrainingStatus = TrainingStatus.NOT_STARTED;
+ }
+
+ @Override
+ public boolean shouldSample()
+ {
+ return zstdTrainer != null && ThreadLocalRandom.current().nextInt(samplingRate) == 0;
+ }
+
+ @Override
+ public void addSample(ByteBuffer sample)
+ {
+ if (closed || sample == null || !sample.hasRemaining() || zstdTrainer == null)
+ return;
+
+ byte[] sampleBytes = new byte[sample.remaining()];
+ sample.duplicate().get(sampleBytes);
+
+ if (zstdTrainer.addSample(sampleBytes))
+ {
+ // Update the totalSampleSize and sampleCount if the sample is added
+ totalSampleSize.addAndGet(sampleBytes.length);
+ sampleCount.incrementAndGet();
+ }
+ }
+
+ @Override
+ public CompressionDictionary trainDictionary(boolean force)
+ {
+ boolean isReady = isReady();
+ if (!force && !isReady)
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ throw new IllegalStateException("Trainer is not ready");
+ }
+
+ long currentSampleCount = sampleCount.get();
+ if (currentSampleCount < MIN_SAMPLES_REQUIRED) // minimum samples should be required even if force training
+ {
+ currentTrainingStatus = TrainingStatus.FAILED;
+ String errorMsg = String.format("Insufficient samples for training: %d (minimum required: %d)",
+ currentSampleCount, MIN_SAMPLES_REQUIRED);
+ throw new IllegalStateException(errorMsg);
Review Comment:
This will be visible in server's logs:
java.lang.IllegalStateException: Insufficient samples for training: 0 (minimum required: 10)
at org.apache.cassandra.db.compression.ZstdDictionaryTrainer.trainDictionary(ZstdDictionaryTrainer.java:119)
at org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.lambda$trainDictionaryAsync$0(ICompressionDictionaryTrainer.java:80)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
I am not sure we want to have this visible like that, we should log it only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410718577
##########
src/java/org/apache/cassandra/io/compress/IDictionaryCompressor.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.compress;
+
+import org.apache.cassandra.db.compression.CompressionDictionary;
+
+/**
+ * Interface for compressors that support dictionary-based compression.
+ * <br>
+ * Dictionary compressors can use pre-trained compression dictionaries to achieve
+ * better compression ratios, especially for small data chunks that are similar
+ * to the training data used to create the dictionary.
+ *
+ * @param <T> the specific type of compression dictionary this compressor supports
+ */
+public interface IDictionaryCompressor<T extends CompressionDictionary>
Review Comment:
interesting, should not be `IDictionaryCompressor` extending `ICompressor`? Because in `CompressionParams` there is
private final ICompressor sstableCompressor;
and
public boolean isDictionaryCompressionEnabled()
{
return isEnabled() && sstableCompressor instanceof IDictionaryCompressor;
}
Which is a little bit confusing to me.
I think that `IDictionaryCompressor` extending `ICompressor` is way more conceptually logical.
##########
src/java/org/apache/cassandra/io/compress/IDictionaryCompressor.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.compress;
+
+import org.apache.cassandra.db.compression.CompressionDictionary;
+
+/**
+ * Interface for compressors that support dictionary-based compression.
+ * <br>
+ * Dictionary compressors can use pre-trained compression dictionaries to achieve
+ * better compression ratios, especially for small data chunks that are similar
+ * to the training data used to create the dictionary.
+ *
+ * @param <T> the specific type of compression dictionary this compressor supports
+ */
+public interface IDictionaryCompressor<T extends CompressionDictionary>
Review Comment:
https://github.com/apache/cassandra/pull/4399/files#r2410732332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410732332
##########
src/java/org/apache/cassandra/io/compress/ZstdDictionaryCompressor.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.compress;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import com.github.luben.zstd.Zstd;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.compression.ZstdCompressionDictionary;
+import org.apache.cassandra.db.compression.CompressionDictionary.Kind;
+import org.apache.cassandra.utils.concurrent.Ref;
+
+import javax.annotation.Nullable;
+
+public class ZstdDictionaryCompressor extends ZstdCompressorBase implements ICompressor, IDictionaryCompressor<ZstdCompressionDictionary>
Review Comment:
if `IDictionaryCompressor` extended `ICompressor` as suggested in `IDictionaryCompressor` interface, then `ZstdDictionaryCompressor` would implement `IDictionaryCompressor` only. `ZstdCompressorBase` implements `ICompressor` already so it is redundant here anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2421638889
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
Review Comment:
This should be fixed in https://github.com/apache/cassandra/pull/4399/commits/1ae168a2efc86aba50a74a3de4fb180ab3916112.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414946121
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
Review Comment:
I do not think starting all nodes at the same moment is a requirement. The refresh interval just defines how frequent each node should check the system table. When there is a new dictionary trained, the node will broadcast the message to all live nodes to notify the dictionary is available. This happens outside of the periodic polling.
Combining both, you can consider the interval as the upper bound of how dictionary is refreshed (when broadcasting fails at some nodes).
When some nodes are running with the prior dictionary or no dictionary, it is still not a huge issue, from the perspective of read and write SSTables. Eventually, all nodes will pick up the latest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#issuecomment-3383497076
I think all comments are addressed. Please tag me if I missed. (GH is not super friendly with many comments)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415149748
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
+ if (kind == Kind.ZSTD)
+ {
+ dictionary = new ZstdCompressionDictionary(dictId, dict);
+ }
+
+ if (dictionary == null)
+ {
+ throw new IOException(kind + " compression dictionary is not created");
+ }
+
+ // update the dictionary manager if it exists
+ if (manager != null)
+ {
+ manager.add(dictionary);
+ }
+ return dictionary;
+ }
+
+ static CompressionDictionary createFromRow(UntypedResultSet.Row row)
+ {
+ String kindStr = row.getString("kind");
+ long id = row.getLong("dict_id");
+ byte[] dict = row.getByteArray("dict");
+ CompressionDictionary.DictId dictId = new CompressionDictionary.DictId(CompressionDictionary.Kind.valueOf(kindStr), id);
Review Comment:
Again, credit goes to Stefan for the refactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415142156
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
Review Comment:
I prefer braces even for single statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "rustyrazorblade (via GitHub)" <gi...@apache.org>.
rustyrazorblade commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415182833
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
What about using a max # of samples instead of a duration? Duration is kind of weird, highly variable. Then a default also makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410701055
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
Review Comment:
I feel like this would be simplified if we did not put this logic around "isEnabled" here but we would pass isEnabled into `ICompressionDictionaryTrainer.create` and there, if it is `false`, we would return a dummy / no-op (singleton) `ICompressionDictionaryTrainer` doing nothing. Then it would not matter that we would call `trainer.start(false)` on a dummy trainer and similar, we would not need to check if a trainer is null or not further in the code.
`this::handleNewDictionary` can be also part of `create` method and set internally, that's purely optional.
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
Review Comment:
here, no-op trainer would always return `false` on `shouldSample`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410950069
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
Review Comment:
Preconditions.checkArgument ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415142156
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
Review Comment:
I prefer braces even for single statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410376440
##########
src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java:
##########
@@ -329,6 +329,7 @@ protected long getExpectedWriteSize()
.setRepairedAt(minRepairedAt)
.setPendingRepair(pendingRepair)
.setSecondaryIndexGroups(cfs.indexManager.listIndexGroups())
- .addDefaultComponents(cfs.indexManager.listIndexGroups());
+ .addDefaultComponents(cfs.indexManager.listIndexGroups())
+ .setCompressionDictionaryManager(cfs.compressionDictionaryManager());
Review Comment:
same, who is closing this?
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
Review Comment:
Do we really need `long` for this? Hard to imagine that `DictId.id` would be bigger than int which is 2,147,483,647 max.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415191161
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
Review Comment:
changed to throw `UnsupportedOperationException` as suggested in another comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414979062
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
+ }
+
+ CompressionDictionary dictionary = null;
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2425782938
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
SSTable-based sampling - that one which also solves Jon's usecase by training existing data? Yeah I think that is more robust. My instinct is that by write-based training we are trying to be too smart, while it works more "real-time-ish" it has also unpredictable behaviour if not enough writes or similar ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410948552
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
Review Comment:
here, no-op trainer would always return `false` on `shouldSample`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2427256985
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
that's a good question. What do you do when there is 10 SSTables already (or if somebody is changing compression algorithm)? You take the last one (newest one) or the first one? (oldest one?) I would take the newest one.
What if you flush and that flush will not have enough data? Like it would flush too few data. Then you take the older one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2427264076
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
Stefan, please check out https://github.com/apache/cassandra/commit/f7a66b47f382ade1fb689135056e2690f128a136 first. The sstable sampling is already implemented in this commit. It basically does random sampling across all data on disk. What it does not have is running 'flush' in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414884582
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryEventHandler.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.Message;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.net.Verb;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "rustyrazorblade (via GitHub)" <gi...@apache.org>.
rustyrazorblade commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414964532
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
+# Min unit: s
+compression_dictionary_refresh_initial_delay: 10s
+
+# Maximum number of compression dictionaries to cache per table.
+# Each table using dictionary compression can have multiple dictionaries cached
+# (current version plus recently used versions for reading older SSTables).
+compression_dictionary_cache_size: 10
+
+# How long to keep compression dictionaries in the cache before they expire.
+# Expired dictionaries will be removed from memory but can be reloaded if needed.
+# Min unit: s
+compression_dictionary_cache_expire: 3600s
Review Comment:
Why such an aggressive expiration time? I'd make this 24h.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "rustyrazorblade (via GitHub)" <gi...@apache.org>.
rustyrazorblade commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414968688
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
+# These settings control the automatic training and caching of compression dictionaries
+# for tables that use ZSTD dictionary compression.
+
+# How often to refresh compression dictionaries across the cluster.
+# During refresh, nodes will check for newer dictionary versions and update their caches.
+# Min unit: s
+compression_dictionary_refresh_interval: 3600s
+
+# Initial delay before starting the first dictionary refresh cycle after node startup.
+# This prevents all nodes from refreshing simultaneously when the cluster starts.
Review Comment:
What is the overhead of this? This seems like a huge bikeshed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415157643
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryCache.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import javax.annotation.Nullable;
+
+/**
+ * Interface for managing compression dictionary caching and current dictionary state.
+ * <p>
+ * Implementations handle:
+ * - Local caching of compression dictionaries with automatic cleanup
+ * - Managing the current active dictionary for write operations
+ * - Thread-safe access to cached dictionaries
+ */
+public interface ICompressionDictionaryCache extends AutoCloseable
+{
+ /**
+ * Gets the current active compression dictionary.
+ *
+ * @return the current compression dictionary, or null if no dictionary is available
+ */
+ @Nullable
+ CompressionDictionary getCurrent();
Review Comment:
Not a huge fan of `Optional`. Null exists anyway, and the callsite would see null in certain cases anyway. If the language can fully eliminate null, then `Optional` makes sense.
```
CompressionDictionary compressionDictionary = compressionDictionaryManager == null ? null : compressionDictionaryManager.getCurrent();
```
##########
src/java/org/apache/cassandra/db/compression/ICompressionDictionaryCache.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import javax.annotation.Nullable;
+
+/**
+ * Interface for managing compression dictionary caching and current dictionary state.
+ * <p>
+ * Implementations handle:
+ * - Local caching of compression dictionaries with automatic cleanup
+ * - Managing the current active dictionary for write operations
+ * - Thread-safe access to cached dictionaries
+ */
+public interface ICompressionDictionaryCache extends AutoCloseable
+{
+ /**
+ * Gets the current active compression dictionary.
+ *
+ * @return the current compression dictionary, or null if no dictionary is available
+ */
+ @Nullable
+ CompressionDictionary getCurrent();
Review Comment:
Not a huge fan of `Optional`. Null exists anyway, and the callsite would see null in certain cases. If the language can fully eliminate null, then `Optional` makes sense.
```
CompressionDictionary compressionDictionary = compressionDictionaryManager == null ? null : compressionDictionaryManager.getCurrent();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415164121
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
Review Comment:
ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415167879
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
Review Comment:
I am not sure whether it simplifies. I would like to have the null check on the trainer object for its existence too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415172972
##########
src/java/org/apache/cassandra/db/compression/ManualTrainingOptions.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Map;
+
+/**
+ * Configuration options for manual compression dictionary training.
+ * This class encapsulates the parsed and validated parameters needed for training.
+ */
+public class ManualTrainingOptions
+{
+ private final int maxSamplingDurationSeconds;
+
+ public ManualTrainingOptions(int maxSamplingDurationSeconds)
+ {
+ if (maxSamplingDurationSeconds <= 0)
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds must be positive, got: " + maxSamplingDurationSeconds);
+ }
+ this.maxSamplingDurationSeconds = maxSamplingDurationSeconds;
+ }
+
+ /**
+ * Parse options from a string map, typically from JMX/MBean calls.
+ *
+ * @param options the string map containing training options
+ * @return parsed and validated ManualTrainingOptions
+ * @throws IllegalArgumentException if required parameters are missing or invalid
+ */
+ public static ManualTrainingOptions fromStringMap(Map<String, String> options)
+ {
+ if (options == null || !options.containsKey("maxSamplingDurationSeconds"))
+ {
+ throw new IllegalArgumentException("maxSamplingDurationSeconds parameter is required for manual dictionary training");
Review Comment:
The nodetool command is supposed to provide a value, always. So the exception should never be thrown. I had been weighing both options for a while; the reason of not providing a default here is that 1) it is not going to be used, and 2) two default values in nodetool and server could diverge and create confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2415141369
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
Review Comment:
yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410425989
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionary.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+
+import org.apache.cassandra.cql3.UntypedResultSet;
+
+public interface CompressionDictionary extends AutoCloseable
+{
+ /**
+ * Get the dictionary id
+ *
+ * @return dictionary id
+ */
+ DictId identifier();
+
+ /**
+ * Get the raw bytes of the compression dictionary
+ *
+ * @return raw compression dictionary
+ */
+ byte[] rawDictionary();
+
+ /**
+ * Get the kind of the compression algorithm
+ *
+ * @return compression algorithm kind
+ */
+ default Kind kind()
+ {
+ return identifier().kind;
+ }
+
+ /**
+ * Write compression dictionary to file
+ *
+ * @param out file output stream
+ * @throws IOException on any I/O exception when writing to the file
+ */
+ default void serialize(DataOutput out) throws IOException
+ {
+ DictId dictId = identifier();
+ int ordinal = dictId.kind.ordinal();
+ out.writeByte(ordinal);
+ out.writeLong(dictId.id);
+ byte[] dict = rawDictionary();
+ out.writeInt(dict.length);
+ out.write(dict);
+ int checksum = calculateChecksum((byte) ordinal, dictId.id, dict);
+ out.writeInt(checksum);
+ }
+
+ /**
+ * A factory method to create concrete CompressionDictionary from the file content
+ *
+ * @param input file input stream
+ * @param manager compression dictionary manager that caches the dictionaries
+ * @return compression dictionary; otherwise, null if there is no dictionary
+ * @throws IOException on any I/O exception when reading from the file
+ */
+ @Nullable
+ static CompressionDictionary deserialize(DataInput input, @Nullable CompressionDictionaryManager manager) throws IOException
+ {
+ int kindOrdinal;
+ try
+ {
+ kindOrdinal = input.readByte();
+ }
+ catch (EOFException eof)
+ {
+ // no dictionary
+ return null;
+ }
+
+ if (kindOrdinal < 0 || kindOrdinal >= Kind.values().length)
+ {
+ throw new IOException("Invalid compression dictionary kind: " + kindOrdinal);
+ }
+ Kind kind = Kind.values()[kindOrdinal];
+ long id = input.readLong();
+ DictId dictId = new DictId(kind, id);
+
+ if (manager != null)
+ {
+ CompressionDictionary dictionary = manager.get(dictId);
+ if (dictionary != null)
+ {
+ return dictionary;
+ }
+ }
+
+ int length = input.readInt();
+ byte[] dict = new byte[length];
+ input.readFully(dict);
+ int checksum = input.readInt();
+ int calculatedChecksum = calculateChecksum((byte) kindOrdinal, id, dict);
+ if (checksum != calculatedChecksum)
+ {
+ throw new IOException("Compression dictionary checksum does not match");
Review Comment:
include both values of `checksum` and `calculatedChecksum` in the message perhaps. A reader of such a message happens to wonder what was computed and what was serialized.
##########
src/java/org/apache/cassandra/db/compression/ZstdCompressionDictionary.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.github.luben.zstd.ZstdDictCompress;
+import com.github.luben.zstd.ZstdDictDecompress;
+import org.apache.cassandra.io.compress.ZstdCompressorBase;
+import org.apache.cassandra.utils.concurrent.Ref;
+import org.apache.cassandra.utils.concurrent.RefCounted;
+import org.apache.cassandra.utils.concurrent.SelfRefCounted;
+
+public class ZstdCompressionDictionary implements CompressionDictionary, SelfRefCounted<ZstdCompressionDictionary>
+{
+ private static final Logger logger = LoggerFactory.getLogger(ZstdCompressionDictionary.class);
+
+ private final DictId dictId;
+ private final byte[] rawDictionary;
+ // One ZstdDictDecompress and multiple ZstdDictCompress (per level) can be derived from the same raw dictionary content
+ private final ConcurrentHashMap<Integer, ZstdDictCompress> zstdDictCompressPerLevel = new ConcurrentHashMap<>();
+ private volatile ZstdDictDecompress dictDecompress;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final Ref<ZstdCompressionDictionary> selfRef;
+
+ public ZstdCompressionDictionary(DictId dictId, byte[] rawDictionary)
+ {
+ this.dictId = dictId;
+ this.rawDictionary = rawDictionary;
+ this.selfRef = new Ref<>(this, new Tidy(zstdDictCompressPerLevel, dictDecompress));
+ }
+
+ @Override
+ public DictId identifier()
+ {
+ return dictId;
+ }
+
+ @Override
+ public Kind kind()
+ {
+ return Kind.ZSTD;
+ }
+
+ @Override
+ public byte[] rawDictionary()
+ {
+ return rawDictionary;
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (!(o instanceof ZstdCompressionDictionary)) return false;
+ ZstdCompressionDictionary that = (ZstdCompressionDictionary) o;
+ return Objects.equals(dictId, that.dictId);
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return dictId.hashCode();
+ }
+
+ /**
+ * Get a pre-processed compression tables that is optimized for compression.
+ * It is derived/computed from dictionary bytes.
+ * The internal data structure is different from the tables for decompression.
+ *
+ * @param compressionLevel compression level to create the compression table
+ * @return ZstdDictCompress
+ */
+ public ZstdDictCompress dictionaryForCompression(int compressionLevel)
+ {
+ if (closed.get())
+ throw new IllegalStateException("Dictionary has been closed");
Review Comment:
maybe being more descriptive in this message (and all other same messages) by including dict id with kind and id would be nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410306927
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
Review Comment:
I am also not completely sure this is the right section of the yaml to put all of this in. Wondering based on what it was put here. Not saying it is wrong, just saying that this is probably the opportunity to be more thoughtful and intentional about where we put it. Given this is probably experimental feature we might also consider to put it among experimental ones at the end of the yaml? Just shooting thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2414928946
##########
conf/cassandra.yaml:
##########
@@ -617,6 +617,54 @@ counter_cache_save_period: 7200s
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100
+# Dictionary compression settings for ZSTD dictionary-based compression
Review Comment:
Make sense. Moved the section to the bottom and replicated to the `cassandra_latest.yaml` too (at the end).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-17021: Support ZSTD dictionary compression [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #4399:
URL: https://github.com/apache/cassandra/pull/4399#discussion_r2410959195
##########
src/java/org/apache/cassandra/db/compression/CompressionDictionaryManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compression;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.compression.ICompressionDictionaryTrainer.TrainingStatus;
+import org.apache.cassandra.schema.CompressionParams;
+import org.apache.cassandra.schema.SystemDistributedKeyspace;
+import org.apache.cassandra.utils.MBeanWrapper;
+import org.apache.cassandra.utils.MBeanWrapper.OnException;
+
+public class CompressionDictionaryManager implements CompressionDictionaryManagerMBean,
+ ICompressionDictionaryCache,
+ ICompressionDictionaryEventHandler,
+ AutoCloseable
+{
+ private static final Logger logger = LoggerFactory.getLogger(CompressionDictionaryManager.class);
+
+ private final String keyspaceName;
+ private final String tableName;
+ private volatile boolean mbeanRegistered;
+ private volatile boolean isEnabled;
+
+ // Components
+ private final ICompressionDictionaryEventHandler eventHandler;
+ private final ICompressionDictionaryCache cache;
+ private final ICompressionDictionaryScheduler scheduler;
+ private ICompressionDictionaryTrainer trainer = null;
+
+ public CompressionDictionaryManager(ColumnFamilyStore columnFamilyStore, boolean registerBookkeeping)
+ {
+ this.keyspaceName = columnFamilyStore.keyspace.getName();
+ this.tableName = columnFamilyStore.getTableName();
+
+ this.isEnabled = columnFamilyStore.metadata().params.compression.isDictionaryCompressionEnabled();
+ this.cache = new CompressionDictionaryCache();
+ this.eventHandler = new CompressionDictionaryEventHandler(columnFamilyStore, cache);
+ this.scheduler = new CompressionDictionaryScheduler(keyspaceName, tableName, cache, isEnabled);
+ if (isEnabled)
+ {
+ // Initialize components
+ this.trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName,
+ columnFamilyStore.metadata().params.compression,
+ createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+
+ scheduler.scheduleRefreshTask();
+
+ trainer.start(false);
+ }
+
+ if (registerBookkeeping)
+ {
+ MBeanWrapper.instance.registerMBean(this, mbeanName(keyspaceName, tableName));
+ }
+ mbeanRegistered = registerBookkeeping;
+ }
+
+ static String mbeanName(String keyspaceName, String tableName)
+ {
+ return "org.apache.cassandra.db.compression:type=CompressionDictionaryManager" +
+ ",keyspace=" + keyspaceName + ",table=" + tableName;
+ }
+
+ public boolean isEnabled()
+ {
+ return isEnabled;
+ }
+
+ /**
+ * Reloads dictionary management configuration when compression parameters change.
+ * This method enables or disables dictionary compression based on the new parameters,
+ * and properly manages the lifecycle of training and refresh tasks.
+ *
+ * @param newParams the new compression parameters to apply
+ */
+ public synchronized void maybeReloadFromSchema(CompressionParams newParams)
+ {
+ this.isEnabled = newParams.isDictionaryCompressionEnabled();
+ scheduler.setEnabled(isEnabled);
+ if (isEnabled)
+ {
+ // Check if we need a new trainer due to compression parameter changes
+ boolean needsNewTrainer = shouldCreateNewTrainer(newParams);
+
+ if (needsNewTrainer)
+ {
+ // The manual training should be cancelled if a new trainer is needed
+ scheduler.cancelManualTraining();
+ // Close existing trainer and create a new one
+ if (trainer != null)
+ {
+ try
+ {
+ trainer.close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close existing trainer for {}.{}", keyspaceName, tableName, e);
+ }
+ }
+
+ trainer = ICompressionDictionaryTrainer.create(keyspaceName, tableName, newParams, createTrainingConfig());
+ trainer.setDictionaryTrainedListener(this::handleNewDictionary);
+ }
+
+ scheduler.scheduleRefreshTask();
+
+ // Start trainer if it exists
+ if (trainer != null)
+ {
+ trainer.start(false);
+ }
+ return;
+ }
+
+ // Clean up when dictionary compression is disabled
+ try
+ {
+ close();
+ }
+ catch (Exception e)
+ {
+ logger.warn("Failed to close CompressionDictionaryManager on disabling " +
+ "dictionary-based compression for table {}.{}", keyspaceName, tableName);
+ }
+ }
+
+ /**
+ * Adds a sample to the dictionary trainer for learning compression patterns.
+ * Samples are randomly selected to avoid bias and improve dictionary quality.
+ *
+ * @param sample the sample data to potentially add for training
+ */
+ public void addSample(ByteBuffer sample)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer != null && dictionaryTrainer.shouldSample())
+ {
+ dictionaryTrainer.addSample(sample);
+ }
+ }
+
+ @Nullable
+ @Override
+ public CompressionDictionary getCurrent()
+ {
+ return cache.getCurrent();
+ }
+
+ @Override
+ public CompressionDictionary get(CompressionDictionary.DictId dictId)
+ {
+ return cache.get(dictId);
+ }
+
+ @Override
+ public void add(CompressionDictionary compressionDictionary)
+ {
+ cache.add(compressionDictionary);
+ }
+
+ @Override
+ public void setCurrentIfNewer(@Nullable CompressionDictionary dictionary)
+ {
+ cache.setCurrentIfNewer(dictionary);
+ }
+
+ @Override
+ public void onNewDictionaryTrained(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryTrained(dictionaryId);
+ }
+
+ @Override
+ public void onNewDictionaryAvailable(long dictionaryId)
+ {
+ eventHandler.onNewDictionaryAvailable(dictionaryId);
+ }
+
+ @Override
+ public synchronized void train(Map<String, String> options)
+ {
+ // Validate table supports dictionary compression
+ if (!isEnabled)
+ {
+ throw new IllegalArgumentException("Table " + keyspaceName + '.' + tableName + " does not support dictionary compression");
+ }
+
+ if (trainer == null)
+ {
+ throw new IllegalStateException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+
+ // Parse and validate training options
+ ManualTrainingOptions trainingOptions = ManualTrainingOptions.fromStringMap(options);
+
+ trainer.start(true);
+ scheduler.scheduleManualTraining(trainingOptions, trainer);
+ }
+
+ @Override
+ public String getTrainingStatus()
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer == null)
+ {
+ return TrainingStatus.NOT_STARTED.toString();
+ }
+ return dictionaryTrainer.getTrainingStatus().toString();
+ }
+
+ @Override
+ public void updateSamplingRate(int samplingRate)
+ {
+ ICompressionDictionaryTrainer dictionaryTrainer = trainer;
+ if (dictionaryTrainer == null)
+ {
+ throw new IllegalArgumentException("Dictionary trainer is not available for table " + keyspaceName + '.' + tableName);
+ }
+ dictionaryTrainer.updateSamplingRate(samplingRate);
+ }
+
+ /**
+ * Close all the resources. The method can be called multiple times.
+ */
+ @Override
+ public synchronized void close()
+ {
+ unregisterMbean();
+ if (trainer != null)
Review Comment:
closing no-op trainer would do nothing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org