You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by je...@apache.org on 2025/05/01 17:57:59 UTC
(cassandra) branch trunk updated (58d1cc9b1e -> 3bc266ca96)
This is an automated email from the ASF dual-hosted git repository.
jeetkundoug pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
from 58d1cc9b1e Increasing default for auto_repair.sstable_upper_threshold considering large Cassandra tables
add 2aeb6a0290 CASSANDRA-20609 - CQLSSTableWriter should support setting the format (5.0)
new 3bc266ca96 Merge branch 'cassandra-5.0' into trunk
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.txt | 1 +
.../cassandra/io/sstable/CQLSSTableWriter.java | 12 ++
.../cassandra/io/sstable/CQLSSTableWriterTest.java | 129 +++++++++++++--------
3 files changed, 95 insertions(+), 47 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jeetkundoug pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 3bc266ca963c291a68a87be50d98e9ba0a6cab51
Merge: 58d1cc9b1e 2aeb6a0290
Author: Doug Rohrer <do...@therohrers.org>
AuthorDate: Thu May 1 13:37:27 2025 -0400
Merge branch 'cassandra-5.0' into trunk
* cassandra-5.0:
CASSANDRA-20609 - CQLSSTableWriter should support setting the format (5.0)
CHANGES.txt | 1 +
.../cassandra/io/sstable/CQLSSTableWriter.java | 12 ++
.../cassandra/io/sstable/CQLSSTableWriterTest.java | 129 +++++++++++++--------
3 files changed, 95 insertions(+), 47 deletions(-)
diff --cc CHANGES.txt
index b07586d866,08e0ecfa16..35cafede97
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,195 -1,13 +1,196 @@@
-5.0.5
+5.1
+ * Increasing default for auto_repair.sstable_upper_threshold considering large Cassandra tables & revert three lines removed from CHANGES.txt due to a merge mistake (CASSANDRA-20586)
+ * Fix token restrictions with MIN_TOKEN (CASSANDRO-20557)
+ * Upgrade logback version to 1.5.18 and slf4j dependencies to 2.0.17 (CASSANDRA-20429)
+ * Switch memtable-related off-heap objects to Native Endian and Memory to Little Endian (CASSANDRA-20190)
+ * Change SSTableSimpleScanner to use SSTableReader#openDataReaderForScan (CASSANDRA-20538)
+ * Automated Repair Inside Cassandra [CEP-37] (CASSANDRA-19918)
+ * Implement appender of slow queries to system_views.slow_queries table (CASSANDRA-13001)
+ * Add autocompletion in CQLSH for built-in functions (CASSANDRA-19631)
+ * Grant permission on keyspaces system_views and system_virtual_schema not possible (CASSANDRA-20171)
+ * General Purpose Transactions (Accord) [CEP-15] (CASSANDRA-17092)
+ * Improve performance when getting writePlacementsAllSettled from ClusterMetadata (CASSANDRA-20526)
+ * Add nodetool command to dump the contents of the system_views.{cluster_metadata_log, cluster_metadata_directory} tables (CASSANDRA-20525)
+ * Fix TreeMap race in CollectionVirtualTableAdapter causing us to lose rows in the virtual table (CASSANDRA-20524)
+ * Improve metadata log catch up with inter-DC mutation forwarding (CASSANDRA-20523)
+ * Support topology-safe changes to Datacenter & Rack for live nodes (CASSANDRA-20528)
+ * Add SSTableIntervalTree latency metric (CASSANDRA-20502)
+ * Ignore repetitions of semicolon in CQLSH (CASSANDRA-19956)
+ * Avoid NPE during cms initialization abort (CASSANDRA-20527)
+ * Avoid failing queries when epoch changes and replica goes up/down (CASSANDRA-20489)
+ * Split out truncation record lock (CASSANDRA-20480)
+ * Throw new IndexBuildInProgressException when queries fail during index build, instead of IndexNotAvailableException (CASSANDRA-20402)
+ * Fix Paxos repair interrupts running transactions (CASSANDRA-20469)
+ * Various fixes in constraint framework (CASSANDRA-20481)
+ * Add support in CAS for -= on numeric types, and fixed improper handling of empty bytes which lead to NPE (CASSANDRA-20477)
+ * Do not fail to start a node with materialized views after they are turned off in config (CASSANDRA-20452)
+ * Fix nodetool gcstats output, support human-readable units and more output formats (CASSANDRA-19022)
+ * Various gossip to TCM upgrade fixes (CASSANDRA-20483)
+ * Add nodetool command to abort failed nodetool cms initialize (CASSANDRA-20482)
+ * Repair Paxos for the distributed metadata log when CMS membership changes (CASSANDRA-20467)
+ * Reintroduce CASSANDRA-17411 in trunk (CASSANDRA-19346)
+ * Add min/max/mean/percentiles to timer metrics vtable (CASSANDRA-20466)
+ * Add support for time, date, timestamp types in scalar constraint (CASSANDRA-20274)
+ * Add regular expression constraint (CASSANDRA-20275)
+ * Improve constraints autocompletion (CASSANDRA-20341)
+ * Add JVM version and Cassandra build date to nodetool version -v (CASSANDRA-19721)
+ * Move all disk error logic to DiskErrorsHandler to enable pluggability (CASSANDRA-20363)
+ * Fix marking an SSTable as suspected and BufferPool leakage in case of a corrupted SSTable read during a compaction (CASSANDRA-20396)
+ * Add missed documentation for CREATE TABLE LIKE (CASSANDRA-20401)
+ * Add OCTET_LENGTH constraint (CASSANDRA-20340)
+ * Reduce memory allocations in miscellaneous places along the hot write path (CASSANDRA-20167)
+ * Provide keystore_password_file and truststore_password_file options to read credentials from a file (CASSANDRA-13428)
+ * Unregistering a node should also remove it from tokenMap if it is there and recalculate the placements (CASSANDRA-20346)
+ * Fix PartitionUpdate.isEmpty deserialization issue to avoid potential EOFException (CASSANDRA-20345)
+ * Avoid adding LEFT nodes to tokenMap on upgrade from gossip (CASSANDRA-20344)
+ * Allow empty placements when deserializing cluster metadata (CASSANDRA-20343)
+ * Reduce heap pressure when initializing CMS (CASSANDRA-20267)
+ * Paxos Repair: NoSuchElementException on DistributedSchema.getKeyspaceMetadata (CASSANDRA-20320)
+ * Improve performance of DistributedSchema.validate for large schemas (CASSANDRA-20360)
+ * Add JSON constraint (CASSANDRA-20273)
+ * Prevent invalid constraint combinations (CASSANDRA-20330)
+ * Support CREATE TABLE LIKE WITH INDEXES (CASSANDRA-19965)
+ * Invalidate relevant prepared statements on every change to TableMetadata (CASSANDRA-20318)
+ * Add per type max size guardrails (CASSANDRA-19677)
+ * Make it possible to abort all kinds of multi step operations (CASSANDRA-20217)
+ * Do not leak non-Java exceptions when calling snapshot operations via JMX (CASSANDRA-20335)
+ * Implement NOT_NULL constraint (CASSANDRA-20276)
+ * Improve error messages for constraints (CASSANDRA-20266)
+ * Add system_views.partition_key_statistics for querying SSTable metadata (CASSANDRA-20161)
+ * CEP-42 - Add Constraints Framework (CASSANDRA-19947)
+ * Add table metric PurgeableTombstoneScannedHistogram and a tracing event for scanned purgeable tombstones (CASSANDRA-20132)
+ * Make sure we can parse the expanded CQL before writing it to the log or sending it to replicas (CASSANDRA-20218)
+ * Add format_bytes and format_time functions (CASSANDRA-19546)
+ * Fix error when trying to assign a tuple to target type not being a tuple (CASSANDRA-20237)
+ * Fail CREATE TABLE LIKE statement if UDTs in target keyspace do not exist or they have different structure from ones in source keyspace (CASSANDRA-19966)
+ * Support octet_length and length functions (CASSANDRA-20102)
+ * Make JsonUtils serialize Instant always with the same format (CASSANDRA-20209)
+ * Port Harry v2 to trunk (CASSANDRA-20200)
+ * Enable filtering of snapshots on keyspace, table and snapshot name in nodetool listsnapshots (CASSANDRA-20151)
+ * Create manifest upon loading where it does not exist or enrich it (CASSANDRA-20150)
+ * Propagate true size of snapshot in SnapshotDetailsTabularData to not call JMX twice in nodetool listsnapshots (CASSANDRA-20149)
+ * Implementation of CEP-43 - copying a table via CQL by CREATE TABLE LIKE (CASSANDRA-19964)
+ * Periodically disconnect roles that are revoked or have LOGIN=FALSE set (CASSANDRA-19385)
+ * AST library for CQL-based fuzz tests (CASSANDRA-20198)
+ * Support audit logging for JMX operations (CASSANDRA-20128)
+ * Enable sorting of nodetool status output (CASSANDRA-20104)
+ * Support downgrading after CMS is initialized (CASSANDRA-20145)
+ * Deprecate IEndpointSnitch (CASSANDRA-19488)
+ * Check presence of a snapshot in a case-insensitive manner on macOS platform to prevent hardlinking failures (CASSANDRA-20146)
+ * Enable JMX server configuration to be in cassandra.yaml (CASSANDRA-11695)
+ * Parallelized UCS compactions (CASSANDRA-18802)
+ * Avoid prepared statement invalidation race when committing schema changes (CASSANDRA-20116)
+ * Restore optimization in MultiCBuilder around building one clustering (CASSANDRA-20129)
+ * Consolidate all snapshot management to SnapshotManager and introduce SnapshotManagerMBean (CASSANDRA-18111)
+ * Fix RequestFailureReason constants codes (CASSANDRA-20126)
+ * Introduce SSTableSimpleScanner for compaction (CASSANDRA-20092)
+ * Include column drop timestamp in alter table transformation (CASSANDRA-18961)
+ * Make JMX SSL configurable in cassandra.yaml (CASSANDRA-18508)
+ * Fix cqlsh CAPTURE command to save query results without trace details when TRACING is ON (CASSANDRA-19105)
+ * Optionally prevent tombstone purging during repair (CASSANDRA-20071)
+ * Add post-filtering support for the IN operator in SAI queries (CASSANDRA-20025)
+ * Don’t finish ongoing decommission and move operations during startup (CASSANDRA-20040)
+ * Nodetool reconfigure cms has correct return code when streaming fails (CASSANDRA-19972)
+ * Reintroduce RestrictionSet#iterator() optimization around multi-column restrictions (CASSANDRA-20034)
+ * Explicitly localize strings to Locale.US for internal implementation (CASSANDRA-19953)
+ * Add -H option for human-friendly output in nodetool compactionhistory (CASSANDRA-20015)
+ * Fix type check for referenced duration type for nested types (CASSANDRA-19890)
+ * In simulation tests, correctly set the tokens of replacement nodes (CASSANDRA-19997)
+ * During TCM upgrade, retain all properties of existing system tables (CASSANDRA-19992)
+ * Properly cancel in-flight futures and reject requests in EpochAwareDebounce during shutdown (CASSANDRA-19848)
+ * Provide clearer exception message on failing commitlog_disk_access_mode combinations (CASSANDRA-19812)
+ * Add total space used for a keyspace to nodetool tablestats (CASSANDRA-19671)
+ * Ensure Relation#toRestriction() handles ReversedType properly (CASSANDRA-19950)
+ * Add JSON and YAML output option to nodetool gcstats (CASSANDRA-19771)
+ * Introduce metadata serialization version V4 (CASSANDRA-19970)
+ * Allow CMS reconfiguration to work around DOWN nodes (CASSANDRA-19943)
+ * Make TableParams.Serializer set allowAutoSnapshots and incrementalBackups (CASSANDRA-19954)
+ * Make sstabledump possible to show tombstones only (CASSANDRA-19939)
+ * Ensure that RFP queries potentially stale replicas even with only key columns in the row filter (CASSANDRA-19938)
+ * Allow nodes to change IP address while upgrading to TCM (CASSANDRA-19921)
+ * Retain existing keyspace params on system tables after upgrade (CASSANDRA-19916)
+ * Deprecate use of gossip state for paxos electorate verification (CASSANDRA-19904)
+ * Update dtest-api to 0.0.17 to fix jvm17 crash in jvm-dtests (CASSANDRA-19239)
+ * Add resource leak test and Update Netty to 4.1.113.Final to fix leak (CASSANDRA-19783)
+ * Fix incorrect nodetool suggestion when gossip mode is running (CASSANDRA-19905)
+ * SAI support for BETWEEN operator (CASSANDRA-19688)
+ * Fix BETWEEN filtering for reversed clustering columns (CASSANDRA-19878)
+ * Retry if node leaves CMS while committing a transformation (CASSANDRA-19872)
+ * Add support for NOT operators in WHERE clauses. Fixed Three Valued Logic (CASSANDRA-18584)
+ * Allow getendpoints for system tables and make sure getNaturalReplicas work for MetaStrategy (CASSANDRA-19846)
+ * On upgrade, handle pre-existing tables with unexpected table ids (CASSANDRA-19845)
+ * Reconfigure CMS before assassinate (CASSANDRA-19768)
+ * Warn about unqualified prepared statement only if it is select or modification statement (CASSANDRA-18322)
+ * Update legacy peers tables during node replacement (CASSANDRA-19782)
+ * Refactor ColumnCondition (CASSANDRA-19620)
+ * Allow configuring log format for Audit Logs (CASSANDRA-19792)
+ * Support for noboolean rpm (centos7 compatible) packages removed (CASSANDRA-19787)
+ * Allow threads waiting for the metadata log follower to be interrupted (CASSANDRA-19761)
+ * Support dictionary lookup for CassandraPasswordValidator (CASSANDRA-19762)
+ * Disallow denylisting keys in system_cluster_metadata (CASSANDRA-19713)
+ * Fix gossip status after replacement (CASSANDRA-19712)
+ * Ignore repair requests for system_cluster_metadata (CASSANDRA-19711)
+ * Avoid ClassCastException when verifying tables with reversed partitioner (CASSANDRA-19710)
+ * Always repair the full range when repairing system_cluster_metadata (CASSANDRA-19709)
+ * Use table-specific partitioners during Paxos repair (CASSANDRA-19714)
+ * Expose current compaction throughput in nodetool (CASSANDRA-13890)
+ * CEP-24 Password validation / generation (CASSANDRA-17457)
+ * Reconfigure CMS after replacement, bootstrap and move operations (CASSANDRA-19705)
+ * Support querying LocalStrategy tables with any partitioner (CASSANDRA-19692)
+ * Relax slow_query_log_timeout for MultiNodeSAITest (CASSANDRA-19693)
+ * Audit Log entries are missing identity for mTLS connections (CASSANDRA-19669)
+ * Add support for the BETWEEN operator in WHERE clauses (CASSANDRA-19604)
+ * Replace Stream iteration with for-loop for SimpleRestriction::bindAndGetClusteringElements (CASSANDRA-19679)
+ * Consolidate logging on trace level (CASSANDRA-19632)
+ * Expand DDL statements on coordinator before submission to the CMS (CASSANDRA-19592)
+ * Fix number of arguments of String.format() in various classes (CASSANDRA-19645)
+ * Remove unused fields from config (CASSANDRA-19599)
+ * Refactor Relation and Restriction hierarchies (CASSANDRA-19341)
+ * Raise priority of TCM internode messages during critical operations (CASSANDRA-19517)
+ * Add nodetool command to unregister LEFT nodes (CASSANDRA-19581)
+ * Add cluster metadata id to gossip syn messages (CASSANDRA-19613)
+ * Reduce heap usage occupied by the metrics (CASSANDRA-19567)
+ * Improve handling of transient replicas during range movements (CASSANDRA-19344)
+ * Enable debounced internode log requests to be cancelled at shutdown (CASSANDRA-19514)
+ * Correctly set last modified epoch when combining multistep operations into a single step (CASSANDRA-19538)
+ * Add new TriggersPolicy configuration to allow operators to disable triggers (CASSANDRA-19532)
+ * Use Transformation.Kind.id in local and distributed log tables (CASSANDRA-19516)
+ * Remove period field from ClusterMetadata and metadata log tables (CASSANDRA-19482)
+ * Enrich system_views.pending_hints vtable with hints sizes (CASSANDRA-19486)
+ * Expose all dropwizard metrics in virtual tables (CASSANDRA-14572)
+ * Ensured that PropertyFileSnitchTest do not overwrite cassandra-toploogy.properties (CASSANDRA-19502)
+ * Add option for MutualTlsAuthenticator to restrict the certificate validity period (CASSANDRA-18951)
+ * Fix StorageService::constructRangeToEndpointMap for non-distributed keyspaces (CASSANDRA-19255)
+ * Group nodetool cms commands into single command group (CASSANDRA-19393)
+ * Register the measurements of the bootstrap process as Dropwizard metrics (CASSANDRA-19447)
+ * Add LIST SUPERUSERS CQL statement (CASSANDRA-19417)
+ * Modernize CQLSH datetime conversions (CASSANDRA-18879)
+ * Harry model and in-JVM tests for partition-restricted 2i queries (CASSANDRA-18275)
+ * Refactor cqlshmain global constants (CASSANDRA-19201)
+ * Remove native_transport_port_ssl (CASSANDRA-19397)
+ * Make nodetool reconfigurecms sync by default and add --cancel to be able to cancel ongoing reconfigurations (CASSANDRA-19216)
+ * Expose auth mode in system_views.clients, nodetool clientstats, metrics (CASSANDRA-19366)
+ * Remove sealed_periods and last_sealed_period tables (CASSANDRA-19189)
+ * Improve setup and initialisation of LocalLog/LogSpec (CASSANDRA-19271)
+ * Refactor structure of caching metrics and expose auth cache metrics via JMX (CASSANDRA-17062)
+ * Allow CQL client certificate authentication to work without sending an AUTHENTICATE request (CASSANDRA-18857)
+ * Extend nodetool tpstats and system_views.thread_pools with detailed pool parameters (CASSANDRA-19289)
+ * Remove dependency on Sigar in favor of OSHI (CASSANDRA-16565)
+ * Simplify the bind marker and Term logic (CASSANDRA-18813)
+ * Limit cassandra startup to supported JDKs, allow higher JDKs by setting CASSANDRA_JDK_UNSUPPORTED (CASSANDRA-18688)
+ * Standardize nodetool tablestats formatting of data units (CASSANDRA-19104)
+ * Make nodetool tablestats use number of significant digits for time and average values consistently (CASSANDRA-19015)
+ * Upgrade jackson to 2.15.3 and snakeyaml to 2.1 (CASSANDRA-18875)
+ * Transactional Cluster Metadata [CEP-21] (CASSANDRA-18330)
+ * Add ELAPSED command to cqlsh (CASSANDRA-18861)
+ * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781)
+ * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787)
+Merged from 5.0:
+ * CQLSSTableWriter supports setting the format (BTI or Big) (CASSANDRA-20609)
* Don't allocate in ThreadLocalReadAheadBuffer#close() (CASSANDRA-20551)
* Ensure RowFilter#isMutableIntersection() properly evaluates numeric ranges on a single column (CASSANDRA-20566)
- * Switch memtable-related off-heap objects to Native Endian and Memory to Little Endian (CASSANDRA-20190)
* SAI marks an index as non-empty when a partial partition/row modifications is flushed due to repair (CASSANDRA-20567)
* SAI fails queries when multiple columns exist and a non-indexed column is a composite with a map (CASSANDRA-19891)
- * Grant permission on keyspaces system_views and system_virtual_schema not possible (CASSANDRA-20171)
- * Fix marking an SSTable as suspected and BufferPool leakage in case of a corrupted SSTable read during a compaction (CASSANDRA-20396)
- * Introduce SSTableSimpleScanner for compaction (CASSANDRA-20092)
* Avoid purging deletions in RowFilter when reconciliation is required (CASSANDRA-20541)
* Fixed multiple single-node SAI query bugs relating to static columns (CASSANDRA-20338)
* Upgrade com.datastax.cassandra:cassandra-driver-core:3.11.5 to org.apache.cassandra:cassandra-driver-core:3.12.1 (CASSANDRA-17231)
diff --cc src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
index c37886364a,64834792fc..0676eb1f56
--- a/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java
@@@ -653,6 -648,18 +653,18 @@@ public class CQLSSTableWriter implement
return this;
}
+ /**
- * Whether the produced sstable should be open or not.
- * By default, the writer does not open the produced sstables
++ * Specifies the SSTable format this CQLSSTableWriter instance should use for writing.
+ *
++ * @param format The format to use
+ * @return this builder
+ */
- public Builder openSSTableOnProduced()
++ public Builder withFormat(SSTableFormat<?, ?> format)
+ {
- this.openSSTableOnProduced = true;
++ this.format = format;
+ return this;
+ }
+
public CQLSSTableWriter build()
{
if (directory == null)
diff --cc test/unit/org/apache/cassandra/io/sstable/CQLSSTableWriterTest.java
index c61a1b50a2,cc41b0ea2a..f2a131b716
--- a/test/unit/org/apache/cassandra/io/sstable/CQLSSTableWriterTest.java
+++ b/test/unit/org/apache/cassandra/io/sstable/CQLSSTableWriterTest.java
@@@ -15,11 -15,13 +15,13 @@@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
package org.apache.cassandra.io.sstable;
+
import java.io.IOException;
import java.nio.ByteBuffer;
+ import java.nio.file.Files;
+ import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
@@@ -49,39 -48,34 +49,45 @@@ import org.junit.Test
import org.junit.rules.TemporaryFolder;
import com.datastax.driver.core.utils.UUIDs;
+ import org.apache.cassandra.Util;
import org.apache.cassandra.cql3.QueryProcessor;
import org.apache.cassandra.cql3.UntypedResultSet;
++import org.apache.cassandra.cql3.constraints.ConstraintViolationException;
import org.apache.cassandra.cql3.functions.types.DataType;
import org.apache.cassandra.cql3.functions.types.LocalDate;
import org.apache.cassandra.cql3.functions.types.TypeCodec;
import org.apache.cassandra.cql3.functions.types.UDTValue;
import org.apache.cassandra.cql3.functions.types.UserType;
-import org.apache.cassandra.db.ColumnFamilyStore;
-import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.marshal.FloatType;
import org.apache.cassandra.db.marshal.UTF8Type;
+ import org.apache.cassandra.dht.ByteOrderedPartitioner;
import org.apache.cassandra.dht.Murmur3Partitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.index.sai.disk.format.IndexDescriptor;
import org.apache.cassandra.index.sai.utils.IndexIdentifier;
+ import org.apache.cassandra.io.sstable.format.SSTableFormat;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.io.sstable.format.big.BigFormat;
+ import org.apache.cassandra.io.sstable.format.bti.BtiFormat;
import org.apache.cassandra.io.util.File;
import org.apache.cassandra.io.util.PathUtils;
+import org.apache.cassandra.locator.RangesAtEndpoint;
+import org.apache.cassandra.schema.KeyspaceMetadata;
import org.apache.cassandra.schema.Schema;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.TableMetadataRef;
+import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.transport.ProtocolVersion;
import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.JavaDriverUtils;
+import org.apache.cassandra.utils.OutputHandler;
+import org.assertj.core.api.Assertions;
import static org.apache.cassandra.utils.Clock.Global.currentTimeMillis;
++import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
@@@ -114,56 -107,82 +120,85 @@@ public abstract class CQLSSTableWriterT
}
@Test
- public void testUnsortedWriter() throws Exception
+ public void testUnsortedWriterBig() throws Exception
{
- String schema = "CREATE TABLE " + qualifiedTable + " ("
- + " k int PRIMARY KEY,"
- + " v1 text,"
- + " v2 int"
- + ")";
- String insert = "INSERT INTO " + qualifiedTable + " (k, v1, v2) VALUES (?, ?, ?)";
- CQLSSTableWriter writer = CQLSSTableWriter.builder()
- .inDirectory(dataDir)
- .forTable(schema)
- .using(insert).build();
+ BigFormat format = BigFormat.getInstance();
+ testWritingSstableWithFormat(format);
+ }
+
+ @Test
+ public void testUnsortedWriterBti() throws Exception
+ {
+ SSTableFormat<?, ?> btiFormat = new BtiFormat.BtiFormatFactory().getInstance(Collections.emptyMap());
+ testWritingSstableWithFormat(btiFormat);
+ }
- writer.addRow(0, "test1", 24);
- writer.addRow(1, "test2", 44);
- writer.addRow(2, "test3", 42);
- writer.addRow(ImmutableMap.<String, Object>of("k", 3, "v2", 12));
+ private void testWritingSstableWithFormat(SSTableFormat<?, ?> format) throws Exception
+ {
+ try (AutoCloseable ignored = Util.switchPartitioner(ByteOrderedPartitioner.instance))
+ {
+ String schema = "CREATE TABLE " + qualifiedTable + " ("
+ + " k int PRIMARY KEY,"
+ + " v1 text,"
+ + " v2 int"
+ + ")";
+ String insert = "INSERT INTO " + qualifiedTable + " (k, v1, v2) VALUES (?, ?, ?)";
+ CQLSSTableWriter writer = CQLSSTableWriter.builder()
+ .inDirectory(dataDir)
+ .forTable(schema)
+ .withFormat(format)
+ .using(insert).build();
+
+ writer.addRow(0, "test1", 24);
+ writer.addRow(1, "test2", 44);
+ writer.addRow(2, "test3", 42);
+ writer.addRow(ImmutableMap.<String, Object>of("k", 3, "v2", 12));
- writer.close();
+ writer.close();
- loadSSTables(dataDir, keyspace, table);
+ validateFilesAreInFormat(format);
- loadSSTables(dataDir, keyspace);
-
- UntypedResultSet rs = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable);
- assertEquals(4, rs.size());
-
- Iterator<UntypedResultSet.Row> iter = rs.iterator();
- UntypedResultSet.Row row;
-
- row = iter.next();
- assertEquals(0, row.getInt("k"));
- assertEquals("test1", row.getString("v1"));
- assertEquals(24, row.getInt("v2"));
++ loadSSTables(dataDir, keyspace, table);
- if (verifyDataAfterLoading)
- row = iter.next();
- assertEquals(1, row.getInt("k"));
- assertEquals("test2", row.getString("v1"));
- assertEquals(44, row.getInt("v2"));
-
- row = iter.next();
- assertEquals(2, row.getInt("k"));
- assertEquals("test3", row.getString("v1"));
- assertEquals(42, row.getInt("v2"));
-
- row = iter.next();
- assertEquals(3, row.getInt("k"));
- assertFalse(row.has("v1"));
- assertEquals(12, row.getInt("v2"));
++ if (verifyDataAfterLoading)
++ {
++ UntypedResultSet rs = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable);
++ assertEquals(4, rs.size());
++
++ Iterator<UntypedResultSet.Row> iter = rs.iterator();
++ UntypedResultSet.Row row;
++
++ row = iter.next();
++ assertEquals(0, row.getInt("k"));
++ assertEquals("test1", row.getString("v1"));
++ assertEquals(24, row.getInt("v2"));
++
++ row = iter.next();
++ assertEquals(1, row.getInt("k"));
++ assertEquals("test2", row.getString("v1"));
++ assertEquals(44, row.getInt("v2"));
++
++ row = iter.next();
++ assertEquals(2, row.getInt("k"));
++ assertEquals("test3", row.getString("v1"));
++ assertEquals(42, row.getInt("v2"));
++
++ row = iter.next();
++ assertEquals(3, row.getInt("k"));
++ assertFalse(row.has("v1"));
++ assertEquals(12, row.getInt("v2"));
++ }
+ }
+ }
+
+ private void validateFilesAreInFormat(SSTableFormat<?, ?> format) throws IOException
+ {
+ try (Stream<Path> dataFilePaths = Files.list(dataDir.toPath()).filter(p -> p.toString().endsWith("Data.db")))
{
- UntypedResultSet rs = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable);
- assertEquals(4, rs.size());
-
- Iterator<UntypedResultSet.Row> iter = rs.iterator();
- UntypedResultSet.Row row;
-
- row = iter.next();
- assertEquals(0, row.getInt("k"));
- assertEquals("test1", row.getString("v1"));
- assertEquals(24, row.getInt("v2"));
-
- row = iter.next();
- assertEquals(1, row.getInt("k"));
- assertEquals("test2", row.getString("v1"));
- //assertFalse(row.has("v2"));
- assertEquals(44, row.getInt("v2"));
-
- row = iter.next();
- assertEquals(2, row.getInt("k"));
- assertEquals("test3", row.getString("v1"));
- assertEquals(42, row.getInt("v2"));
-
- row = iter.next();
- assertEquals(3, row.getInt("k"));
- assertEquals(null, row.getBytes("v1")); // Using getBytes because we know it won't NPE
- assertEquals(12, row.getInt("v2"));
+ dataFilePaths.forEach(dataFilePath -> {
+ File dataFile = new File(dataFilePath.toFile());
+ Descriptor descriptor = Descriptor.fromFile(dataFile);
+ assertEquals(format, descriptor.version.format);
+ });
}
}
@@@ -1578,113 -1520,13 +1613,113 @@@
assertFalse(indexDescriptor.isPerColumnIndexBuildComplete(new IndexIdentifier(keyspace, table, "idx2")));
}
- protected void loadSSTables(File dataDir, String ksName)
+ @Test
+ public void testWritingVectorData() throws Exception
+ {
+ final String schema = "CREATE TABLE " + qualifiedTable + " ("
+ + " k int,"
+ + " v1 VECTOR<FLOAT, 5>,"
+ + " PRIMARY KEY (k)"
+ + ")";
+
+ CQLSSTableWriter writer = CQLSSTableWriter.builder()
+ .inDirectory(dataDir)
+ .forTable(schema)
+ .using("INSERT INTO " + keyspace + "." + table + " (k, v1) " +
+ "VALUES (?, ?)").build();
+
+ for (int i = 0; i < 100; i++)
+ {
+ writer.addRow(i, List.of( (float)i, (float)i, (float)i, (float)i, (float)i));
+ }
+
+ writer.close();
+ loadSSTables(dataDir, keyspace, table);
+
+ if (verifyDataAfterLoading)
+ {
+ UntypedResultSet resultSet = QueryProcessor.executeInternal("SELECT * FROM " + keyspace + "." + table);
+
+ assertEquals(resultSet.size(), 100);
+ int cnt = 0;
+ for (UntypedResultSet.Row row : resultSet)
+ {
+ assertEquals(cnt, row.getInt("k"));
+ List<Float> vector = row.getVector("v1", FloatType.instance, 5);
- Assertions.assertThat(vector).hasSize(5);
++ assertThat(vector).hasSize(5);
+ final float floatCount = (float)cnt;
- Assertions.assertThat(vector).allMatch(val -> val == floatCount);
++ assertThat(vector).allMatch(val -> val == floatCount);
+ cnt++;
+ }
+ }
+ }
+
+ @Test
+ public void testConstraintViolation() throws Exception
{
- ColumnFamilyStore cfs = Keyspace.openWithoutSSTables(ksName).getColumnFamilyStore(table);
- Set<String> dataFilePaths = Set.of(dataDir.absolutePath());
- cfs.importNewSSTables(dataFilePaths, false, false, false,
- false, false, false, false,
- true, false);
+ final String schema = "CREATE TABLE " + qualifiedTable + " ("
+ + " k int,"
+ + " v1 int CHECK v1 < 5 ,"
+ + " PRIMARY KEY (k)"
+ + ")";
+
+ CQLSSTableWriter writer = CQLSSTableWriter.builder()
+ .inDirectory(dataDir)
+ .forTable(schema)
+ .using("INSERT INTO " + keyspace + "." + table + " (k, v1) " +
+ "VALUES (?, ?)").build();
+
+ writer.addRow(1, 4);
+
+ Assertions.assertThatThrownBy(() -> writer.addRow(2, 11))
+ .describedAs("Should throw when adding a row that violates constraints")
+ .isInstanceOf(ConstraintViolationException.class)
+ .hasMessageContaining("Column value does not satisfy value constraint for column 'v1'. It should be v1 < 5");
+
+ writer.close();
+ loadSSTables(dataDir, keyspace, table);
+
+ if (verifyDataAfterLoading)
+ {
+ UntypedResultSet resultSet = QueryProcessor.executeInternal("SELECT * FROM " + keyspace + "." + table);
+
+ assertEquals(resultSet.size(), 1);
+ UntypedResultSet.Row row = resultSet.one();
+ assertEquals(1, row.getInt("k"));
+ assertEquals(4, row.getInt("v1"));
+ }
+ }
+
+ protected static void loadSSTables(File dataDir, final String ks, final String tb) throws ExecutionException, InterruptedException
+ {
+ SSTableLoader loader = new SSTableLoader(dataDir, new SSTableLoader.Client()
+ {
+ private String keyspace;
+
+ @Override
+ public void init(String keyspace)
+ {
+ this.keyspace = keyspace;
+
+ KeyspaceMetadata keyspaceMetadata = Schema.instance.getKeyspaceMetadata(keyspace);
+
+ RangesAtEndpoint addressReplicas = keyspaceMetadata.replicationStrategy.getAddressReplicas(ClusterMetadata.current(), FBUtilities.getBroadcastAddressAndPort());
+
+ for (Range<Token> range : addressReplicas.ranges())
+ addRangeForEndpoint(range, FBUtilities.getBroadcastAddressAndPort());
+ }
+
+ @Override
+ public TableMetadataRef getTableMetadata(String tableName)
+ {
+ KeyspaceMetadata keyspaceMetadata = ClusterMetadata.current().schema.getKeyspaceMetadata(keyspace);
+ TableMetadata tableMetadata = keyspaceMetadata.tables.getNullable(tableName);
+ assert tableMetadata != null;
+ return tableMetadata.ref;
+ }
+ }, new OutputHandler.SystemOutput(false, false), 1, ks, tb);
+
+ loader.stream().get();
}
private class WriterThread extends Thread
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org