You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by ad...@apache.org on 2022/05/20 10:59:24 UTC
[cassandra] branch cassandra-4.1 updated (c635f198f6 -> c1d163abad)
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a change to branch cassandra-4.1
in repository https://gitbox.apache.org/repos/asf/cassandra.git
from c635f198f6 Fix leak of non-standard Java types in our Exceptions as clients using JMX are unable to handle them Remove useless validation that leads to unnecessary additional read of cassandra.yaml on startup patch by Ekaterina Dimitrova; review by David Capwell for CASSANDRA-17638
new ffc4c89c3d Fix issue where frozen maps may not be serialized in the correct order
new 0e1d068d92 Merge branch 'cassandra-3.0' into cassandra-3.11
new 5cf6db30f6 Merge branch 'cassandra-3.11' into cassandra-4.0
new c1d163abad Merge branch 'cassandra-4.0' into cassandra-4.1
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.txt | 1 +
src/java/org/apache/cassandra/cql3/Maps.java | 11 +-
src/java/org/apache/cassandra/cql3/Sets.java | 2 +-
test/unit/org/apache/cassandra/cql3/CQLTester.java | 34 +++-
.../cql3/conditions/ColumnConditionTest.java | 6 +-
.../cql3/validation/entities/CollectionsTest.java | 99 ++++++++++++
.../cassandra/io/sstable/CQLSSTableWriterTest.java | 171 ++++++++++++++++++++-
7 files changed, 308 insertions(+), 16 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-4.0' into cassandra-4.1
Posted by ad...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-4.1
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit c1d163abad7279c7d8bb4e76bb39de47c90bbec7
Merge: c635f198f6 5cf6db30f6
Author: Andrés de la Peña <a....@gmail.com>
AuthorDate: Fri May 20 11:52:34 2022 +0100
Merge branch 'cassandra-4.0' into cassandra-4.1
CHANGES.txt | 1 +
src/java/org/apache/cassandra/cql3/Maps.java | 11 +-
src/java/org/apache/cassandra/cql3/Sets.java | 2 +-
test/unit/org/apache/cassandra/cql3/CQLTester.java | 34 +++-
.../cql3/conditions/ColumnConditionTest.java | 6 +-
.../cql3/validation/entities/CollectionsTest.java | 99 ++++++++++++
.../cassandra/io/sstable/CQLSSTableWriterTest.java | 171 ++++++++++++++++++++-
7 files changed, 308 insertions(+), 16 deletions(-)
diff --cc CHANGES.txt
index a9623ab152,6c31e17981..f8a438c8d2
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,174 -1,7 +1,175 @@@
-4.0.5
+4.1
+ * Fix leak of non-standard Java types in our Exceptions as clients using JMX are unable to handle them.
+ Remove useless validation that leads to unnecessary additional read of cassandra.yaml on startup (CASSANDRA-17638)
+ * Fix repair_request_timeout_in_ms and remove paxos_auto_repair_threshold_mb (CASSANDRA-17557)
+ * Incremental repair leaks SomeRepairFailedException after switch away from flatMap (CASSANDRA-17620)
+ * StorageService read threshold get methods throw NullPointerException due to not handling null configs (CASSANDRA-17593)
+ * Rename truncate_drop guardrail to drop_truncate_table (CASSANDRA-17592)
+ * nodetool enablefullquerylog can NPE when directory has no files (CASSANDRA-17595)
+ * Add auto_snapshot_ttl configuration (CASSANDRA-16790)
+ * List snapshots of dropped tables (CASSANDRA-16843)
+ * Add information whether sstables are dropped to SchemaChangeListener (CASSANDRA-17582)
+ * Add a pluggable memtable API (CEP-11 / CASSANDRA-17034)
+ * Save sstable id as string in activity table (CASSANDRA-17585)
+ * Implement startup check to prevent Cassandra to potentially spread zombie data (CASSANDRA-17180)
+ * Allow failing startup on duplicate config keys (CASSANDRA-17379)
+ * Migrate threshold for minimum keyspace replication factor to guardrails (CASSANDRA-17212)
+ * Add guardrail to disallow TRUNCATE and DROP TABLE commands (CASSANDRA-17558)
+ * Add plugin support for CQLSH (CASSANDRA-16456)
+ * Add guardrail to disallow querying with ALLOW FILTERING (CASSANDRA-17370)
+ * Enhance SnakeYAML properties to be reusable outside of YAML parsing, support camel case conversion to snake case, and add support to ignore properties (CASSANDRA-17166)
+ * nodetool compact should support using a key string to find the range to avoid operators having to manually do this (CASSANDRA-17537)
+ * Add guardrail for data disk usage (CASSANDRA-17150)
+ * Tool to list data paths of existing tables (CASSANDRA-17568)
+ * Migrate track_warnings to more standard naming conventions and use latest configuration types rather than long (CASSANDRA-17560)
+ * Add support for CONTAINS and CONTAINS KEY in conditional UPDATE and DELETE statement (CASSANDRA-10537)
+ * Migrate advanced config parameters to the new Config types (CASSANDRA-17431)
+ * Make null to be meaning disabled and leave 0 as a valid value for permissions_update_interval, roles_update_interval, credentials_update_interval (CASSANDRA-17431)
+ * Fix typo in Config annotation (CASSANDRA-17431)
+ * Made Converters type safe and fixed a few cases where converters used the wrong type (CASSANDRA-17431)
+ * Fix null bug in DataStorageSpec and DurationSpec and require units to be added when providing 0 value (CASSANDRA-17431)
+ * Shutdown ScheduledExecutors as part of node drainage (CASSANDRA-17493)
+ * Provide JMX endpoint to allow transient logging of blocking read repairs (CASSANDRA-17471)
+ * Add guardrail for GROUP BY queries (CASSANDRA-17509)
+ * make pylib PEP and pylint compliant (CASSANDRA-17546)
+ * Add support for vnodes in jvm-dtest (CASSANDRA-17332)
+ * Remove guardrails global enable flag (CASSANDRA-17499)
+ * Clients using JMX are unable to handle non-standard java types but we leak this into our interfaces (CASSANDRA-17527)
+ * Remove stress server functionality (CASSANDRA-17535)
+ * Reduce histogram snapshot long[] allocation overhead during speculative read and write threshold updates (CASSANDRA-17523)
+ * Add guardrail for creation of secondary indexes (CASSANDRA-17498)
+ * Add guardrail to disallow creation of uncompressed tables (CASSANDRA-17504)
+ * Add guardrail to disallow creation of new COMPACT STORAGE tables (CASSANDRA-17522)
+ * repair vtables should expose a completed field due to lack of filtering options in CQL (CASSANDRA-17520)
+ * remove outdated code from cqlsh (CASSANDRA-17490)
+ * remove support for deprecated version specific TLS in Python 3.6 (CASSANDRA-17365)
+ * Add support for IF EXISTS and IF NOT EXISTS in ALTER statements (CASSANDRA-16916)
+ * resolve several pylint issues in cqlsh.py and pylib (CASSANDRA-17480)
+ * Streaming sessions longer than 3 minutes fail with timeout (CASSANDRA-17510)
+ * Add ability to track state in repair (CASSANDRA-15399)
+ * Remove unused 'parse' module (CASSANDRA-17484)
+ * change six functions in cqlshlib to native Python 3 (CASSANDRA-17417)
+ * reduce hot-path object allocations required to record local/remote requests against the client request metrics (CASSANDRA-17424)
+ * Disallow removing DC from system_auth while nodes are active in the DC (CASSANDRA-17478)
+ * Add guardrail for the number of fields per UDT (CASSANDRA-17385)
+ * Allow users to change cqlsh history location using env variable (CASSANDRA-17448)
+ * Add required -f option to use nodetool verify and standalone sstableverify (CASSANDRA-17017)
+ * Add support for UUID based sstable generation identifiers (CASSANDRA-17048)
+ * Log largest memtable flush at info instead of debug (CASSANDRA-17472)
+ * Add native transport rate limiter options to example cassandra.yaml, and expose metric for dispatch rate (CASSANDRA-17423)
+ * Add diagnostic events for guardrails (CASSANDRA-17197)
+ * Pre hashed passwords in CQL (CASSANDRA-17334)
+ * Increase cqlsh version (CASSANDRA-17432)
+ * Update SUPPORTED_UPGRADE_PATHS to include 3.0 and 3.x to 4.1 paths and remove obsolete tests (CASSANDRA-17362)
+ * Support DELETE in CQLSSTableWriter (CASSANDRA-14797)
+ * Failed inbound internode authentication failures generate ugly warning with stack trace (CASSANDRA-17068)
+ * Expose gossip information in system_views.gossip_info virtual table (CASSANDRA-17002)
+ * Add guardrails for collection items and size (CASSANDRA-17153)
+ * Improve guardrails messages (CASSANDRA-17430)
+ * Remove all usages of junit.framework and ban them via Checkstyle (CASSANDRA-17316)
+ * Add guardrails for read/write consistency levels (CASSANDRA-17188)
+ * Add guardrail for SELECT IN terms and their cartesian product (CASSANDRA-17187)
+ * remove unused imports in cqlsh.py and cqlshlib (CASSANDRA-17413)
+ * deprecate property windows_timer_interval (CASSANDRA-17404)
+ * Expose streaming as a vtable (CASSANDRA-17390)
+ * Expose all client options via system_views.clients and nodetool clientstats (CASSANDRA-16378)
+ * Make startup checks configurable (CASSANDRA-17220)
+ * Add guardrail for number of partition keys on IN queries (CASSANDRA-17186)
+ * update Python test framework from nose to pytest (CASSANDRA-17293)
+ * Fix improper CDC commit log segments deletion in non-blocking mode (CASSANDRA-17233)
+ * Add support for string concatenations through the + operator (CASSANDRA-17190)
+ * Limit the maximum hints size per host (CASSANDRA-17142)
+ * Add a virtual table for exposing batch metrics (CASSANDRA-17225)
+ * Flatten guardrails config (CASSANDRA-17353)
+ * Instance failed to start up due to NPE in StartupClusterConnectivityChecker (CASSANDRA-17347)
+ * add the shorter version of version flag (-v) in cqlsh (CASSANDRA-17236)
+ * Make vtables accessible via internode messaging (CASSANDRA-17295)
+ * Add support for PEM based key material for SSL (CASSANDRA-17031)
+ * Standardize storage configuration parameters' names. Support unit suffixes. (CASSANDRA-15234)
+ * Remove support for Windows (CASSANDRA-16956)
+ * Runtime-configurable YAML option to prohibit USE statements (CASSANDRA-17318)
+ * When streaming sees a ClosedChannelException this triggers the disk failure policy (CASSANDRA-17116)
+ * Add a virtual table for exposing prepared statements metrics (CASSANDRA-17224)
+ * Remove python 2.x support from cqlsh (CASSANDRA-17242)
+ * Prewarm role and credential caches to avoid timeouts at startup (CASSANDRA-16958)
+ * Make capacity/validity/updateinterval/activeupdate for Auth Caches configurable via nodetool (CASSANDRA-17063)
+ * Added startup check for read_ahead_kb setting (CASSANDRA-16436)
+ * Avoid unecessary array allocations and initializations when performing query checks (CASSANDRA-17209)
+ * Add guardrail for list operations that require read before write (CASSANDRA-17154)
+ * Migrate thresholds for number of keyspaces and tables to guardrails (CASSANDRA-17195)
+ * Remove self-reference in SSTableTidier (CASSANDRA-17205)
+ * Add guardrail for query page size (CASSANDRA-17189)
+ * Allow column_index_size_in_kb to be configurable through nodetool (CASSANDRA-17121)
+ * Emit a metric for number of local read and write calls
+ * Add non-blocking mode for CDC writes (CASSANDRA-17001)
+ * Add guardrails framework (CASSANDRA-17147)
+ * Harden resource management on SSTable components to prevent future leaks (CASSANDRA-17174)
+ * Make nodes more resilient to local unrelated files during startup (CASSANDRA-17082)
+ * repair prepare message would produce a wrong error message if network timeout happened rather than reply wait timeout (CASSANDRA-16992)
+ * Log queries that fail on timeout or unavailable errors up to once per minute by default (CASSANDRA-17159)
+ * Refactor normal/preview/IR repair to standardize repair cleanup and error handling of failed RepairJobs (CASSANDRA-17069)
+ * Log missing peers in StartupClusterConnectivityChecker (CASSANDRA-17130)
+ * Introduce separate rate limiting settings for entire SSTable streaming (CASSANDRA-17065)
+ * Implement Virtual Tables for Auth Caches (CASSANDRA-16914)
+ * Actively update auth cache in the background (CASSANDRA-16957)
+ * Add unix time conversion functions (CASSANDRA-17029)
+ * JVMStabilityInspector.forceHeapSpaceOomMaybe should handle all non-heap OOMs rather than only supporting direct only (CASSANDRA-17128)
+ * Forbid other Future implementations with checkstyle (CASSANDRA-17055)
+ * commit log was switched from non-daemon to daemon threads, which causes the JVM to exit in some case as no non-daemon threads are active (CASSANDRA-17085)
+ * Add a Denylist to block reads and writes on specific partition keys (CASSANDRA-12106)
+ * v4+ protocol did not clean up client warnings, which caused leaking the state (CASSANDRA-17054)
+ * Remove duplicate toCQLString in ReadCommand (CASSANDRA-17023)
+ * Ensure hint window is persistent across restarts of a node (CASSANDRA-14309)
+ * Allow to GRANT or REVOKE multiple permissions in a single statement (CASSANDRA-17030)
+ * Allow to grant permission for all tables in a keyspace (CASSANDRA-17027)
+ * Log time spent writing keys during compaction (CASSANDRA-17037)
+ * Make nodetool compactionstats and sstable_tasks consistent (CASSANDRA-16976)
+ * Add metrics and logging around index summary redistribution (CASSANDRA-17036)
+ * Add configuration options for minimum allowable replication factor and default replication factor (CASSANDRA-14557)
+ * Expose information about stored hints via a nodetool command and a virtual table (CASSANDRA-14795)
+ * Add broadcast_rpc_address to system.local (CASSANDRA-11181)
+ * Add support for type casting in WHERE clause components and in the values of INSERT/UPDATE statements (CASSANDRA-14337)
+ * add credentials file support to CQLSH (CASSANDRA-16983)
+ * Skip remaining bytes in the Envelope buffer when a ProtocolException is thrown to avoid double decoding (CASSANDRA-17026)
+ * Allow reverse iteration of resources during permissions checking (CASSANDRA-17016)
+ * Add feature to verify correct ownership of attached locations on disk at startup (CASSANDRA-16879)
+ * Make SSLContext creation pluggable/extensible (CASSANDRA-16666)
+ * Add soft/hard limits to local reads to protect against reading too much data in a single query (CASSANDRA-16896)
+ * Avoid token cache invalidation for removing a non-member node (CASSANDRA-15290)
+ * Allow configuration of consistency levels on auth operations (CASSANDRA-12988)
+ * Add number of sstables in a compaction to compactionstats output (CASSANDRA-16844)
+ * Upgrade Caffeine to 2.9.2 (CASSANDRA-15153)
+ * Allow DELETE and TRUNCATE to work on Virtual Tables if the implementation allows it (CASSANDRA-16806)
+ * Include SASI components to snapshots (CASSANDRA-15134)
+ * Fix missed wait latencies in the output of `nodetool tpstats -F` (CASSANDRA-16938)
+ * Reduce native transport max frame size to 16MB (CASSANDRA-16886)
+ * Add support for filtering using IN restrictions (CASSANDRA-14344)
+ * Provide a nodetool command to invalidate auth caches (CASSANDRA-16404)
+ * Catch read repair timeout exceptions and add metric (CASSANDRA-16880)
+ * Exclude Jackson 1.x transitive dependency of hadoop* provided dependencies (CASSANDRA-16854)
+ * Add client warnings and abort to tombstone and coordinator reads which go past a low/high watermark (CASSANDRA-16850)
+ * Add TTL support to nodetool snapshots (CASSANDRA-16789)
+ * Allow CommitLogSegmentReader to optionally skip sync marker CRC checks (CASSANDRA-16842)
+ * allow blocking IPs from updating metrics about traffic (CASSANDRA-16859)
+ * Request-Based Native Transport Rate-Limiting (CASSANDRA-16663)
+ * Implement nodetool getauditlog command (CASSANDRA-16725)
+ * Clean up repair code (CASSANDRA-13720)
+ * Background schedule to clean up orphaned hints files (CASSANDRA-16815)
+ * Modify SecondaryIndexManager#indexPartition() to retrieve only columns for which indexes are actually being built (CASSANDRA-16776)
+ * Batch the token metadata update to improve the speed (CASSANDRA-15291)
+ * Reduce the log level on "expected" repair exceptions (CASSANDRA-16775)
+ * Make JMXTimer expose attributes using consistent time unit (CASSANDRA-16760)
+ * Remove check on gossip status from DynamicEndpointSnitch::updateScores (CASSANDRA-11671)
+ * Fix AbstractReadQuery::toCQLString not returning valid CQL (CASSANDRA-16510)
+ * Log when compacting many tombstones (CASSANDRA-16780)
+ * Display bytes per level in tablestats for LCS tables (CASSANDRA-16799)
+ * Add isolated flush timer to CommitLogMetrics and ensure writes correspond to single WaitingOnCommit data points (CASSANDRA-16701)
+ * GossiperTest.testHasVersion3Nodes didn't take into account trunk version changes, fixed to rely on latest version (CASSANDRA-16651)
+ * Update JNA library to 5.9.0 and snappy-java to version 1.1.8.4 (CASSANDRA-17040)
+Merged from 4.0:
Merged from 3.11:
Merged from 3.0:
+ * Fix issue where frozen maps may not be serialized in the correct order (CASSANDRA-17623)
* Suppress CVE-2022-24823 (CASSANDRA-17633)
* fsync TOC and digest files (CASSANDRA-10709)
diff --cc src/java/org/apache/cassandra/cql3/Maps.java
index 8bb7ae192c,3a9575c833..a2d23a699a
--- a/src/java/org/apache/cassandra/cql3/Maps.java
+++ b/src/java/org/apache/cassandra/cql3/Maps.java
@@@ -422,34 -422,14 +423,34 @@@ public abstract class Map
static void doPut(Term.Terminal value, ColumnMetadata column, UpdateParameters params) throws InvalidRequestException
{
+ if (value == null)
+ {
+ // for frozen maps, we're overwriting the whole cell
+ if (!column.type.isMultiCell())
+ params.addTombstone(column);
+
+ return;
+ }
+
- Map<ByteBuffer, ByteBuffer> elements = ((Value) value).map;
++ SortedMap<ByteBuffer, ByteBuffer> elements = ((Value) value).map;
+
if (column.type.isMultiCell())
{
- if (value == null)
+ if (elements.size() == 0)
return;
- SortedMap<ByteBuffer, ByteBuffer> elements = ((Value) value).map;
+ // Guardrails about collection size are only checked for the added elements without considering
+ // already existent elements. This is done so to avoid read-before-write, having additional checks
+ // during SSTable write.
+ Guardrails.itemsPerCollection.guard(elements.size(), column.name.toString(), false, params.clientState);
+
+ int dataSize = 0;
for (Map.Entry<ByteBuffer, ByteBuffer> entry : elements.entrySet())
- params.addCell(column, CellPath.create(entry.getKey()), entry.getValue());
+ {
+ Cell<?> cell = params.addCell(column, CellPath.create(entry.getKey()), entry.getValue());
+ dataSize += cell.dataSize();
+ }
+ Guardrails.collectionSize.guard(dataSize, column.name.toString(), false, params.clientState);
}
else
{
diff --cc test/unit/org/apache/cassandra/cql3/CQLTester.java
index d8a23fdad8,a8ca1e5d24..f5e93e8985
--- a/test/unit/org/apache/cassandra/cql3/CQLTester.java
+++ b/test/unit/org/apache/cassandra/cql3/CQLTester.java
@@@ -45,9 -46,9 +45,10 @@@ import javax.management.remote.rmi.RMIC
import com.google.common.base.Objects;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableSet;
+ import com.google.common.collect.Iterables;
import org.junit.*;
+
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@@ -1164,41 -943,6 +1165,41 @@@ public abstract class CQLTeste
Assert.assertEquals(expectedArgTypes != null ? Arrays.asList(expectedArgTypes) : null, schemaChange.argTypes);
}
+ protected static void assertWarningsContain(Message.Response response, String message)
+ {
+ assertWarningsContain(response.getWarnings(), message);
+ }
+
+ protected static void assertWarningsContain(List<String> warnings, String message)
+ {
+ Assert.assertNotNull(warnings);
+ assertTrue(warnings.stream().anyMatch(s -> s.contains(message)));
+ }
-
++
+ protected static void assertWarningsEquals(ResultSet rs, String... messages)
+ {
+ assertWarningsEquals(rs.getExecutionInfo().getWarnings(), messages);
+ }
+
+ protected static void assertWarningsEquals(List<String> warnings, String... messages)
+ {
+ Assert.assertNotNull(warnings);
+ Assertions.assertThat(messages).hasSameElementsAs(warnings);
+ }
+
+ protected static void assertNoWarningContains(Message.Response response, String message)
+ {
+ assertNoWarningContains(response.getWarnings(), message);
+ }
+
+ protected static void assertNoWarningContains(List<String> warnings, String message)
+ {
- if (warnings != null)
++ if (warnings != null)
+ {
+ assertFalse(warnings.stream().anyMatch(s -> s.contains(message)));
+ }
+ }
+
protected static ResultMessage schemaChange(String query)
{
try
diff --cc test/unit/org/apache/cassandra/cql3/conditions/ColumnConditionTest.java
index a896125430,f62119d4e6..dd40cf81db
--- a/test/unit/org/apache/cassandra/cql3/conditions/ColumnConditionTest.java
+++ b/test/unit/org/apache/cassandra/cql3/conditions/ColumnConditionTest.java
@@@ -159,15 -144,7 +159,15 @@@ public class ColumnConditionTes
return bound.appliesTo(newRow(definition, rowValue));
}
- private static boolean conditionApplies(Map<ByteBuffer, ByteBuffer> rowValue, Operator op, SortedMap<ByteBuffer, ByteBuffer> conditionValue)
+ private static boolean conditionContainsApplies(SortedSet<ByteBuffer> rowValue, Operator op, ByteBuffer conditionValue)
+ {
+ ColumnMetadata definition = ColumnMetadata.regularColumn("ks", "cf", "c", SetType.getInstance(Int32Type.instance, true));
+ ColumnCondition condition = ColumnCondition.condition(definition, op, Terms.of(new Constants.Value(conditionValue)));
+ ColumnCondition.Bound bound = condition.bind(QueryOptions.DEFAULT);
+ return bound.appliesTo(newRow(definition, rowValue));
+ }
+
- private static boolean conditionApplies(Map<ByteBuffer, ByteBuffer> rowValue, Operator op, Map<ByteBuffer, ByteBuffer> conditionValue)
++ private static boolean conditionApplies(SortedMap<ByteBuffer, ByteBuffer> rowValue, Operator op, SortedMap<ByteBuffer, ByteBuffer> conditionValue)
{
ColumnMetadata definition = ColumnMetadata.regularColumn("ks", "cf", "c", MapType.getInstance(Int32Type.instance, Int32Type.instance, true));
ColumnCondition condition = ColumnCondition.condition(definition, op, Terms.of(new Maps.Value(conditionValue)));
diff --cc test/unit/org/apache/cassandra/io/sstable/CQLSSTableWriterTest.java
index ab3edc832c,f43294c8ec..f5546b0714
--- a/test/unit/org/apache/cassandra/io/sstable/CQLSSTableWriterTest.java
+++ b/test/unit/org/apache/cassandra/io/sstable/CQLSSTableWriterTest.java
@@@ -45,6 -47,8 +46,7 @@@ import org.apache.cassandra.cql3.functi
import org.apache.cassandra.cql3.functions.types.*;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.commitlog.CommitLog;
-import org.apache.cassandra.db.rows.Row;
+ import org.apache.cassandra.db.marshal.UTF8Type;
import org.apache.cassandra.dht.*;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.schema.Schema;
@@@ -947,6 -680,172 +950,172 @@@ public class CQLSSTableWriterTes
}
}
+ @Test
+ public void testFrozenMapType() throws Exception
+ {
+ // Test to make sure we can write to `date` fields in both old and new formats
+ String schema = "CREATE TABLE " + qualifiedTable + " ("
+ + " k text,"
+ + " c frozen<map<text, text>>,"
+ + " PRIMARY KEY (k, c)"
+ + ")";
+ String insert = "INSERT INTO " + qualifiedTable + " (k, c) VALUES (?, ?)";
+ CQLSSTableWriter writer = CQLSSTableWriter.builder()
+ .inDirectory(dataDir)
+ .forTable(schema)
+ .using(insert)
- .withBufferSizeInMB(1)
++ .withBufferSizeInMiB(1)
+ .build();
+ for (int i = 0; i < 100; i++)
+ {
+ LinkedHashMap<String, String> map = new LinkedHashMap<>();
+ map.put("a_key", "av" + i);
+ map.put("b_key", "zv" + i);
+ writer.addRow(String.valueOf(i), map);
+ }
+ for (int i = 100; i < 200; i++)
+ {
+ LinkedHashMap<String, String> map = new LinkedHashMap<>();
+ map.put("b_key", "zv" + i);
+ map.put("a_key", "av" + i);
+ writer.addRow(String.valueOf(i), map);
+ }
+ writer.close();
+ loadSSTables(dataDir, keyspace);
+
+ UntypedResultSet rs = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + ";");
+ assertEquals(200, rs.size());
+ Map<String, Map<String, String>> map = StreamSupport.stream(rs.spliterator(), false)
+ .collect(Collectors.toMap(r -> r.getString("k"), r -> r.getFrozenMap("c", UTF8Type.instance, UTF8Type.instance)));
+ for (int i = 0; i < 200; i++)
+ {
+ final String expectedKey = String.valueOf(i);
+ assertTrue(map.containsKey(expectedKey));
+ Map<String, String> innerMap = map.get(expectedKey);
+ assertTrue(innerMap.containsKey("a_key"));
+ assertEquals(innerMap.get("a_key"), "av" + i);
+ assertTrue(innerMap.containsKey("b_key"));
+ assertEquals(innerMap.get("b_key"), "zv" + i);
+ }
+
+ // Make sure we can filter with map values regardless of which order we put the keys in
+ UntypedResultSet filtered;
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='0' and c={'a_key': 'av0', 'b_key': 'zv0'};");
+ assertEquals(1, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='0' and c={'b_key': 'zv0', 'a_key': 'av0'};");
+ assertEquals(1, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='100' and c={'b_key': 'zv100', 'a_key': 'av100'};");
+ assertEquals(1, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='100' and c={'a_key': 'av100', 'b_key': 'zv100'};");
+ assertEquals(1, filtered.size());
+ }
+
+ @Test
+ public void testFrozenMapTypeCustomOrdered() throws Exception
+ {
+ // Test to make sure we can write to `date` fields in both old and new formats
+ String schema = "CREATE TABLE " + qualifiedTable + " ("
+ + " k text,"
+ + " c frozen<map<timeuuid, int>>,"
+ + " PRIMARY KEY (k, c)"
+ + ")";
+ String insert = "INSERT INTO " + qualifiedTable + " (k, c) VALUES (?, ?)";
+ CQLSSTableWriter writer = CQLSSTableWriter.builder()
+ .inDirectory(dataDir)
+ .forTable(schema)
+ .using(insert)
- .withBufferSizeInMB(1)
++ .withBufferSizeInMiB(1)
+ .build();
+ UUID uuid1 = UUIDs.timeBased();
+ UUID uuid2 = UUIDs.timeBased();
+ UUID uuid3 = UUIDs.timeBased();
+ UUID uuid4 = UUIDs.timeBased();
+ Map<UUID, Integer> map = new LinkedHashMap<>();
+ // NOTE: if these two `put` calls are switched, the test passes
+ map.put(uuid2, 2);
+ map.put(uuid1, 1);
+ writer.addRow(String.valueOf(1), map);
+
+ Map<UUID, Integer> map2 = new LinkedHashMap<>();
+ map2.put(uuid3, 1);
+ map2.put(uuid4, 2);
+ writer.addRow(String.valueOf(2), map2);
+
+ writer.close();
+ loadSSTables(dataDir, keyspace);
+
+ UntypedResultSet rs = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + ";");
+ assertEquals(2, rs.size());
+
+ // Make sure we can filter with map values regardless of which order we put the keys in
+ UntypedResultSet filtered;
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='1' and c={" + uuid1 + ": 1, " + uuid2 + ": 2};");
+ assertEquals(1, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='1' and c={" + uuid2 + ": 2, " + uuid1 + ": 1};");
+ assertEquals(1, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='2' and c={" + uuid3 + ": 1, " + uuid4 + ": 2};");
+ assertEquals(1, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='2' and c={" + uuid4 + ": 2, " + uuid3 + ": 1};");
+ assertEquals(1, filtered.size());
+ UUID other = UUIDs.startOf(1234L); // Just some other TimeUUID
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='2' and c={" + uuid3 + ": 1, " + other + ": 2};");
+ assertEquals(0, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='2' and c={" + uuid4 + ": 2, " + other + ": 1};");
+ assertEquals(0, filtered.size());
+ }
+
+ @Test
+ public void testFrozenSetTypeCustomOrdered() throws Exception
+ {
+ // Test to make sure we can write to `date` fields in both old and new formats
+ String schema = "CREATE TABLE " + qualifiedTable + " ("
+ + " k text,"
+ + " c frozen<set<timeuuid>>,"
+ + " PRIMARY KEY (k, c)"
+ + ")";
+ String insert = "INSERT INTO " + qualifiedTable + " (k, c) VALUES (?, ?)";
+ CQLSSTableWriter writer = CQLSSTableWriter.builder()
+ .inDirectory(dataDir)
+ .forTable(schema)
+ .using(insert)
- .withBufferSizeInMB(1)
++ .withBufferSizeInMiB(1)
+ .build();
+ UUID uuid1 = UUIDs.startOf(0L);
+ UUID uuid2 = UUIDs.startOf(10000000L);
+
+ LinkedHashSet<UUID> set = new LinkedHashSet<>();
+ set.add(uuid1);
+ set.add(uuid2);
+ writer.addRow(String.valueOf(1), set);
+
+ LinkedHashSet<UUID> set2 = new LinkedHashSet<>();
+ set2.add(uuid2);
+ set2.add(uuid1);
+ writer.addRow(String.valueOf(2), set2);
+
+ writer.close();
+ loadSSTables(dataDir, keyspace);
+
+ UntypedResultSet rs = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + ";");
+ assertEquals(2, rs.size());
+
+ // Make sure we can filter with map values regardless of which order we put the keys in
+ UntypedResultSet filtered;
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='1' and c={" + uuid1 + ", " + uuid2 + "};");
+ assertEquals(1, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='1' and c={" + uuid2 + ", " + uuid1 + "};");
+ assertEquals(1, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='2' and c={" + uuid1 + ", " + uuid2 + "};");
+ assertEquals(1, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='2' and c={" + uuid2 + ", " + uuid1 + "};");
+ assertEquals(1, filtered.size());
+ UUID other = UUIDs.startOf(10000000L + 1L); // Pick one that's really close just to make sure clustering filters are working
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='2' and c={" + uuid1 + ", " + other + "};");
+ assertEquals(0, filtered.size());
+ filtered = QueryProcessor.executeInternal("SELECT * FROM " + qualifiedTable + " where k='2' and c={" + other + ", " + uuid1 + "};");
+ assertEquals(0, filtered.size());
+ }
+
private static void loadSSTables(File dataDir, String ks) throws ExecutionException, InterruptedException
{
SSTableLoader loader = new SSTableLoader(dataDir, new SSTableLoader.Client()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org