You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by ma...@apache.org on 2025/02/12 23:38:11 UTC
(cassandra) branch trunk updated (0e41c80c81 -> e0c7d7b938)
This is an automated email from the ASF dual-hosted git repository.
maedhroz pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
from 0e41c80c81 Merge branch 'cassandra-5.0' into trunk
new bd99331bf2 Avoid fetching entire partitions on unresolved static rows in RFP when no static column predicates exist
new e0c7d7b938 Merge branch 'cassandra-5.0' into trunk
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.txt | 1 +
.../org/apache/cassandra/db/filter/RowFilter.java | 11 ++++-
.../cassandra/index/sai/plan/QueryController.java | 6 ++-
.../cassandra/service/reads/DataResolver.java | 2 +-
.../service/reads/ReplicaFilteringProtection.java | 10 ++--
.../test/ReplicaFilteringProtectionTest.java | 19 ++++++++
.../test/cql3/SingleNodeTableWalkTest.java | 53 +++++-----------------
.../distributed/test/sai/StrictFilteringTest.java | 23 +++++++++-
.../cassandra/fuzz/sai/SingleNodeSAITestBase.java | 16 ++++---
.../unit/org/apache/cassandra/cql3/KnownIssue.java | 6 +--
10 files changed, 86 insertions(+), 61 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
(cassandra) 01/01: Merge branch 'cassandra-5.0' into trunk
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
maedhroz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit e0c7d7b9389fedf9b3f982ede2a93a9710f7fe52
Merge: 0e41c80c81 bd99331bf2
Author: Caleb Rackliffe <ca...@gmail.com>
AuthorDate: Wed Feb 12 17:32:50 2025 -0600
Merge branch 'cassandra-5.0' into trunk
* cassandra-5.0:
Avoid fetching entire partitions on unresolved static rows in RFP when no static column predicates exist
CHANGES.txt | 1 +
.../org/apache/cassandra/db/filter/RowFilter.java | 11 ++++-
.../cassandra/index/sai/plan/QueryController.java | 6 ++-
.../cassandra/service/reads/DataResolver.java | 2 +-
.../service/reads/ReplicaFilteringProtection.java | 10 ++--
.../test/ReplicaFilteringProtectionTest.java | 19 ++++++++
.../test/cql3/SingleNodeTableWalkTest.java | 53 +++++-----------------
.../distributed/test/sai/StrictFilteringTest.java | 23 +++++++++-
.../cassandra/fuzz/sai/SingleNodeSAITestBase.java | 16 ++++---
.../unit/org/apache/cassandra/cql3/KnownIssue.java | 6 +--
10 files changed, 86 insertions(+), 61 deletions(-)
diff --cc CHANGES.txt
index 49cce9e160,f659a1f220..aba2b957f7
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,135 -1,5 +1,136 @@@
-5.0.4
+5.1
+ * Improve error messages for constraints (CASSANDRA-20266)
+ * Add system_views.partition_key_statistics for querying SSTable metadata (CASSANDRA-20161)
+ * CEP-42 - Add Constraints Framework (CASSANDRA-19947)
+ * Add table metric PurgeableTombstoneScannedHistogram and a tracing event for scanned purgeable tombstones (CASSANDRA-20132)
+ * Make sure we can parse the expanded CQL before writing it to the log or sending it to replicas (CASSANDRA-20218)
+ * Add format_bytes and format_time functions (CASSANDRA-19546)
+ * Fix error when trying to assign a tuple to target type not being a tuple (CASSANDRA-20237)
+ * Fail CREATE TABLE LIKE statement if UDTs in target keyspace do not exist or they have different structure from ones in source keyspace (CASSANDRA-19966)
+ * Support octet_length and length functions (CASSANDRA-20102)
+ * Make JsonUtils serialize Instant always with the same format (CASSANDRA-20209)
+ * Port Harry v2 to trunk (CASSANDRA-20200)
+ * Enable filtering of snapshots on keyspace, table and snapshot name in nodetool listsnapshots (CASSANDRA-20151)
+ * Create manifest upon loading where it does not exist or enrich it (CASSANDRA-20150)
+ * Propagate true size of snapshot in SnapshotDetailsTabularData to not call JMX twice in nodetool listsnapshots (CASSANDRA-20149)
+ * Implementation of CEP-43 - copying a table via CQL by CREATE TABLE LIKE (CASSANDRA-19964)
+ * Periodically disconnect roles that are revoked or have LOGIN=FALSE set (CASSANDRA-19385)
+ * AST library for CQL-based fuzz tests (CASSANDRA-20198)
+ * Support audit logging for JMX operations (CASSANDRA-20128)
+ * Enable sorting of nodetool status output (CASSANDRA-20104)
+ * Support downgrading after CMS is initialized (CASSANDRA-20145)
+ * Deprecate IEndpointSnitch (CASSANDRA-19488)
+ * Check presence of a snapshot in a case-insensitive manner on macOS platform to prevent hardlinking failures (CASSANDRA-20146)
+ * Enable JMX server configuration to be in cassandra.yaml (CASSANDRA-11695)
+ * Parallelized UCS compactions (CASSANDRA-18802)
+ * Avoid prepared statement invalidation race when committing schema changes (CASSANDRA-20116)
+ * Restore optimization in MultiCBuilder around building one clustering (CASSANDRA-20129)
+ * Consolidate all snapshot management to SnapshotManager and introduce SnapshotManagerMBean (CASSANDRA-18111)
+ * Fix RequestFailureReason constants codes (CASSANDRA-20126)
+ * Introduce SSTableSimpleScanner for compaction (CASSANDRA-20092)
+ * Include column drop timestamp in alter table transformation (CASSANDRA-18961)
+ * Make JMX SSL configurable in cassandra.yaml (CASSANDRA-18508)
+ * Fix cqlsh CAPTURE command to save query results without trace details when TRACING is ON (CASSANDRA-19105)
+ * Optionally prevent tombstone purging during repair (CASSANDRA-20071)
+ * Add post-filtering support for the IN operator in SAI queries (CASSANDRA-20025)
+ * Don’t finish ongoing decommission and move operations during startup (CASSANDRA-20040)
+ * Nodetool reconfigure cms has correct return code when streaming fails (CASSANDRA-19972)
+ * Reintroduce RestrictionSet#iterator() optimization around multi-column restrictions (CASSANDRA-20034)
+ * Explicitly localize strings to Locale.US for internal implementation (CASSANDRA-19953)
+ * Add -H option for human-friendly output in nodetool compactionhistory (CASSANDRA-20015)
+ * Fix type check for referenced duration type for nested types (CASSANDRA-19890)
+ * In simulation tests, correctly set the tokens of replacement nodes (CASSANDRA-19997)
+ * During TCM upgrade, retain all properties of existing system tables (CASSANDRA-19992)
+ * Properly cancel in-flight futures and reject requests in EpochAwareDebounce during shutdown (CASSANDRA-19848)
+ * Provide clearer exception message on failing commitlog_disk_access_mode combinations (CASSANDRA-19812)
+ * Add total space used for a keyspace to nodetool tablestats (CASSANDRA-19671)
+ * Ensure Relation#toRestriction() handles ReversedType properly (CASSANDRA-19950)
+ * Add JSON and YAML output option to nodetool gcstats (CASSANDRA-19771)
+ * Introduce metadata serialization version V4 (CASSANDRA-19970)
+ * Allow CMS reconfiguration to work around DOWN nodes (CASSANDRA-19943)
+ * Make TableParams.Serializer set allowAutoSnapshots and incrementalBackups (CASSANDRA-19954)
+ * Make sstabledump possible to show tombstones only (CASSANDRA-19939)
+ * Ensure that RFP queries potentially stale replicas even with only key columns in the row filter (CASSANDRA-19938)
+ * Allow nodes to change IP address while upgrading to TCM (CASSANDRA-19921)
+ * Retain existing keyspace params on system tables after upgrade (CASSANDRA-19916)
+ * Deprecate use of gossip state for paxos electorate verification (CASSANDRA-19904)
+ * Update dtest-api to 0.0.17 to fix jvm17 crash in jvm-dtests (CASSANDRA-19239)
+ * Add resource leak test and Update Netty to 4.1.113.Final to fix leak (CASSANDRA-19783)
+ * Fix incorrect nodetool suggestion when gossip mode is running (CASSANDRA-19905)
+ * SAI support for BETWEEN operator (CASSANDRA-19688)
+ * Fix BETWEEN filtering for reversed clustering columns (CASSANDRA-19878)
+ * Retry if node leaves CMS while committing a transformation (CASSANDRA-19872)
+ * Add support for NOT operators in WHERE clauses. Fixed Three Valued Logic (CASSANDRA-18584)
+ * Allow getendpoints for system tables and make sure getNaturalReplicas work for MetaStrategy (CASSANDRA-19846)
+ * On upgrade, handle pre-existing tables with unexpected table ids (CASSANDRA-19845)
+ * Reconfigure CMS before assassinate (CASSANDRA-19768)
+ * Warn about unqualified prepared statement only if it is select or modification statement (CASSANDRA-18322)
+ * Update legacy peers tables during node replacement (CASSANDRA-19782)
+ * Refactor ColumnCondition (CASSANDRA-19620)
+ * Allow configuring log format for Audit Logs (CASSANDRA-19792)
+ * Support for noboolean rpm (centos7 compatible) packages removed (CASSANDRA-19787)
+ * Allow threads waiting for the metadata log follower to be interrupted (CASSANDRA-19761)
+ * Support dictionary lookup for CassandraPasswordValidator (CASSANDRA-19762)
+ * Disallow denylisting keys in system_cluster_metadata (CASSANDRA-19713)
+ * Fix gossip status after replacement (CASSANDRA-19712)
+ * Ignore repair requests for system_cluster_metadata (CASSANDRA-19711)
+ * Avoid ClassCastException when verifying tables with reversed partitioner (CASSANDRA-19710)
+ * Always repair the full range when repairing system_cluster_metadata (CASSANDRA-19709)
+ * Use table-specific partitioners during Paxos repair (CASSANDRA-19714)
+ * Expose current compaction throughput in nodetool (CASSANDRA-13890)
+ * CEP-24 Password validation / generation (CASSANDRA-17457)
+ * Reconfigure CMS after replacement, bootstrap and move operations (CASSANDRA-19705)
+ * Support querying LocalStrategy tables with any partitioner (CASSANDRA-19692)
+ * Relax slow_query_log_timeout for MultiNodeSAITest (CASSANDRA-19693)
+ * Audit Log entries are missing identity for mTLS connections (CASSANDRA-19669)
+ * Add support for the BETWEEN operator in WHERE clauses (CASSANDRA-19604)
+ * Replace Stream iteration with for-loop for SimpleRestriction::bindAndGetClusteringElements (CASSANDRA-19679)
+ * Consolidate logging on trace level (CASSANDRA-19632)
+ * Expand DDL statements on coordinator before submission to the CMS (CASSANDRA-19592)
+ * Fix number of arguments of String.format() in various classes (CASSANDRA-19645)
+ * Remove unused fields from config (CASSANDRA-19599)
+ * Refactor Relation and Restriction hierarchies (CASSANDRA-19341)
+ * Raise priority of TCM internode messages during critical operations (CASSANDRA-19517)
+ * Add nodetool command to unregister LEFT nodes (CASSANDRA-19581)
+ * Add cluster metadata id to gossip syn messages (CASSANDRA-19613)
+ * Reduce heap usage occupied by the metrics (CASSANDRA-19567)
+ * Improve handling of transient replicas during range movements (CASSANDRA-19344)
+ * Enable debounced internode log requests to be cancelled at shutdown (CASSANDRA-19514)
+ * Correctly set last modified epoch when combining multistep operations into a single step (CASSANDRA-19538)
+ * Add new TriggersPolicy configuration to allow operators to disable triggers (CASSANDRA-19532)
+ * Use Transformation.Kind.id in local and distributed log tables (CASSANDRA-19516)
+ * Remove period field from ClusterMetadata and metadata log tables (CASSANDRA-19482)
+ * Enrich system_views.pending_hints vtable with hints sizes (CASSANDRA-19486)
+ * Expose all dropwizard metrics in virtual tables (CASSANDRA-14572)
+ * Ensured that PropertyFileSnitchTest do not overwrite cassandra-toploogy.properties (CASSANDRA-19502)
+ * Add option for MutualTlsAuthenticator to restrict the certificate validity period (CASSANDRA-18951)
+ * Fix StorageService::constructRangeToEndpointMap for non-distributed keyspaces (CASSANDRA-19255)
+ * Group nodetool cms commands into single command group (CASSANDRA-19393)
+ * Register the measurements of the bootstrap process as Dropwizard metrics (CASSANDRA-19447)
+ * Add LIST SUPERUSERS CQL statement (CASSANDRA-19417)
+ * Modernize CQLSH datetime conversions (CASSANDRA-18879)
+ * Harry model and in-JVM tests for partition-restricted 2i queries (CASSANDRA-18275)
+ * Refactor cqlshmain global constants (CASSANDRA-19201)
+ * Remove native_transport_port_ssl (CASSANDRA-19397)
+ * Make nodetool reconfigurecms sync by default and add --cancel to be able to cancel ongoing reconfigurations (CASSANDRA-19216)
+ * Expose auth mode in system_views.clients, nodetool clientstats, metrics (CASSANDRA-19366)
+ * Remove sealed_periods and last_sealed_period tables (CASSANDRA-19189)
+ * Improve setup and initialisation of LocalLog/LogSpec (CASSANDRA-19271)
+ * Refactor structure of caching metrics and expose auth cache metrics via JMX (CASSANDRA-17062)
+ * Allow CQL client certificate authentication to work without sending an AUTHENTICATE request (CASSANDRA-18857)
+ * Extend nodetool tpstats and system_views.thread_pools with detailed pool parameters (CASSANDRA-19289)
+ * Remove dependency on Sigar in favor of OSHI (CASSANDRA-16565)
+ * Simplify the bind marker and Term logic (CASSANDRA-18813)
+ * Limit cassandra startup to supported JDKs, allow higher JDKs by setting CASSANDRA_JDK_UNSUPPORTED (CASSANDRA-18688)
+ * Standardize nodetool tablestats formatting of data units (CASSANDRA-19104)
+ * Make nodetool tablestats use number of significant digits for time and average values consistently (CASSANDRA-19015)
+ * Upgrade jackson to 2.15.3 and snakeyaml to 2.1 (CASSANDRA-18875)
+ * Transactional Cluster Metadata [CEP-21] (CASSANDRA-18330)
+ * Add ELAPSED command to cqlsh (CASSANDRA-18861)
+ * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781)
+ * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787)
+Merged from 5.0:
+ * Avoid fetching entire partitions on unresolved static rows in RFP when no static column predicates exist (CASSANDRA-20243)
* Avoid indexing empty values for non-literals and types that do not allow them (CASSANDRA-20313)
* Fix incorrect results of min / max in-built functions on clustering columns in descending order (CASSANDRA-20295)
* Avoid possible consistency violations for SAI intersection queries over repaired index matches and multiple non-indexed column matches (CASSANDRA-20189)
diff --cc test/distributed/org/apache/cassandra/distributed/test/cql3/SingleNodeTableWalkTest.java
index 6412fdf2d9,0000000000..c18d6cfecf
mode 100644,000000..100644
--- a/test/distributed/org/apache/cassandra/distributed/test/cql3/SingleNodeTableWalkTest.java
+++ b/test/distributed/org/apache/cassandra/distributed/test/cql3/SingleNodeTableWalkTest.java
@@@ -1,594 -1,0 +1,563 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.test.cql3;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.NavigableSet;
+import java.util.Optional;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Sets;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import accord.utils.Gen;
+import accord.utils.Gens;
+import accord.utils.Property;
+import accord.utils.RandomSource;
+import org.apache.cassandra.cql3.KnownIssue;
+import org.apache.cassandra.cql3.ast.Bind;
+import org.apache.cassandra.cql3.ast.Conditional;
+import org.apache.cassandra.cql3.ast.CreateIndexDDL;
+import org.apache.cassandra.cql3.ast.FunctionCall;
+import org.apache.cassandra.cql3.ast.Mutation;
+import org.apache.cassandra.cql3.ast.ReferenceExpression;
+import org.apache.cassandra.cql3.ast.Select;
+import org.apache.cassandra.cql3.ast.Symbol;
+import org.apache.cassandra.cql3.ast.TableReference;
+import org.apache.cassandra.cql3.ast.Value;
+import org.apache.cassandra.db.Clustering;
+import org.apache.cassandra.db.marshal.AbstractType;
+import org.apache.cassandra.db.marshal.InetAddressType;
+import org.apache.cassandra.db.marshal.UTF8Type;
+import org.apache.cassandra.dht.Murmur3Partitioner;
+import org.apache.cassandra.distributed.Cluster;
+import org.apache.cassandra.distributed.test.sai.SAIUtil;
+import org.apache.cassandra.harry.model.BytesPartitionState;
+import org.apache.cassandra.schema.ColumnMetadata;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.utils.ASTGenerators;
+import org.apache.cassandra.utils.AbstractTypeGenerators;
+import org.apache.cassandra.utils.AbstractTypeGenerators.TypeGenBuilder;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.CassandraGenerators.TableMetadataBuilder;
+import org.apache.cassandra.utils.ImmutableUniqueList;
+import org.quicktheories.generators.SourceDSL;
+
+import static accord.utils.Property.commands;
+import static accord.utils.Property.stateful;
+import static org.apache.cassandra.utils.AbstractTypeGenerators.getTypeSupport;
+import static org.apache.cassandra.utils.Generators.toGen;
+
+public class SingleNodeTableWalkTest extends StatefulASTBase
+{
+ private static final Logger logger = LoggerFactory.getLogger(SingleNodeTableWalkTest.class);
+
+ protected void preCheck(Cluster cluster, Property.StatefulBuilder builder)
+ {
+ // if a failing seed is detected, populate here
+ // Example: builder.withSeed(42L);
+ // CQL operations may have opertors such as +, -, and / (example 4 + 4), to "apply" them to get a constant value
+ // CQL_DEBUG_APPLY_OPERATOR = true;
+ }
+
+ protected TypeGenBuilder supportedTypes()
+ {
+ return AbstractTypeGenerators.withoutUnsafeEquality(AbstractTypeGenerators.builder()
+ .withTypeKinds(AbstractTypeGenerators.TypeKind.PRIMITIVE));
+ }
+
+ protected List<CreateIndexDDL.Indexer> supportedIndexers()
+ {
+ // since legacy is async it's not clear how the test can wait for the background write to complete...
+ return Collections.singletonList(CreateIndexDDL.SAI);
+ }
+
+ public Property.Command<State, Void, ?> selectExisting(RandomSource rs, State state)
+ {
+ NavigableSet<BytesPartitionState.Ref> keys = state.model.partitionKeys();
+ BytesPartitionState.Ref ref = rs.pickOrderedSet(keys);
+ Clustering<ByteBuffer> key = ref.key;
+
+ Select.Builder builder = Select.builder().table(state.metadata);
+ ImmutableUniqueList<Symbol> pks = state.model.factory.partitionColumns;
+ ImmutableUniqueList<Symbol> cks = state.model.factory.clusteringColumns;
+ for (Symbol pk : pks)
+ builder.value(pk, key.bufferAt(pks.indexOf(pk)));
+
+ boolean wholePartition = cks.isEmpty() || rs.nextBoolean();
+ if (!wholePartition)
+ {
+ // find a row to select
+ BytesPartitionState partition = state.model.get(ref);
+ if (partition.isEmpty())
+ {
+ wholePartition = true;
+ }
+ else
+ {
+ NavigableSet<Clustering<ByteBuffer>> clusteringKeys = partition.clusteringKeys();
+ Clustering<ByteBuffer> clusteringKey = rs.pickOrderedSet(clusteringKeys);
+ for (Symbol ck : cks)
+ builder.value(ck, clusteringKey.bufferAt(cks.indexOf(ck)));
+ }
+ }
+ Select select = builder.build();
+ return state.command(rs, select, (wholePartition ? "Whole Partition" : "Single Row"));
+ }
+
+ public Property.Command<State, Void, ?> selectToken(RandomSource rs, State state)
+ {
+ NavigableSet<BytesPartitionState.Ref> keys = state.model.partitionKeys();
+ BytesPartitionState.Ref ref = rs.pickOrderedSet(keys);
+
+ Select.Builder builder = Select.builder().table(state.metadata);
+ builder.where(FunctionCall.tokenByColumns(state.model.factory.partitionColumns),
+ Conditional.Where.Inequality.EQUAL,
+ token(state, ref));
+
+ Select select = builder.build();
+ return state.command(rs, select, "by token");
+ }
+
+ public Property.Command<State, Void, ?> selectTokenRange(RandomSource rs, State state)
+ {
+ NavigableSet<BytesPartitionState.Ref> keys = state.model.partitionKeys();
+ BytesPartitionState.Ref start, end;
+ switch (keys.size())
+ {
+ case 1:
+ start = end = Iterables.get(keys, 0);
+ break;
+ case 2:
+ start = Iterables.get(keys, 0);
+ end = Iterables.get(keys, 1);
+ break;
+ case 0:
+ throw new IllegalArgumentException("Unable to select token ranges when no partitions exist");
+ default:
+ {
+ int si = rs.nextInt(0, keys.size() - 1);
+ int ei = rs.nextInt(si + 1, keys.size());
+ start = Iterables.get(keys, si);
+ end = Iterables.get(keys, ei);
+ }
+ break;
+ }
+ Select.Builder builder = Select.builder().table(state.metadata);
+ FunctionCall pkToken = FunctionCall.tokenByColumns(state.model.factory.partitionColumns);
+ boolean startInclusive = rs.nextBoolean();
+ boolean endInclusive = rs.nextBoolean();
+ if (startInclusive && endInclusive && rs.nextBoolean())
+ {
+ // between
+ builder.between(pkToken, token(state, start), token(state, end));
+ }
+ else
+ {
+ builder.where(pkToken,
+ startInclusive ? Conditional.Where.Inequality.GREATER_THAN_EQ : Conditional.Where.Inequality.GREATER_THAN,
+ token(state, start));
+ builder.where(pkToken,
+ endInclusive ? Conditional.Where.Inequality.LESS_THAN_EQ : Conditional.Where.Inequality.LESS_THAN,
+ token(state, end));
+ }
+ Select select = builder.build();
+ return state.command(rs, select, "by token range");
+ }
+
+ public Property.Command<State, Void, ?> partitionRestrictedQuery(RandomSource rs, State state)
+ {
+ //TODO (now): remove duplicate logic
+ NavigableSet<BytesPartitionState.Ref> keys = state.model.partitionKeys();
+ BytesPartitionState.Ref ref = rs.pickOrderedSet(keys);
+ Clustering<ByteBuffer> key = ref.key;
+
+ Select.Builder builder = Select.builder().table(state.metadata);
+ ImmutableUniqueList<Symbol> pks = state.model.factory.partitionColumns;
+ for (Symbol pk : pks)
+ builder.value(pk, key.bufferAt(pks.indexOf(pk)));
+
+
- Symbol symbol;
+ List<Symbol> searchableColumns = state.nonPartitionColumns;
- if (state.hasMultiNodeAllowFilteringWithLocalWritesIssue())
- {
- if (state.nonPkIndexedColumns.isEmpty())
- throw new AssertionError("Ignoring AF_MULTI_NODE_AND_NODE_LOCAL_WRITES is defined, but no non-partition columns are indexed");
- symbol = rs.pick(state.nonPkIndexedColumns);
- }
- else
- {
- symbol = rs.pick(searchableColumns);
- }
++ Symbol symbol = rs.pick(searchableColumns);
+
+ TreeMap<ByteBuffer, List<BytesPartitionState.PrimaryKey>> universe = state.model.index(ref, symbol);
+ // we need to index 'null' so LT works, but we can not directly query it... so filter out when selecting values
+ NavigableSet<ByteBuffer> allowed = Sets.filter(universe.navigableKeySet(), b -> !ByteBufferUtil.EMPTY_BYTE_BUFFER.equals(b));
+ if (allowed.isEmpty())
+ return Property.ignoreCommand();
+ ByteBuffer value = rs.pickOrderedSet(allowed);
+
+ EnumSet<CreateIndexDDL.QueryType> supported = !state.indexes.containsKey(symbol)
+ ? EnumSet.noneOf(CreateIndexDDL.QueryType.class)
+ : state.indexes.get(symbol).supportedQueries();
+ if (supported.isEmpty() || !supported.contains(CreateIndexDDL.QueryType.Range))
+ builder.allowFiltering();
+
+ // there are known SAI bugs, so need to avoid them to stay stable...
+ if (state.indexes.containsKey(symbol) && state.indexes.get(symbol).indexDDL.indexer == CreateIndexDDL.SAI)
+ {
+ if (symbol.type() == InetAddressType.instance
+ && IGNORED_ISSUES.contains(KnownIssue.SAI_INET_MIXED))
+ return eqSearch(rs, state, symbol, value, builder);
+ }
+
+ if (rs.nextBoolean())
+ return simpleRangeSearch(rs, state, symbol, value, builder);
+ //TODO (coverage): define search that has a upper and lower bound: a > and a < | a beteeen ? and ?
+ return eqSearch(rs, state, symbol, value, builder);
+ }
+
+ public Property.Command<State, Void, ?> nonPartitionQuery(RandomSource rs, State state)
+ {
- Symbol symbol;
- if (state.hasMultiNodeAllowFilteringWithLocalWritesIssue())
- {
- symbol = rs.pickUnorderedSet(state.indexes.keySet());
- }
- else
- {
- symbol = rs.pick(state.searchableColumns);
- }
++ Symbol symbol = rs.pick(state.searchableColumns);
+ TreeMap<ByteBuffer, List<BytesPartitionState.PrimaryKey>> universe = state.model.index(symbol);
+ // we need to index 'null' so LT works, but we can not directly query it... so filter out when selecting values
+ NavigableSet<ByteBuffer> allowed = Sets.filter(universe.navigableKeySet(), b -> !ByteBufferUtil.EMPTY_BYTE_BUFFER.equals(b));
+ if (allowed.isEmpty())
+ return Property.ignoreCommand();
+ ByteBuffer value = rs.pickOrderedSet(allowed);
+ Select.Builder builder = Select.builder().table(state.metadata);
+
+ EnumSet<CreateIndexDDL.QueryType> supported = !state.indexes.containsKey(symbol) ? EnumSet.noneOf(CreateIndexDDL.QueryType.class) : state.indexes.get(symbol).supportedQueries();
+ if (supported.isEmpty() || !supported.contains(CreateIndexDDL.QueryType.Range))
+ builder.allowFiltering();
+
+ // there are known SAI bugs, so need to avoid them to stay stable...
+ if (state.indexes.containsKey(symbol) && state.indexes.get(symbol).indexDDL.indexer == CreateIndexDDL.SAI)
+ {
+ if (symbol.type() == InetAddressType.instance
+ && IGNORED_ISSUES.contains(KnownIssue.SAI_INET_MIXED))
+ return eqSearch(rs, state, symbol, value, builder);
+ }
+
+ if (rs.nextBoolean())
+ return simpleRangeSearch(rs, state, symbol, value, builder);
+ //TODO (coverage): define search that has a upper and lower bound: a > and a < | a beteeen ? and ?
+ return eqSearch(rs, state, symbol, value, builder);
+ }
+
+ public Property.Command<State, Void, ?> multiColumnQuery(RandomSource rs, State state)
+ {
+ List<Symbol> allowedColumns = state.multiColumnQueryColumns();
+
+ if (allowedColumns.size() <= 1)
+ throw new IllegalArgumentException("Unable to do multiple column query when there is only a single column");
+
+ int numColumns = rs.nextInt(1, allowedColumns.size()) + 1;
+
+ List<Symbol> cols = Gens.lists(Gens.pick(allowedColumns)).unique().ofSize(numColumns).next(rs);
+
+ Select.Builder builder = Select.builder().table(state.metadata).allowFiltering();
+
+ for (Symbol symbol : cols)
+ {
+ TreeMap<ByteBuffer, List<BytesPartitionState.PrimaryKey>> universe = state.model.index(symbol);
+ NavigableSet<ByteBuffer> allowed = Sets.filter(universe.navigableKeySet(), b -> !ByteBufferUtil.EMPTY_BYTE_BUFFER.equals(b));
+ //TODO (now): support
+ if (allowed.isEmpty())
+ return Property.ignoreCommand();
+ ByteBuffer value = rs.pickOrderedSet(allowed);
+ builder.value(symbol, value);
+ }
+
+ Select select = builder.build();
+ String annotate = cols.stream().map(symbol -> {
+ var indexed = state.indexes.get(symbol);
+ return symbol.detailedName() + (indexed == null ? "" : " (indexed with " + indexed.indexDDL.indexer.name() + ")");
+ }).collect(Collectors.joining(", "));
+ return state.command(rs, select, annotate);
+ }
+
+ private Property.Command<State, Void, ?> simpleRangeSearch(RandomSource rs, State state, Symbol symbol, ByteBuffer value, Select.Builder builder)
+ {
+ // do a simple search, like > or <
+ Conditional.Where.Inequality kind = state.rangeInequalityGen.next(rs);
+ builder.where(symbol, kind, value);
+ Select select = builder.build();
+ var indexed = state.indexes.get(symbol);
+ return state.command(rs, select, symbol.detailedName() + (indexed == null ? "" : ", indexed with " + indexed.indexDDL.indexer.name()));
+ }
+
+ private Property.Command<State, Void, ?> eqSearch(RandomSource rs, State state, Symbol symbol, ByteBuffer value, Select.Builder builder)
+ {
+ builder.value(symbol, value);
+
+ Select select = builder.build();
+ var indexed = state.indexes.get(symbol);
+ return state.command(rs, select, symbol.detailedName() + (indexed == null ? "" : ", indexed with " + indexed.indexDDL.indexer.name()));
+ }
+
+ protected State createState(RandomSource rs, Cluster cluster)
+ {
+ return new State(rs, cluster);
+ }
+
+ protected Cluster createCluster() throws IOException
+ {
+ return createCluster(1, i -> {});
+ }
+
+ @Test
+ public void test() throws IOException
+ {
+ try (Cluster cluster = createCluster())
+ {
+ Property.StatefulBuilder statefulBuilder = stateful().withExamples(10).withSteps(400);
+ preCheck(cluster, statefulBuilder);
+ statefulBuilder.check(commands(() -> rs -> createState(rs, cluster))
+ .add(StatefulASTBase::insert)
+ .add(StatefulASTBase::fullTableScan)
+ .addIf(State::hasPartitions, this::selectExisting)
+ .addAllIf(State::supportTokens, b -> b.add(this::selectToken)
+ .add(this::selectTokenRange))
+ .addIf(State::hasEnoughMemtable, StatefulASTBase::flushTable)
+ .addIf(State::hasEnoughSSTables, StatefulASTBase::compactTable)
+ .addIf(State::allowNonPartitionQuery, this::nonPartitionQuery)
+ .addIf(State::allowNonPartitionMultiColumnQuery, this::multiColumnQuery)
+ .addIf(State::allowPartitionQuery, this::partitionRestrictedQuery)
+ .destroyState(State::close)
+ .onSuccess(onSuccess(logger))
+ .build());
+ }
+ }
+
+ protected TableMetadata defineTable(RandomSource rs, String ks)
+ {
+ //TODO (correctness): the id isn't correct... this is what we use to create the table, so would miss the actual ID
+ // Defaults may also be incorrect, but given this is the same version it "shouldn't"
+ //TODO (coverage): partition is defined at the cluster level, so have to hard code in this model as the table is changed rather than cluster being recreated... this limits coverage
+ return toGen(new TableMetadataBuilder()
+ .withTableKinds(TableMetadata.Kind.REGULAR)
+ .withKnownMemtables()
+ .withKeyspaceName(ks).withTableName("tbl")
+ .withSimpleColumnNames()
+ .withDefaultTypeGen(supportedTypes())
+ .withPartitioner(Murmur3Partitioner.instance)
+ .build())
+ .next(rs);
+ }
+
+ private List<CreateIndexDDL.Indexer> columnSupportsIndexing(TableMetadata metadata, ColumnMetadata col)
+ {
+ return supportedIndexers().stream()
+ .filter(i -> i.supported(metadata, col))
+ .collect(Collectors.toList());
+ }
+
+ private static FunctionCall token(State state, BytesPartitionState.Ref ref)
+ {
+ Preconditions.checkNotNull(ref.key);
+ List<Value> values = new ArrayList<>(ref.key.size());
+ for (int i = 0; i < ref.key.size(); i++)
+ {
+ ByteBuffer bb = ref.key.bufferAt(i);
+ Symbol type = state.model.factory.partitionColumns.get(i);
+ values.add(new Bind(bb, type.type()));
+ }
+ return FunctionCall.tokenByValue(values);
+ }
+
+ public class State extends CommonState
+ {
+ protected final LinkedHashMap<Symbol, IndexedColumn> indexes;
+ private final Gen<Mutation> mutationGen;
+ private final List<Symbol> nonPartitionColumns;
+ private final List<Symbol> searchableColumns;
++ private final List<Symbol> nonPkIndexedColumns;
+
+ public State(RandomSource rs, Cluster cluster)
+ {
+ super(rs, cluster, defineTable(rs, nextKeyspace()));
+
+ this.indexes = createIndexes(rs, metadata);
+
+ cluster.forEach(i -> i.nodetoolResult("disableautocompaction", metadata.keyspace, this.metadata.name).asserts().success());
+
+ List<LinkedHashMap<Symbol, Object>> uniquePartitions;
+ {
+ int unique = rs.nextInt(1, 10);
+ List<Symbol> columns = model.factory.partitionColumns;
+ List<Gen<?>> gens = new ArrayList<>(columns.size());
+ for (int i = 0; i < columns.size(); i++)
+ gens.add(toGen(getTypeSupport(columns.get(i).type()).valueGen));
+ uniquePartitions = Gens.lists(r2 -> {
+ LinkedHashMap<Symbol, Object> vs = new LinkedHashMap<>();
+ for (int i = 0; i < columns.size(); i++)
+ vs.put(columns.get(i), gens.get(i).next(r2));
+ return vs;
+ }).uniqueBestEffort().ofSize(unique).next(rs);
+ }
+
+ ASTGenerators.MutationGenBuilder mutationGenBuilder = new ASTGenerators.MutationGenBuilder(metadata)
+ .withoutTransaction()
+ .withoutTtl()
+ .withoutTimestamp()
+ .withPartitions(SourceDSL.arbitrary().pick(uniquePartitions));
+ if (IGNORED_ISSUES.contains(KnownIssue.SAI_EMPTY_TYPE))
+ {
+ model.factory.regularAndStaticColumns.stream()
+ // exclude SAI indexed columns
+ .filter(s -> !indexes.containsKey(s) || indexes.get(s).indexDDL.indexer != CreateIndexDDL.SAI)
+ .forEach(mutationGenBuilder::allowEmpty);
+ }
+ else
+ {
+ model.factory.regularAndStaticColumns.forEach(mutationGenBuilder::allowEmpty);
+ }
+ this.mutationGen = toGen(mutationGenBuilder.build());
+
+ nonPartitionColumns = ImmutableList.<Symbol>builder()
+ .addAll(model.factory.clusteringColumns)
+ .addAll(model.factory.staticColumns)
+ .addAll(model.factory.regularColumns)
+ .build();
++ nonPkIndexedColumns = nonPartitionColumns.stream()
++ .filter(indexes::containsKey)
++ .collect(Collectors.toList());
+
+ searchableColumns = metadata.partitionKeyColumns().size() > 1 ? model.factory.selectionOrder : nonPartitionColumns;
+ }
+
+ @Override
+ protected Gen<Mutation> mutationGen()
+ {
+ return mutationGen;
+ }
+
+ private LinkedHashMap<Symbol, IndexedColumn> createIndexes(RandomSource rs, TableMetadata metadata)
+ {
+ LinkedHashMap<Symbol, IndexedColumn> indexed = new LinkedHashMap<>();
+ // for some test runs, avoid using indexes
+ if (rs.nextBoolean())
+ return indexed;
+ for (ColumnMetadata col : metadata.columnsInFixedOrder())
+ {
+ Symbol symbol = Symbol.from(col);
+ AbstractType<?> type = symbol.type();
+
+ if (col.name.toString().length() >= 48
+ && IGNORED_ISSUES.contains(KnownIssue.CUSTOM_INDEX_MAX_COLUMN_48))
+ continue;
+
+ if (type.isCollection() && !type.isFrozenCollection()) continue; //TODO (coverage): include non-frozen collections; the index part works fine, its the select that fails... basic equality isn't allowed for map type... so how do you query?
+ List<CreateIndexDDL.Indexer> allowed = columnSupportsIndexing(metadata, col);
+ if (allowed.isEmpty()) continue;
+ CreateIndexDDL.Indexer indexer = rs.pick(allowed);
+ ReferenceExpression colExpression = Symbol.from(col);
+ if (type.isFrozenCollection())
+ colExpression = new CreateIndexDDL.CollectionReference(CreateIndexDDL.CollectionReference.Kind.FULL, colExpression);
+
+ String name = "tbl_" + col.name;
+ CreateIndexDDL ddl = new CreateIndexDDL(rs.pick(CreateIndexDDL.Version.values()),
+ indexer,
+ Optional.of(new Symbol(name, UTF8Type.instance)),
+ TableReference.from(metadata),
+ Collections.singletonList(colExpression),
+ Collections.emptyMap());
+ String stmt = ddl.toCQL();
+ logger.info(stmt);
+ cluster.schemaChange(stmt);
+
+ //noinspection OptionalGetWithoutIsPresent
+ SAIUtil.waitForIndexQueryable(cluster, metadata.keyspace, ddl.name.get().name());
+
+ indexed.put(symbol, new IndexedColumn(symbol, ddl));
+ }
+ return indexed;
+ }
+
+ public boolean hasPartitions()
+ {
+ return !model.isEmpty();
+ }
+
+ public boolean supportTokens()
+ {
+ return hasPartitions();
+ }
+
+ public boolean allowNonPartitionQuery()
+ {
- boolean result = !model.isEmpty() && !searchableColumns.isEmpty();
- if (hasMultiNodeAllowFilteringWithLocalWritesIssue())
- {
- return hasNonPkIndexedColumns() && result;
- }
- return result;
++ return !model.isEmpty() && !searchableColumns.isEmpty();
+ }
+
+ public boolean allowNonPartitionMultiColumnQuery()
+ {
+ return allowNonPartitionQuery() && multiColumnQueryColumns().size() > 1;
+ }
+
+ private List<Symbol> multiColumnQueryColumns()
+ {
+ List<Symbol> allowedColumns = searchableColumns;
- if (hasMultiNodeAllowFilteringWithLocalWritesIssue())
++ if (hasMultiNodeMultiColumnAllowFilteringWithLocalWritesIssue())
+ allowedColumns = nonPkIndexedColumns;
+ return allowedColumns;
+ }
+
- private boolean hasMultiNodeAllowFilteringWithLocalWritesIssue()
++ private boolean hasMultiNodeMultiColumnAllowFilteringWithLocalWritesIssue()
+ {
- return isMultiNode() && IGNORED_ISSUES.contains(KnownIssue.AF_MULTI_NODE_AND_NODE_LOCAL_WRITES);
++ return isMultiNode() && IGNORED_ISSUES.contains(KnownIssue.AF_MULTI_NODE_MULTI_COLUMN_AND_NODE_LOCAL_WRITES);
+ }
+
- public List<Symbol> nonPkIndexedColumns;
+ public boolean allowPartitionQuery()
+ {
- if (model.isEmpty() || nonPartitionColumns.isEmpty()) return false;
- if (hasMultiNodeAllowFilteringWithLocalWritesIssue())
- return hasNonPkIndexedColumns();
- return true;
- }
-
- private boolean hasNonPkIndexedColumns()
- {
- nonPkIndexedColumns = nonPartitionColumns.stream()
- .filter(indexes::containsKey)
- .collect(Collectors.toList());
- return !nonPkIndexedColumns.isEmpty();
++ return !(model.isEmpty() || nonPartitionColumns.isEmpty());
+ }
+
+ @Override
+ public String toString()
+ {
+ StringBuilder sb = new StringBuilder();
+ sb.append("\nSetup:\n");
+ toString(sb);
+ indexes.values().forEach(c -> sb.append('\n').append(c.indexDDL.toCQL()).append(';'));
+ return sb.toString();
+ }
+ }
+
+ public static class IndexedColumn
+ {
+ public final Symbol symbol;
+ public final CreateIndexDDL indexDDL;
+
+ public IndexedColumn(Symbol symbol, CreateIndexDDL indexDDL)
+ {
+ this.symbol = symbol;
+ this.indexDDL = indexDDL;
+ }
+
+ public EnumSet<CreateIndexDDL.QueryType> supportedQueries()
+ {
+ return indexDDL.indexer.supportedQueries(symbol.type());
+ }
+ }
+}
diff --cc test/distributed/org/apache/cassandra/distributed/test/sai/StrictFilteringTest.java
index 60646c6630,5ef92bb9ef..5905c4f8a7
--- a/test/distributed/org/apache/cassandra/distributed/test/sai/StrictFilteringTest.java
+++ b/test/distributed/org/apache/cassandra/distributed/test/sai/StrictFilteringTest.java
@@@ -25,10 -26,11 +25,9 @@@ import org.junit.AfterClass
import org.junit.BeforeClass;
import org.junit.Test;
-import org.apache.cassandra.cql3.Operator;
import org.apache.cassandra.distributed.Cluster;
import org.apache.cassandra.distributed.api.ConsistencyLevel;
- import org.apache.cassandra.distributed.shared.AssertUtils;
import org.apache.cassandra.distributed.test.TestBaseImpl;
-import org.apache.cassandra.index.sai.plan.StorageAttachedIndexQueryPlan;
import static org.apache.cassandra.distributed.api.Feature.GOSSIP;
import static org.apache.cassandra.distributed.api.Feature.NETWORK;
diff --cc test/distributed/org/apache/cassandra/fuzz/sai/SingleNodeSAITestBase.java
index afdb52e716,0000000000..b915d1748e
mode 100644,000000..100644
--- a/test/distributed/org/apache/cassandra/fuzz/sai/SingleNodeSAITestBase.java
+++ b/test/distributed/org/apache/cassandra/fuzz/sai/SingleNodeSAITestBase.java
@@@ -1,350 -1,0 +1,354 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.fuzz.sai;
+
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
++import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+
+import com.google.common.collect.Streams;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.CassandraRelevantProperties;
+import org.apache.cassandra.distributed.Cluster;
+import org.apache.cassandra.distributed.api.ConsistencyLevel;
+import org.apache.cassandra.distributed.api.IInstanceConfig;
+import org.apache.cassandra.distributed.test.TestBaseImpl;
+import org.apache.cassandra.harry.SchemaSpec;
+import org.apache.cassandra.harry.dsl.HistoryBuilder;
+import org.apache.cassandra.harry.dsl.HistoryBuilderHelper;
+import org.apache.cassandra.harry.dsl.ReplayingHistoryBuilder;
+import org.apache.cassandra.harry.execution.InJvmDTestVisitExecutor;
+import org.apache.cassandra.harry.gen.EntropySource;
+import org.apache.cassandra.harry.gen.Generator;
+import org.apache.cassandra.harry.gen.Generators;
+import org.apache.cassandra.harry.gen.SchemaGenerators;
+import org.apache.cassandra.index.sai.utils.IndexTermType;
+
+import static org.apache.cassandra.distributed.api.Feature.GOSSIP;
+import static org.apache.cassandra.distributed.api.Feature.NETWORK;
+import static org.apache.cassandra.harry.checker.TestHelper.withRandom;
+import static org.apache.cassandra.harry.dsl.SingleOperationBuilder.IdxRelation;
+
+// TODO: "WITH OPTIONS = {'case_sensitive': 'false', 'normalize': 'true', 'ascii': 'true'};",
+public abstract class SingleNodeSAITestBase extends TestBaseImpl
+{
+ private static final int ITERATIONS = 5;
+
+ private static final int VALIDATION_SKIP = 739;
+ private static final int QUERIES_PER_VALIDATION = 8;
+
- private static final int FLUSH_SKIP = 1499;
- private static final int COMPACTION_SKIP = 1503;
- private static final int DEFAULT_REPAIR_SKIP = 6101;
++ private static final int FLUSH_SKIP = 2217;
++ private static final int COMPACTION_SKIP = 4435;
++ private static final int DEFAULT_REPAIR_SKIP = 8869;
+
- private static final int OPERATIONS_PER_RUN = DEFAULT_REPAIR_SKIP * 5;
++ private static final int OPERATIONS_PER_RUN = 30_000;
+
+ private static final int NUM_PARTITIONS = 64;
+ private static final int NUM_VISITED_PARTITIONS = 16;
+ private static final int MAX_PARTITION_SIZE = 2000;
+
+ private static final int UNIQUE_CELL_VALUES = 5;
+
+ protected static final Logger logger = LoggerFactory.getLogger(SingleNodeSAITest.class);
+ protected static Cluster cluster;
+
+ protected SingleNodeSAITestBase() {}
+
+ @BeforeClass
+ public static void before() throws Throwable
+ {
+ init(1,
+ // At lower fetch sizes, queries w/ hundreds or thousands of matches can take a very long time.
+ defaultConfig().andThen(c -> c.set("range_request_timeout", "180s")
+ .set("read_request_timeout", "180s")
+ .set("write_request_timeout", "180s")
+ .set("native_transport_timeout", "180s")
+ .set("slow_query_log_timeout", "180s")
+ .with(GOSSIP).with(NETWORK))
+ );
+ }
+
+ protected static void init(int nodes, Consumer<IInstanceConfig> cfg) throws Throwable
+ {
+ cluster = Cluster.build()
+ .withNodes(nodes)
+ .withConfig(cfg)
+ .createWithoutStarting();
+ cluster.startup();
+ cluster = init(cluster);
+ }
+ @AfterClass
+ public static void afterClass()
+ {
+ cluster.close();
+ }
+
+ @Before
+ public void beforeEach()
+ {
+ cluster.schemaChange("DROP KEYSPACE IF EXISTS " + KEYSPACE);
+ cluster.schemaChange("CREATE KEYSPACE " + KEYSPACE + " WITH replication = {'class': 'SimpleStrategy', 'replication_factor': " + rf() + "};");
+ }
+
+ protected int rf()
+ {
+ return 1;
+ }
+
+ @Test
+ public void simplifiedSaiTest()
+ {
+ withRandom(rng -> saiTest(rng,
+ SchemaGenerators.trivialSchema(KEYSPACE, "simplified", 1000).generate(rng),
+ () -> true,
+ DEFAULT_REPAIR_SKIP));
+ }
+
+ @Test
+ public void indexOnlySaiTest()
+ {
+ for (int i = 0; i < ITERATIONS; i++)
+ {
+ logger.info("Starting iteration {}...", i);
+ withRandom(rng -> saiTest(rng,
+ schemaGenerator(rng.nextBoolean()).generate(rng),
+ () -> true,
+ rng.nextBoolean() ? DEFAULT_REPAIR_SKIP : Integer.MAX_VALUE));
+ }
+ }
+
+ @Test
+ public void mixedFilteringSaiTest()
+ {
+ for (int i = 0; i < ITERATIONS; i++)
+ {
+ logger.info("Starting iteration {}...", i);
+ withRandom(rng -> saiTest(rng,
+ schemaGenerator(rng.nextBoolean()).generate(rng),
+ () -> rng.nextFloat() < 0.7f,
+ rng.nextBoolean() ? DEFAULT_REPAIR_SKIP : Integer.MAX_VALUE));
+ }
+ }
+
+ private void saiTest(EntropySource rng, SchemaSpec schema, Supplier<Boolean> createIndex, int repairSkip)
+ {
+ logger.info(schema.compile());
+
+ Generator<Integer> globalPkGen = Generators.int32(0, Math.min(NUM_PARTITIONS, schema.valueGenerators.pkPopulation()));
+ Generator<Integer> ckGen = Generators.int32(0, schema.valueGenerators.ckPopulation());
+
- CassandraRelevantProperties.SAI_INTERSECTION_CLAUSE_LIMIT.setInt(100);
+ beforeEach();
+ cluster.forEach(i -> i.nodetool("disableautocompaction"));
+
+ cluster.schemaChange(schema.compile());
+ cluster.schemaChange(schema.compile().replace(schema.keyspace + '.' + schema.table, schema.keyspace + ".debug_table"));
+
++ AtomicInteger indexCount = new AtomicInteger();
++
+ Streams.concat(schema.clusteringKeys.stream(), schema.regularColumns.stream(), schema.staticColumns.stream())
+ .forEach(column -> {
+ if (createIndex.get())
+ {
+ logger.info("Adding index to column {}...", column.name);
+ cluster.schemaChange(String.format("CREATE INDEX %s_sai_idx ON %s.%s (%s) USING 'sai' ",
- column.name, schema.keyspace, schema.table, column.name));
++ column.name, schema.keyspace, schema.table, column.name));
++ indexCount.incrementAndGet();
+ }
+ else
+ {
+ logger.info("Leaving column {} unindexed...", column.name);
+ }
+ });
+
++ CassandraRelevantProperties.SAI_INTERSECTION_CLAUSE_LIMIT.setInt(indexCount.get());
+ waitForIndexesQueryable(schema);
+
+ HistoryBuilder history = new ReplayingHistoryBuilder(schema.valueGenerators,
+ (hb) -> InJvmDTestVisitExecutor.builder()
+ .pageSizeSelector(pageSizeSelector(rng))
+ .consistencyLevel(consistencyLevelSelector())
+ .doubleWriting(schema, hb, cluster, "debug_table"));
+ Set<Integer> partitions = new HashSet<>();
+ int attempts = 0;
+ while (partitions.size() < NUM_VISITED_PARTITIONS && attempts < NUM_VISITED_PARTITIONS * 10)
+ {
+ partitions.add(globalPkGen.generate(rng));
+ attempts++;
+ }
+
+ if (partitions.size() < NUM_VISITED_PARTITIONS)
+ logger.warn("Unable to generate {} partitions to visit. Continuing with {}...", NUM_VISITED_PARTITIONS, partitions.size());
+
+ Generator<Integer> pkGen = Generators.pick(List.copyOf(partitions));
+
+ // Ensure that we don't attempt to use range queries against SAI indexes that don't support them:
+ Set<Integer> eqOnlyRegularColumns = new HashSet<>();
+ for (int i = 0; i < schema.regularColumns.size(); i++)
+ if (IndexTermType.isEqOnlyType(schema.regularColumns.get(i).type.asServerType()))
+ eqOnlyRegularColumns.add(i);
+
+ Set<Integer> eqOnlyStaticColumns = new HashSet<>();
+ for (int i = 0; i < schema.staticColumns.size(); i++)
+ if (IndexTermType.isEqOnlyType(schema.staticColumns.get(i).type.asServerType()))
+ eqOnlyStaticColumns.add(i);
+
+ Set<Integer> eqOnlyClusteringColumns = new HashSet<>();
+ for (int i = 0; i < schema.clusteringKeys.size(); i++)
+ if (IndexTermType.isEqOnlyType(schema.clusteringKeys.get(i).type.asServerType()))
+ eqOnlyClusteringColumns.add(i);
+
+ for (int i = 0; i < OPERATIONS_PER_RUN; i++)
+ {
+ int partitionIndex = pkGen.generate(rng);
+ HistoryBuilderHelper.insertRandomData(schema, partitionIndex, ckGen.generate(rng), rng, 0.5d, history);
+
+ if (rng.nextFloat() > 0.99f)
+ {
+ int row1 = ckGen.generate(rng);
+ int row2 = ckGen.generate(rng);
+ history.deleteRowRange(partitionIndex,
+ Math.min(row1, row2),
+ Math.max(row1, row2),
+ rng.nextInt(schema.clusteringKeys.size()),
+ rng.nextBoolean(),
+ rng.nextBoolean());
+ }
+
+ if (rng.nextFloat() > 0.995f)
+ HistoryBuilderHelper.deleteRandomColumns(schema, partitionIndex, ckGen.generate(rng), rng, history);
+
+ if (rng.nextFloat() > 0.9995f)
+ history.deletePartition(partitionIndex);
+
+ if (i % FLUSH_SKIP == 0)
+ history.custom(() -> flush(schema), "Flush");
+ else if (i % COMPACTION_SKIP == 0)
+ history.custom(() -> compact(schema), "Compact");
+ else if (i % repairSkip == 0)
+ history.custom(() -> repair(schema), "Repair");
+
+ if (i > 0 && i % VALIDATION_SKIP == 0)
+ {
+ for (int j = 0; j < QUERIES_PER_VALIDATION; j++)
+ {
+ List<IdxRelation> regularRelations =
+ HistoryBuilderHelper.generateValueRelations(rng,
+ schema.regularColumns.size(),
+ column -> Math.min(schema.valueGenerators.regularPopulation(column), UNIQUE_CELL_VALUES),
+ eqOnlyRegularColumns::contains);
+
+ List<IdxRelation> staticRelations =
+ HistoryBuilderHelper.generateValueRelations(rng,
+ schema.staticColumns.size(),
+ column -> Math.min(schema.valueGenerators.staticPopulation(column), UNIQUE_CELL_VALUES),
+ eqOnlyStaticColumns::contains);
+
+ Integer pk = pkGen.generate(rng);
+
+ IdxRelation[] ckRelations =
+ HistoryBuilderHelper.generateClusteringRelations(rng,
+ schema.clusteringKeys.size(),
+ ckGen,
+ eqOnlyClusteringColumns).toArray(new IdxRelation[0]);
+
+ IdxRelation[] regularRelationsArray = regularRelations.toArray(new IdxRelation[regularRelations.size()]);
+ IdxRelation[] staticRelationsArray = staticRelations.toArray(new IdxRelation[staticRelations.size()]);
+
+ history.select(pk, ckRelations, regularRelationsArray, staticRelationsArray);
+ }
+ }
+ }
+ }
+
+ protected Generator<SchemaSpec> schemaGenerator(boolean disableReadRepair)
+ {
+ SchemaSpec.OptionsBuilder builder = SchemaSpec.optionsBuilder().disableReadRepair(disableReadRepair)
+ .compactionStrategy("LeveledCompactionStrategy");
+ return SchemaGenerators.schemaSpecGen(KEYSPACE, "basic_sai", MAX_PARTITION_SIZE, builder);
+ }
+
+ protected void flush(SchemaSpec schema)
+ {
+ cluster.get(1).nodetool("flush", schema.keyspace, schema.table);
+ }
+
+ protected void compact(SchemaSpec schema)
+ {
+ cluster.get(1).nodetool("compact", schema.keyspace);
+ }
+
+ protected void repair(SchemaSpec schema)
+ {
+ // Repair is nonsensical for a single node, but a repair does flush first, so do that at least.
+ cluster.get(1).nodetool("flush", schema.keyspace, schema.table);
+ }
+
+ protected void waitForIndexesQueryable(SchemaSpec schema) {}
+
+ public static Consumer<IInstanceConfig> defaultConfig()
+ {
+ return (cfg) -> {
+ cfg.set("row_cache_size", "50MiB")
+ .set("index_summary_capacity", "50MiB")
+ .set("counter_cache_size", "50MiB")
+ .set("key_cache_size", "50MiB")
+ .set("file_cache_size", "50MiB")
+ .set("index_summary_capacity", "50MiB")
+ .set("memtable_heap_space", "128MiB")
+ .set("memtable_offheap_space", "128MiB")
+ .set("memtable_flush_writers", 1)
+ .set("concurrent_compactors", 1)
+ .set("concurrent_reads", 5)
+ .set("concurrent_writes", 5)
+ .set("compaction_throughput_mb_per_sec", 10)
+ .set("hinted_handoff_enabled", false);
+ };
+ }
+
+ protected InJvmDTestVisitExecutor.ConsistencyLevelSelector consistencyLevelSelector()
+ {
+ return visit -> {
+ if (visit.selectOnly)
+ return ConsistencyLevel.ALL;
+
+ // The goal here is to make replicas as out of date as possible, modulo the efforts of repair
+ // and read-repair in the test itself.
+ return ConsistencyLevel.NODE_LOCAL;
+
+ };
+ }
+
+ protected InJvmDTestVisitExecutor.PageSizeSelector pageSizeSelector(EntropySource rng)
+ {
+ // Chosing a fetch size has implications for how well this test will excercise paging, short-read protection, and
+ // other important parts of the distributed query apparatus. This should be set low enough to ensure a significant
+ // number of queries during validation page, but not too low that more expesive queries time out and fail the test.
+ return lts -> rng.nextInt(1, 20);
+ }
+}
diff --cc test/unit/org/apache/cassandra/cql3/KnownIssue.java
index 37ef255e54,0000000000..a1924a91f9
mode 100644,000000..100644
--- a/test/unit/org/apache/cassandra/cql3/KnownIssue.java
+++ b/test/unit/org/apache/cassandra/cql3/KnownIssue.java
@@@ -1,53 -1,0 +1,53 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.cql3;
+
+import java.util.EnumSet;
+
+/**
+ * In order to have a clean CI some known issues must be excluded from some tests until those issues are addressed.
+ *
+ * This type exists to make it easier to descover known issues and places in the code that account for them
+ */
+public enum KnownIssue
+{
+ BETWEEN_START_LARGER_THAN_END("https://issues.apache.org/jira/browse/CASSANDRA-20154",
+ "BETWEEN is matching values when start > end, which should never return anything"),
+ SAI_INET_MIXED("https://issues.apache.org/jira/browse/CASSANDRA-19492",
+ "SAI converts ipv4 to ipv6 to simplify the index, this causes issues with range search as it starts to mix the values, which isn't always desirable or intuative"),
+ CUSTOM_INDEX_MAX_COLUMN_48("https://issues.apache.org/jira/browse/CASSANDRA-19897",
+ "Columns can be up to 50 chars, but CREATE CUSTOM INDEX only allows up to 48"),
- AF_MULTI_NODE_AND_NODE_LOCAL_WRITES("https://issues.apache.org/jira/browse/CASSANDRA-20243",
- "When writes are done at NODE_LOCAL and the select is ALL, AF should be able to return the correct data but it doesn't"),
+ SHORT_AND_VARINT_GET_INT_FUNCTIONS("https://issues.apache.org/jira/browse/CASSANDRA-19874",
+ "Function inference maybe unable to infer the correct function or chooses one for a smaller type"),
+ SAI_EMPTY_TYPE("ML: Meaningless emptiness and filtering",
- "Some types allow empty bytes, but define them as meaningless. AF can be used to query them using <, <=, and =; but SAI can not")
++ "Some types allow empty bytes, but define them as meaningless. AF can be used to query them using <, <=, and =; but SAI can not"),
++ AF_MULTI_NODE_MULTI_COLUMN_AND_NODE_LOCAL_WRITES("https://issues.apache.org/jira/browse/CASSANDRA-19007",
++ "When doing multi node/multi column queries, AF can miss data when the nodes are not in-sync"),
+ ;
+
+ KnownIssue(String url, String description)
+ {
+ // don't actually care to save the values, just there to force documentation
+ }
+
+ public static EnumSet<KnownIssue> ignoreAll()
+ {
+ return EnumSet.allOf(KnownIssue.class);
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org