You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "smiklosovic (via GitHub)" <gi...@apache.org> on 2024/01/05 15:01:03 UTC
[PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
smiklosovic opened a new pull request, #3029:
URL: https://github.com/apache/cassandra/pull/3029
Thanks for sending a pull request! Here are some tips if you're new here:
* Ensure you have added or run the [appropriate tests](https://cassandra.apache.org/_/development/testing.html) for your PR.
* Be sure to keep the PR description updated to reflect all changes.
* Write your PR title to summarize what this PR proposes.
* If possible, provide a concise example to reproduce the issue for a faster review.
* Read our [contributor guidelines](https://cassandra.apache.org/_/development/index.html)
* If you're making a documentation change, see our [guide to documentation contribution](https://cassandra.apache.org/_/development/documentation.html)
Commit messages should follow the following format:
```
<One sentence description, usually Jira title or CHANGES.txt summary>
<Optional lengthier description (context on patch)>
patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####
Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>
```
The [Cassandra Jira](https://issues.apache.org/jira/projects/CASSANDRA/issues/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446970126
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
for `--skip-index-validation`, I do not like this one because if you notice what other flags are, there is prefix `--no` - `--no-verify`, `--no-tokens`, `--no-invalidate-caches` - there is no `--skip...`. We might do `--no-index-validation`, that would work, but not `--skip-index-validation`.
When it comes to `--require-index-component`, I do not like the fact that this does not say what happens when they are required and they are not present. To prepend it with `--fail` is pretty obvious to me. Anyway, one can conclude that if something is required and it is not there, it will probably complain and fail eventually ... yeah, I think this could fly (and the fact it will fail is also mentioned in the description field).
flags are now `-ri` and `-niv` as in `--require-index-components` and `--no-index-validation`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445857974
##########
src/java/org/apache/cassandra/cql3/statements/schema/CreateIndexStatement.java:
##########
@@ -80,11 +80,11 @@ public final class CreateIndexStatement extends AlterSchemaStatement
public static final String KEYSPACE_DOES_NOT_MATCH_INDEX = "Keyspace name '%s' doesn't match index name '%s'";
public static final String MUST_SPECIFY_INDEX_IMPLEMENTATION = "Must specify index implementation via USING";
- private final String indexName;
- private final String tableName;
- private final List<IndexTarget.Raw> rawIndexTargets;
- private final IndexAttributes attrs;
- private final boolean ifNotExists;
+ public final String indexName;
Review Comment:
I need to have this public in `CQLSSTableWriter.createIndexes`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446768392
##########
src/java/org/apache/cassandra/index/sai/StorageAttachedIndexGroup.java:
##########
@@ -353,7 +353,7 @@ public boolean validateSSTableAttachedIndexes(Collection<SSTableReader> sstables
if (indexDescriptor.isPerSSTableIndexBuildComplete())
{
- indexDescriptor.checksumPerSSTableComponents();
+ indexDescriptor.checksumPerSSTableComponents(validateChecksum);
for (StorageAttachedIndex index : indexes)
{
Review Comment:
We should pass `validateChecksum` to `checksumPerIndexComponents()` (which should probably be renamed or merged w/ the other validation method, see https://github.com/apache/cassandra/pull/3029/files#r1446766349) as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446638628
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
+ description = "Fail import when SSTables do not contain SAI components")
+ private boolean failOnMissingIndex = false;
+
+ @Option(title = "no_validate_index_checksum",
+ name = {"-i", "--no-validate-index-checksum"},
+ description = "Do not validate checksum of indexes")
Review Comment:
```suggestion
description = "Skip SSTable-attached index checksum validation")
```
Works w/ Doug's naming suggestion and is a tiny bit more specific (as I'm guessing this is only ever going to apply to SAI-like indexes that are SSTable-attached)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "JeetKunDoug (via GitHub)" <gi...@apache.org>.
JeetKunDoug commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446516235
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +99,19 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.verifySSTables || options.verifyTokens)
+ verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
+ {
+ IndexDescriptor indexDescriptor = IndexDescriptor.create(descriptor,
+ cfs.getPartitioner(),
Review Comment:
Agreed - hard-coding the partitioner here would be bad.
##########
src/java/org/apache/cassandra/db/ColumnFamilyStore.java:
##########
@@ -859,6 +859,22 @@ public void loadNewSSTables()
* #{@inheritDoc}
*/
public synchronized List<String> importNewSSTables(Set<String> srcPaths, boolean resetLevel, boolean clearRepaired, boolean verifySSTables, boolean verifyTokens, boolean invalidateCaches, boolean extendedVerify, boolean copyData)
+ {
+ return importNewSSTables(srcPaths, resetLevel, clearRepaired, verifySSTables, verifyTokens, invalidateCaches, extendedVerify, false, false, true);
Review Comment:
This lost the `copyData` value - I wonder if it wouldn't be better to just duplicate the building of the import options and call to `sstableImporter#importNewSSTables` just to get rid of the huge number of parameters that can be easily mixed up here, since almost all of them are booleans?
If we do this, each method should simply set the Options it needs and leave any option that isn't specified in the method signature alone, so the default from `SSTableImporter#Options` is used.
##########
src/java/org/apache/cassandra/db/ColumnFamilyStore.java:
##########
@@ -859,6 +859,22 @@ public void loadNewSSTables()
* #{@inheritDoc}
*/
public synchronized List<String> importNewSSTables(Set<String> srcPaths, boolean resetLevel, boolean clearRepaired, boolean verifySSTables, boolean verifyTokens, boolean invalidateCaches, boolean extendedVerify, boolean copyData)
Review Comment:
Does this need to be `synchronized`? The `sstableImporter#importNewSSTables` call already synchronizes on the importer itself, and none of the other methods are synchronized, so I think we can remove this...
Also, a nit - all of them should have `@Override` annotations per the style guide.
##########
src/java/org/apache/cassandra/tools/NodeProbe.java:
##########
@@ -1544,9 +1544,15 @@ public void loadNewSSTables(String ksName, String cfName)
ssProxy.loadNewSSTables(ksName, cfName);
}
- public List<String> importNewSSTables(String ksName, String cfName, Set<String> srcPaths, boolean resetLevel, boolean clearRepaired, boolean verifySSTables, boolean verifyTokens, boolean invalidateCaches, boolean extendedVerify, boolean copyData)
- {
- return getCfsProxy(ksName, cfName).importNewSSTables(srcPaths, resetLevel, clearRepaired, verifySSTables, verifyTokens, invalidateCaches, extendedVerify, copyData);
+ public List<String> importNewSSTables(String ksName, String cfName, Set<String> srcPaths, boolean resetLevel,
Review Comment:
So your comment says you added a new one, but it looks like you modified the old one... I don't know if there's anything other than the `Import` command that actually uses it, but wanted to check if it was intentional that this comment doesn't match the code - or did you mean you made a new one in `ColumnFamilyStore/ColumnFamilyStoreMBean` which would also make sense, and which you did... so maybe I just misunderstood this PR comment?
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
What do you think about `-ri`/`--require-index-component` and `-siv`/`--skip-index-validation`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446745403
##########
test/unit/org/apache/cassandra/db/ImportTest.java:
##########
@@ -710,6 +710,154 @@ public void importExoticTableNamesTest() throws Throwable
}
}
+ @Test
+ public void mustNotFailOnBuiltSAIIndexesWhenRequiredTest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .build();
+ assertTrue(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_test"));
+ }
+ }
+
+ @Test
+ public void mustNotFailOnMissingSAIIndexWhenSAIDoNotExistTest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_less_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_less_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_less_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_less_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ // this does not mean anything
+ // becase our table does not have any SAI index
+ .failOnMissingIndex(true)
+ .build();
+ assertTrue(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_less_test"));
+ }
+ }
+
+ @Test
+ public void mustFailOnMissingSAIWhenRequiredTest() throws Throwable
+ {
+ File backupDir = null;
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ // create index and load sstables, they will be without indexes (because we created
+ // data when index was not created yet)
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .build();
+ assertFalse(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ if (backupDir != null)
+ {
+ backupDir.deleteRecursive();
+ backupDir.parent().deleteRecursive();
+ }
+
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_test"));
+ }
+ }
+
+ @Test
+ public void skipIndexChecksumOnSAITest() throws Throwable
Review Comment:
I guess we could just corrupt one of the column index files and then try to issue a query that matches all 10 rows after import?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1449212939
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +101,39 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
+ {
+ Index.Group saiIndexGroup = cfs.indexManager.getIndexGroup(StorageAttachedIndexGroup.GROUP_KEY);
+ if (saiIndexGroup != null)
+ {
+ IndexDescriptor indexDescriptor = IndexDescriptor.create(descriptor,
+ cfs.getPartitioner(),
+ cfs.metadata().comparator);
+
+ String keyspace = cfs.getKeyspaceName();
+ String table = cfs.getTableName();
+
+ if (!indexDescriptor.isPerSSTableIndexBuildComplete())
+ throw new IllegalStateException(String.format("Missing SAI index to import for SSTable %s on %s.%s",
+ indexDescriptor.sstableDescriptor.toString(),
+ keyspace,
+ table));
+
+ for (Index index : saiIndexGroup.getIndexes())
+ {
+ IndexIdentifier indexIdentifier = new IndexIdentifier(keyspace, table, index.getIndexMetadata().name);
+ if (!indexDescriptor.isPerColumnIndexBuildComplete(indexIdentifier))
+ throw new IllegalStateException(String.format("Missing SAI index to import for index %s on %s.%s",
+ index.getIndexMetadata().name,
+ keyspace,
+ table));
Review Comment:
nit: You could use the `logMessage()` helpers from `IndexDescriptor` and `IndexIdentifier` for the exceptions above, but not critical. It would give you the standardized prefix (which we did to make it easier to search for all SAI-related logging) but that's it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#issuecomment-1894545183
@maedhroz yes, in 3046 there are refactored tests, your commit and one extra commit fixing the issue we talked about
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1450921348
##########
src/java/org/apache/cassandra/db/ColumnFamilyStore.java:
##########
@@ -200,22 +200,36 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
are finished. By having flushExecutor size the same size as each of the perDiskflushExecutors we make sure we can
have that many flushes going at the same time.
*/
- private static final ExecutorPlus flushExecutor = executorFactory()
+ private static ExecutorPlus flushExecutor;
+
+ // post-flush executor is single threaded to provide guarantee that any flush Future on a CF will never return until prior flushes have completed
+ private static ExecutorPlus postFlushExecutor;
+
+ private static ExecutorPlus reclaimExecutor;
+
+ private static PerDiskFlushExecutors perDiskflushExecutors;
+
+ static
+ {
+ if (!DatabaseDescriptor.isClientOrToolInitialized())
Review Comment:
I kept these final in https://github.com/apache/cassandra/commit/fecac728ec945e9b376a5fb2c4af54f337d744cf. Even if you don't take that patch wholesale, we should keep that part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1450391868
##########
src/java/org/apache/cassandra/db/memtable/AbstractAllocatorMemtable.java:
##########
@@ -78,9 +79,17 @@ public abstract class AbstractAllocatorMemtable extends AbstractMemtableWithComm
static MemtablePool createMemtableAllocatorPool()
{
Config.MemtableAllocationType allocationType = DatabaseDescriptor.getMemtableAllocationType();
- long heapLimit = DatabaseDescriptor.getMemtableHeapSpaceInMiB() << 20;
- long offHeapLimit = DatabaseDescriptor.getMemtableOffheapSpaceInMiB() << 20;
- float memtableCleanupThreshold = DatabaseDescriptor.getMemtableCleanupThreshold();
+
+ long heapLimit = new DataStorageSpec.IntMebibytesBound((int) (Runtime.getRuntime().maxMemory() / (4 * 1048576))).toMebibytes() << 20;
Review Comment:
I took this from the logic when `DatabaseDescriptor` is normally initialized in daemon mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic merged PR #3029:
URL: https://github.com/apache/cassandra/pull/3029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445864116
##########
src/java/org/apache/cassandra/tools/NodeProbe.java:
##########
@@ -1544,9 +1544,15 @@ public void loadNewSSTables(String ksName, String cfName)
ssProxy.loadNewSSTables(ksName, cfName);
}
- public List<String> importNewSSTables(String ksName, String cfName, Set<String> srcPaths, boolean resetLevel, boolean clearRepaired, boolean verifySSTables, boolean verifyTokens, boolean invalidateCaches, boolean extendedVerify, boolean copyData)
- {
- return getCfsProxy(ksName, cfName).importNewSSTables(srcPaths, resetLevel, clearRepaired, verifySSTables, verifyTokens, invalidateCaches, extendedVerify, copyData);
+ public List<String> importNewSSTables(String ksName, String cfName, Set<String> srcPaths, boolean resetLevel,
Review Comment:
I created new method instead of modifying the old one to be backward compatible / not break stuff for people using the older version already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445878502
##########
src/java/org/apache/cassandra/db/ColumnFamilyStoreMBean.java:
##########
@@ -219,6 +221,33 @@ public List<String> importNewSSTables(Set<String> srcPaths,
boolean extendedVerify,
boolean copyData);
+ /**
+ * Load new sstables from the given directory
+ *
+ * @param srcPaths the path to the new sstables - if it is an empty set, the data directories will be scanned
+ * @param resetLevel if the level should be reset to 0 on the new sstables
+ * @param clearRepaired if repaired info should be wiped from the new sstables
+ * @param verifySSTables if the new sstables should be verified that they are not corrupt
+ * @param verifyTokens if the tokens in the new sstables should be verified that they are owned by the current node
+ * @param invalidateCaches if row cache should be invalidated for the keys in the new sstables
+ * @param extendedVerify if we should run an extended verify checking all values in the new sstables
+ * @param copyData if we should copy data from source paths instead of moving them
+ * @param validateIndexChecksum if we should also validate checksum for SAI indexes
+ * @param failOnMissingIndex if loading should fail when SSTables do not contain built SAI indexes too
+ *
+ * @return list of failed import directories
+ */
+ public List<String> importNewSSTables(Set<String> srcPaths,
Review Comment:
I added a new method and deprecated the old one to preserve backward compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446657385
##########
src/java/org/apache/cassandra/db/ColumnFamilyStore.java:
##########
@@ -859,6 +859,22 @@ public void loadNewSSTables()
* #{@inheritDoc}
*/
public synchronized List<String> importNewSSTables(Set<String> srcPaths, boolean resetLevel, boolean clearRepaired, boolean verifySSTables, boolean verifyTokens, boolean invalidateCaches, boolean extendedVerify, boolean copyData)
Review Comment:
If we want to keep our behavior in line with the pre-patch code, we would move the `synchronized` to the overload below on line 874. However, as Doug mentions, the importer is already synchronizing on its `importNewSSTables()` method. We can just remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446970126
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
for `--skip-index-validation`, I do not like this one because if you notice what other flags are, there is prefix `--no` - `--no-verify`, `--no-tokens`, `--no-invalidate-caches` - there is no `--skip...`. We might do `--no-index-validation`, that would work, but no `--skip-index-validation`.
When it comes to `--require-index-component`, I do not like the fact that this does not say what happens when they are required and they are not present. To prepend it with `--fail` is pretty obvious to me. Anyway, one can conclude that if something is required and it is not there, it will probably complain and fail eventually ... yeah, I think this could fly (and the fact it will fail is also mentioned in the description field).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445860406
##########
src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java:
##########
@@ -618,6 +654,11 @@ public CQLSSTableWriter build()
Types types = createTypes(keyspaceName);
Schema.instance.transform(SchemaTransformations.addTypes(types, true));
tableMetadata = createTable(types);
+
+ Indexes indexes = createIndexes(tableMetadata);
Review Comment:
I need non-null `tableMetadata` first to create `Indexes`. So then I have to unbuild that to set them to these metadata afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446637490
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
+1 to `-siv`/`--skip-index-validation`
Still thinking about the other...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446659680
##########
src/java/org/apache/cassandra/db/ColumnFamilyStore.java:
##########
@@ -859,6 +859,22 @@ public void loadNewSSTables()
* #{@inheritDoc}
*/
public synchronized List<String> importNewSSTables(Set<String> srcPaths, boolean resetLevel, boolean clearRepaired, boolean verifySSTables, boolean verifyTokens, boolean invalidateCaches, boolean extendedVerify, boolean copyData)
+ {
+ return importNewSSTables(srcPaths, resetLevel, clearRepaired, verifySSTables, verifyTokens, invalidateCaches, extendedVerify, false, false, true);
Review Comment:
Yeah, +100 to just building a `SSTableImporter.Options` as early as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446766509
##########
src/java/org/apache/cassandra/index/sai/disk/format/IndexDescriptor.java:
##########
@@ -409,9 +409,9 @@ public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentif
}
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public void checksumPerSSTableComponents()
+ public void checksumPerSSTableComponents(boolean validateChecksum)
Review Comment:
CC @pkolaczk
(Making sure I'm not missing something.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446970126
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
for `--skip-index-validation`, I do not like this one because if you notice what other flags are, there is prefix `--no` - `--no-verify`, `--no-tokens`, `--no-invalidate-caches` - there is no `--skip...`. We might do `--no-index-validation`, that would work, but no `--skip-index-validation`.
When it comes to `--require-index-component`, I do not like the fact that this does not say what happens when they are required and they are not present. To prepend it with `--fail` is pretty obvious to me. Anyway, one can conclude that if something is required and it is not there, it will probably complain and fail eventually ... yeah, I think this could fly (and the fact it will fail is also mentioned in the description field).
flags are now "-ri" and "-niv" as in "--require-index-components" and "--no-index-validation".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1447339733
##########
src/java/org/apache/cassandra/index/sai/disk/format/IndexDescriptor.java:
##########
@@ -409,9 +409,9 @@ public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentif
}
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public void checksumPerSSTableComponents()
+ public void checksumPerSSTableComponents(boolean validateChecksum)
Review Comment:
I have a hard time to understand what should be done here ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1449250435
##########
src/java/org/apache/cassandra/index/sai/disk/format/IndexDescriptor.java:
##########
@@ -384,34 +384,30 @@ public boolean validatePerIndexComponents(IndexTermType indexTermType, IndexIden
}
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public boolean validatePerSSTableComponents(IndexValidation validation)
+ public boolean validatePerSSTableComponents(IndexValidation validation, boolean rethrow)
{
if (validation == IndexValidation.NONE)
return true;
- logger.info(logMessage("Validating per-sstable index components using mode " + validation));
- boolean checksum = validation == IndexValidation.CHECKSUM;
+ logger.info(logMessage("Validating per-sstable index components for SSTable {} using mode {}"), sstableDescriptor.toString(), validation);
try
{
- version.onDiskFormat().validatePerSSTableIndexComponents(this, checksum);
+ version.onDiskFormat().validatePerSSTableIndexComponents(this, validation == IndexValidation.CHECKSUM);
return true;
}
catch (UncheckedIOException e)
{
- return false;
+ if (rethrow)
+ throw e;
+ else
+ return false;
}
}
- public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentifier indexIdentifier)
- {
- version.onDiskFormat().validatePerColumnIndexComponents(this, indexTermType, indexIdentifier, true);
- }
-
- @SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public void checksumPerSSTableComponents()
+ public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentifier indexIdentifier, boolean validateChecksum)
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1447102510
##########
src/java/org/apache/cassandra/index/sai/disk/format/IndexDescriptor.java:
##########
@@ -409,9 +409,9 @@ public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentif
}
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public void checksumPerSSTableComponents()
+ public void checksumPerSSTableComponents(boolean validateChecksum)
Review Comment:
@maedhroz `logMessage` in `validatePerSSTableComponents()` does print keyspace and table at least if you look into its implementation:
public String logMessage(String message)
{
// Index names are unique only within a keyspace.
return String.format("[%s.%s.*] %s",
sstableDescriptor.ksname,
sstableDescriptor.cfname,
message);
}
This however does not print Descrpitor's
@Override
public String toString()
{
return baseFile().absolutePath();
}
We could include it into `logMessage` argument here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1447377885
##########
src/java/org/apache/cassandra/index/sai/disk/format/IndexDescriptor.java:
##########
@@ -409,9 +409,9 @@ public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentif
}
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public void checksumPerSSTableComponents()
+ public void checksumPerSSTableComponents(boolean validateChecksum)
Review Comment:
I simplified that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445889120
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +99,19 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex && cfs.indexManager.getIndexGroup(StorageAttachedIndexGroup.GROUP_KEY) != null)
Review Comment:
we also enter this `if` only if there are some SAI indexes for this table present ... obviously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445890908
##########
test/unit/org/apache/cassandra/db/ImportTest.java:
##########
@@ -710,6 +710,154 @@ public void importExoticTableNamesTest() throws Throwable
}
}
+ @Test
+ public void mustNotFailOnBuiltSAIIndexesWhenRequiredTest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .build();
+ assertTrue(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_test"));
+ }
+ }
+
+ @Test
+ public void mustNotFailOnMissingSAIIndexWhenSAIDoNotExistTest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_less_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_less_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_less_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_less_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ // this does not mean anything
+ // becase our table does not have any SAI index
+ .failOnMissingIndex(true)
+ .build();
+ assertTrue(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_less_test"));
+ }
+ }
+
+ @Test
+ public void mustFailOnMissingSAIWhenRequiredTest() throws Throwable
+ {
+ File backupDir = null;
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ // create index and load sstables, they will be without indexes (because we created
+ // data when index was not created yet)
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .build();
+ assertFalse(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ if (backupDir != null)
+ {
+ backupDir.deleteRecursive();
+ backupDir.parent().deleteRecursive();
+ }
+
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_test"));
+ }
+ }
+
+ @Test
+ public void skipIndexChecksumOnSAITest() throws Throwable
Review Comment:
It is a little bit tricky this was actually skipped, all I am doing is to just be sure that it does not fail. It is easy to see that it is skipped by debugging this manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446641165
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
+ description = "Fail import when SSTables do not contain SAI components")
Review Comment:
```suggestion
description = "Require existing index components for SSTables w/ attached indexes")
```
Trying something that works w/ Doug's flag name in case we change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446970126
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
for `--skip-index-validation`, I do not like this one because if you notice what other flags there are, there is suffix `--no` - `--no-verify`, `--no-tokens`, `--no-invalidate-caches` - there is no `--skip...`. We might do `--no-index-validation`, that would work, but no `--skip-index-validation`.
When it comes to `--require-index-component`, I do not like the fact that this does not say what happens when they are required and they are not present. To prepend it with `--fail` is pretty obvious to me. Anyway, one can conclude that if something is required and it is not there, it will probably complain and fail eventually ... yeah, I think this could fly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446753408
##########
test/unit/org/apache/cassandra/db/ImportTest.java:
##########
@@ -710,6 +710,154 @@ public void importExoticTableNamesTest() throws Throwable
}
}
+ @Test
+ public void mustNotFailOnBuiltSAIIndexesWhenRequiredTest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .build();
+ assertTrue(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_test"));
+ }
+ }
+
+ @Test
+ public void mustNotFailOnMissingSAIIndexWhenSAIDoesNotExistTest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_less_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_less_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_less_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_less_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ // this does not mean anything
+ // because our table does not have any SAI index
+ .failOnMissingIndex(true)
+ .build();
+ assertTrue(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_less_test"));
+ }
+ }
+
+ @Test
+ public void mustFailOnMissingSAIWhenRequiredTest() throws Throwable
+ {
+ File backupDir = null;
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ // create index and load sstables, they will be without indexes (because we created
+ // data when index was not created yet)
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .build();
+ assertFalse(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ if (backupDir != null)
+ {
+ backupDir.deleteRecursive();
+ backupDir.parent().deleteRecursive();
+ }
+
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_test"));
+ }
+ }
+
+ @Test
+ public void skipIndexChecksumOnSAITest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .validateIndexChecksum(false)
Review Comment:
This reminded me of the edge case of empty SSTable indexes. We made sure those don't fail streaming/validation in CASSANDRA-19017, but it might be good to at least have a basic sanity check for the importer. Creating an empty index should be pretty easy. (see `StorageAttachedIndexDDLTest#verifyFlushAndCompactEmptyIndex()`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446970126
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
for `--skip-index-validation`, I do not like this one because if you notice what other flags there are, there is prefix `--no` - `--no-verify`, `--no-tokens`, `--no-invalidate-caches` - there is no `--skip...`. We might do `--no-index-validation`, that would work, but no `--skip-index-validation`.
When it comes to `--require-index-component`, I do not like the fact that this does not say what happens when they are required and they are not present. To prepend it with `--fail` is pretty obvious to me. Anyway, one can conclude that if something is required and it is not there, it will probably complain and fail eventually ... yeah, I think this could fly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446966346
##########
src/java/org/apache/cassandra/tools/NodeProbe.java:
##########
@@ -1544,9 +1544,15 @@ public void loadNewSSTables(String ksName, String cfName)
ssProxy.loadNewSSTables(ksName, cfName);
}
- public List<String> importNewSSTables(String ksName, String cfName, Set<String> srcPaths, boolean resetLevel, boolean clearRepaired, boolean verifySSTables, boolean verifyTokens, boolean invalidateCaches, boolean extendedVerify, boolean copyData)
- {
- return getCfsProxy(ksName, cfName).importNewSSTables(srcPaths, resetLevel, clearRepaired, verifySSTables, verifyTokens, invalidateCaches, extendedVerify, copyData);
+ public List<String> importNewSSTables(String ksName, String cfName, Set<String> srcPaths, boolean resetLevel,
Review Comment:
@JeetKunDoug correct, I created new one on ColumnFamilyStore/MBean but I changed the one on NodeProbe. That is just fine, because NodeProbe one is "internal" to Cassandra. Nobody depends on _NodeProbe_ across versions ... Does that make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446753408
##########
test/unit/org/apache/cassandra/db/ImportTest.java:
##########
@@ -710,6 +710,154 @@ public void importExoticTableNamesTest() throws Throwable
}
}
+ @Test
+ public void mustNotFailOnBuiltSAIIndexesWhenRequiredTest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .build();
+ assertTrue(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_test"));
+ }
+ }
+
+ @Test
+ public void mustNotFailOnMissingSAIIndexWhenSAIDoesNotExistTest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_less_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_less_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_less_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_less_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ // this does not mean anything
+ // because our table does not have any SAI index
+ .failOnMissingIndex(true)
+ .build();
+ assertTrue(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_less_test"));
+ }
+ }
+
+ @Test
+ public void mustFailOnMissingSAIWhenRequiredTest() throws Throwable
+ {
+ File backupDir = null;
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ // create index and load sstables, they will be without indexes (because we created
+ // data when index was not created yet)
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .build();
+ assertFalse(importer.importNewSSTables(options).isEmpty());
+ }
+ finally
+ {
+ if (backupDir != null)
+ {
+ backupDir.deleteRecursive();
+ backupDir.parent().deleteRecursive();
+ }
+
+ execute(String.format("DROP TABLE IF EXISTS %s.%s", KEYSPACE, "sai_test"));
+ }
+ }
+
+ @Test
+ public void skipIndexChecksumOnSAITest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .validateIndexChecksum(false)
Review Comment:
This reminded me of the edge case of empty SSTable indexes. We made sure those don't fail streaming in CASSANDRA-19017, but it might be good to at least have a basic sanity check for the importer. Creating an empty index should be pretty easy. (see `StorageAttachedIndexDDLTest#verifyFlushAndCompactEmptyIndex()`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1450393359
##########
src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java:
##########
@@ -645,14 +652,31 @@ public CQLSSTableWriter build()
KeyspaceMetadata ksm = Schema.instance.getKeyspaceMetadata(keyspaceName);
TableMetadata tableMetadata = ksm.tables.getNullable(schemaStatement.table());
+
+ ColumnFamilyStore cfs = null;
+
if (tableMetadata == null)
{
Types types = createTypes(keyspaceName);
Schema.instance.transform(SchemaTransformations.addTypes(types, true));
tableMetadata = createTable(types);
- if (!indexStatements.isEmpty())
- tableMetadata = applyIndexes(ksm.withSwapped(ksm.tables.with(tableMetadata)));
+ if (buildIndexes && !indexStatements.isEmpty())
+ {
+ cfs = new ColumnFamilyStore(Keyspace.mockKS(ksm),
+ tableMetadata.name,
+ () -> new SequenceBasedSSTableId(0),
+ TableMetadataRef.forOfflineTools(tableMetadata),
+ new Directories(tableMetadata, new Directories.DataDirectory[] {
+ new Directories.DataDirectory("/tmp/dirs")
Review Comment:
it really does not matter what we put there but it just cant be null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1450921622
##########
src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java:
##########
@@ -324,7 +324,8 @@ private static AbstractReplicationStrategy createInternal(String keyspaceName,
try
{
Constructor<? extends AbstractReplicationStrategy> constructor = strategyClass.getConstructor(parameterTypes);
- strategy = constructor.newInstance(keyspaceName, tokenMetadata, snitch, strategyOptions);
+ IEndpointSnitch endpointSnitch = snitch == null ? new SimpleSnitch() : snitch;
Review Comment:
Agreed. I did this in https://github.com/apache/cassandra/commit/fecac728ec945e9b376a5fb2c4af54f337d744cf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1450390385
##########
src/java/org/apache/cassandra/db/ColumnFamilyStore.java:
##########
@@ -200,22 +200,36 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean, Memtable.Owner
are finished. By having flushExecutor size the same size as each of the perDiskflushExecutors we make sure we can
have that many flushes going at the same time.
*/
- private static final ExecutorPlus flushExecutor = executorFactory()
+ private static ExecutorPlus flushExecutor;
+
+ // post-flush executor is single threaded to provide guarantee that any flush Future on a CF will never return until prior flushes have completed
+ private static ExecutorPlus postFlushExecutor;
+
+ private static ExecutorPlus reclaimExecutor;
+
+ private static PerDiskFlushExecutors perDiskflushExecutors;
+
+ static
+ {
+ if (!DatabaseDescriptor.isClientOrToolInitialized())
Review Comment:
this needs to be completely skipped in the client mode because DatabaseDescriptor is not initialized yet so it would fail to instantiate these executors because invalid values would be passed to that. Interestingly, the fact these fields are left to be null does not hurt anything because they are never accessed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#issuecomment-1894144652
This PR is replaced by https://github.com/apache/cassandra/pull/3046, yes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445853626
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +99,19 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.verifySSTables || options.verifyTokens)
+ verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
+ {
+ IndexDescriptor indexDescriptor = IndexDescriptor.create(descriptor,
+ cfs.getPartitioner(),
Review Comment:
keep in mind that for SAI, the only partitioner currently supported is Murmur3Partitioner and it needs to be explicitly set on `CQLSSTableWriter` by `withPartitioner` (visible from tests in `CQLSSTableWriterTest`.
Anyway, I still think that this is the correct way to do this. We should not hardcode `Murmur3Partitioner.instance` here even it is the only one which makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1449221930
##########
src/java/org/apache/cassandra/index/sai/disk/format/IndexDescriptor.java:
##########
@@ -384,34 +384,30 @@ public boolean validatePerIndexComponents(IndexTermType indexTermType, IndexIden
}
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public boolean validatePerSSTableComponents(IndexValidation validation)
+ public boolean validatePerSSTableComponents(IndexValidation validation, boolean rethrow)
{
if (validation == IndexValidation.NONE)
return true;
- logger.info(logMessage("Validating per-sstable index components using mode " + validation));
- boolean checksum = validation == IndexValidation.CHECKSUM;
+ logger.info(logMessage("Validating per-sstable index components for SSTable {} using mode {}"), sstableDescriptor.toString(), validation);
try
{
- version.onDiskFormat().validatePerSSTableIndexComponents(this, checksum);
+ version.onDiskFormat().validatePerSSTableIndexComponents(this, validation == IndexValidation.CHECKSUM);
return true;
}
catch (UncheckedIOException e)
{
- return false;
+ if (rethrow)
+ throw e;
+ else
+ return false;
}
}
- public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentifier indexIdentifier)
- {
- version.onDiskFormat().validatePerColumnIndexComponents(this, indexTermType, indexIdentifier, true);
- }
-
- @SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public void checksumPerSSTableComponents()
+ public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentifier indexIdentifier, boolean validateChecksum)
Review Comment:
We could just remove this method and modify `validatePerIndexComponents()` to take a `rethrow` argument (similar to `validatePerSSTableComponents()`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1449240486
##########
src/java/org/apache/cassandra/index/sai/StorageAttachedIndexGroup.java:
##########
@@ -353,12 +353,12 @@ public boolean validateSSTableAttachedIndexes(Collection<SSTableReader> sstables
if (indexDescriptor.isPerSSTableIndexBuildComplete())
{
- indexDescriptor.checksumPerSSTableComponents();
+ indexDescriptor.validatePerSSTableComponents(IndexValidation.CHECKSUM, true);
for (StorageAttachedIndex index : indexes)
{
if (indexDescriptor.isPerColumnIndexBuildComplete(index.identifier()))
- indexDescriptor.checksumPerIndexComponents(index.termType(), index.identifier());
+ indexDescriptor.checksumPerIndexComponents(index.termType(), index.identifier(), validateChecksum);
else if (throwOnIncomplete)
throw new IllegalStateException(indexDescriptor.logMessage("Incomplete per-column index build"));
Review Comment:
yeah I can fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1450393359
##########
src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java:
##########
@@ -645,14 +652,31 @@ public CQLSSTableWriter build()
KeyspaceMetadata ksm = Schema.instance.getKeyspaceMetadata(keyspaceName);
TableMetadata tableMetadata = ksm.tables.getNullable(schemaStatement.table());
+
+ ColumnFamilyStore cfs = null;
+
if (tableMetadata == null)
{
Types types = createTypes(keyspaceName);
Schema.instance.transform(SchemaTransformations.addTypes(types, true));
tableMetadata = createTable(types);
- if (!indexStatements.isEmpty())
- tableMetadata = applyIndexes(ksm.withSwapped(ksm.tables.with(tableMetadata)));
+ if (buildIndexes && !indexStatements.isEmpty())
+ {
+ cfs = new ColumnFamilyStore(Keyspace.mockKS(ksm),
+ tableMetadata.name,
+ () -> new SequenceBasedSSTableId(0),
+ TableMetadataRef.forOfflineTools(tableMetadata),
+ new Directories(tableMetadata, new Directories.DataDirectory[] {
+ new Directories.DataDirectory("/tmp/dirs")
Review Comment:
it really does not matter what we put there but it just cant be null, it will create these directories with keyspace and table dirs on disk.
##########
src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java:
##########
@@ -645,14 +652,31 @@ public CQLSSTableWriter build()
KeyspaceMetadata ksm = Schema.instance.getKeyspaceMetadata(keyspaceName);
TableMetadata tableMetadata = ksm.tables.getNullable(schemaStatement.table());
+
+ ColumnFamilyStore cfs = null;
+
if (tableMetadata == null)
{
Types types = createTypes(keyspaceName);
Schema.instance.transform(SchemaTransformations.addTypes(types, true));
tableMetadata = createTable(types);
- if (!indexStatements.isEmpty())
- tableMetadata = applyIndexes(ksm.withSwapped(ksm.tables.with(tableMetadata)));
+ if (buildIndexes && !indexStatements.isEmpty())
+ {
+ cfs = new ColumnFamilyStore(Keyspace.mockKS(ksm),
+ tableMetadata.name,
+ () -> new SequenceBasedSSTableId(0),
+ TableMetadataRef.forOfflineTools(tableMetadata),
+ new Directories(tableMetadata, new Directories.DataDirectory[] {
+ new Directories.DataDirectory("/tmp/dirs")
Review Comment:
it will create these directories with keyspace and table dirs on disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#issuecomment-1889940322
Pushed up an alternate approach to fixing the client mode issues here: https://github.com/apache/cassandra/commit/fecac728ec945e9b376a5fb2c4af54f337d744cf
Tried to minimize the changes to existing code by bypassing certain paths earlier. Also took some inspiration from what `StressCQLSSTableWriter` does in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446747489
##########
test/unit/org/apache/cassandra/io/sstable/CQLSSTableWriterTest.java:
##########
@@ -31,8 +31,11 @@
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
+import org.apache.cassandra.index.sai.disk.format.IndexDescriptor;
+import org.apache.cassandra.index.sai.utils.IndexIdentifier;
import org.apache.cassandra.io.sstable.format.big.BigFormat;
import org.apache.cassandra.io.util.File;
Review Comment:
nit: I guess we should just move all 4 of these to where they belong below...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446970126
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
for `--skip-index-validation`, I do not like this one because if you notice what other flags there are, there is prefix `--no` - `--no-verify`, `--no-tokens`, `--no-invalidate-caches` - there is no `--skip...`. We might do `--no-index-validation`, that would work, but no `--skip-index-validation`.
When it comes to `--require-index-component`, I do not like the fact that this does not say what happens when they are required and they are not present. To prepend it with `--fail` is pretty obvious to me. Anyway, one can conclude that if something is required and it is not there, it will probably complain and fail eventually ... yeah, I think this could fly (and the fact it will fail is also mentioned in the description field).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445860406
##########
src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java:
##########
@@ -618,6 +654,11 @@ public CQLSSTableWriter build()
Types types = createTypes(keyspaceName);
Schema.instance.transform(SchemaTransformations.addTypes(types, true));
tableMetadata = createTable(types);
+
+ Indexes indexes = createIndexes(tableMetadata);
Review Comment:
I need non-null `tableMetadata` first to create `Indexes`. So then I have to unbuild that to set indexes to metadata afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445866575
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +99,19 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.verifySSTables || options.verifyTokens)
+ verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
Review Comment:
I put it here to be still in the "verification" stage of the import so we can fail fast before any sstables are actually imported. Also, I moved it before other verification in order to fail as fast as possible because just checking if index was built is way faster than `verifySSTableForImport` afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "JeetKunDoug (via GitHub)" <gi...@apache.org>.
JeetKunDoug commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446441279
##########
src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java:
##########
@@ -648,6 +696,29 @@ private Types createTypes(String keyspace)
return builder.build();
}
+ private Indexes createIndexes(TableMetadata tableMetadata)
+ {
+ ClientState state = ClientState.forInternalCalls();
+ Indexes.Builder builder = Indexes.builder();
+
+ for (CreateIndexStatement.Raw createIndexStatement : indexStatements)
+ {
+ CreateIndexStatement statement = createIndexStatement.prepare(state);
+ statement.validate(ClientState.forInternalCalls());
+
+ List<IndexTarget> indexTargets = statement.rawIndexTargets.stream()
+ .map(raw -> raw.prepare(tableMetadata))
+ .collect(Collectors.toList());
+
+ builder.add(IndexMetadata.fromIndexTargets(indexTargets,
+ statement.indexName,
+ IndexMetadata.Kind.CUSTOM,
+ statement.attrs.getOptions()));
+ }
+
+ return builder.build();
+ }
Review Comment:
If we in stead do something like `applyIndexes` we can do this in a way I think is a bit cleaner, and will pick up any changes to the index statement logic, by using `CreateIndexStatement#apply`
```suggestion
/**
* Applies any provided index definitions to the target table
* @param ksm the KeyspaceMetadata object that has the table defined
* @return an updated TableMetadata instance with the indexe create statements applied
*/
private TableMetadata applyIndexes(KeyspaceMetadata ksm)
{
ClientState state = ClientState.forInternalCalls();
Keyspaces keyspaces = Keyspaces.of(ksm);
for (CreateIndexStatement.Raw statement: indexStatements)
{
keyspaces = statement.prepare(state).apply(keyspaces);
}
return keyspaces.get(ksm.name).get()
.tables.get(schemaStatement.table()).get();
}
```
And then, in `#build` we end up with something like this:
``` java
if (tableMetadata == null)
{
Types types = createTypes(keyspaceName);
Schema.instance.transform(SchemaTransformations.addTypes(types, true));
tableMetadata = createTable(types);
if (!indexStatements.isEmpty())
tableMetadata = applyIndexes(ksm.withSwapped(ksm.tables.with(tableMetadata)));
Schema.instance.transform(SchemaTransformations.addTable(tableMetadata, true));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446759775
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +99,19 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex && cfs.indexManager.getIndexGroup(StorageAttachedIndexGroup.GROUP_KEY) != null)
+ {
+ IndexDescriptor indexDescriptor = IndexDescriptor.create(descriptor,
+ cfs.getPartitioner(),
+ cfs.metadata().comparator);
+
+ if (!indexDescriptor.isPerSSTableIndexBuildComplete())
+ throw new IllegalStateException("Missing SAI indexes to import!");
Review Comment:
I think we need to check the per-column components here as well.
(See `isPerColumnIndexBuildComplete()`. We should be able to get the indexes and their identifiers from the group above.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446754095
##########
test/unit/org/apache/cassandra/io/sstable/CQLSSTableWriterTest.java:
##########
@@ -1309,6 +1311,86 @@ public void testWriteWithSortedAndMaxSize() throws Exception
}
}
+ @Test
+ public void testWriteWithSAI() throws Exception
+ {
+ String schema = "CREATE TABLE " + qualifiedTable + " ("
+ + " k int PRIMARY KEY,"
+ + " v1 text,"
+ + " v2 text )";
Review Comment:
nit: I'd suggest making v2 an int, just to make sure we're covering all the numeric index component validations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446970126
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
for `--skip-index-validation`, I do not like this one because if you notice what other flags are, there is prefix `--no` - `--no-verify`, `--no-tokens`, `--no-invalidate-caches` - there is no `--skip...`. We might do `--no-index-validation`, that would work, but no `--skip-index-validation`.
When it comes to `--require-index-component`, I do not like the fact that this does not say what happens when they are required and they are not present. To prepend it with `--fail` is pretty obvious to me. Anyway, one can conclude that if something is required and it is not there, it will probably complain and fail eventually ... yeah, I think this could fly (and the fact it will fail is also mentioned in the description field).
flags are now `-ri` and `-niv` as in "--require-index-components" and "--no-index-validation".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446970126
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
for `--skip-index-validation`, I do not like this one because if you notice what other flags are, there is prefix `--no` - `--no-verify`, `--no-tokens`, `--no-invalidate-caches` - there is no `--skip...`. We might do `--no-index-validation`, that would work, but no `--skip-index-validation`.
When it comes to `--require-index-component`, I do not like the fact that this does not say what happens when they are required and they are not present. To prepend it with `--fail` is pretty obvious to me. Anyway, one can conclude that if something is required and it is not there, it will probably complain and fail eventually ... yeah, I think this could fly (and the fact it will fail is also mentioned in the description field).
flags are now `-ri` and `-niv` as in `--require-index-components` and `--no-index-validation`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1450442646
##########
src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java:
##########
@@ -324,7 +324,8 @@ private static AbstractReplicationStrategy createInternal(String keyspaceName,
try
{
Constructor<? extends AbstractReplicationStrategy> constructor = strategyClass.getConstructor(parameterTypes);
- strategy = constructor.newInstance(keyspaceName, tokenMetadata, snitch, strategyOptions);
+ IEndpointSnitch endpointSnitch = snitch == null ? new SimpleSnitch() : snitch;
Review Comment:
This will propagate `null` snitch in client mode. I just have to put there something non-nullish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1450692832
##########
src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java:
##########
@@ -324,7 +324,8 @@ private static AbstractReplicationStrategy createInternal(String keyspaceName,
try
{
Constructor<? extends AbstractReplicationStrategy> constructor = strategyClass.getConstructor(parameterTypes);
- strategy = constructor.newInstance(keyspaceName, tokenMetadata, snitch, strategyOptions);
+ IEndpointSnitch endpointSnitch = snitch == null ? new SimpleSnitch() : snitch;
Review Comment:
to be extra safe, we could probably wrap this in DatabaseDescriptor.isClientOrToolInitialized() to set it to SimpleSnitch only in that case, otherwise we would just let this fail ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446746544
##########
test/unit/org/apache/cassandra/db/ImportTest.java:
##########
@@ -710,6 +710,154 @@ public void importExoticTableNamesTest() throws Throwable
}
}
+ @Test
+ public void mustNotFailOnBuiltSAIIndexesWhenRequiredTest() throws Throwable
+ {
+ try
+ {
+ schemaChange(String.format("CREATE TABLE %s.%s (id int primary key, d int)", KEYSPACE, "sai_test"));
+ schemaChange(String.format("CREATE INDEX idx1 ON %s.%s (d) USING 'sai'", KEYSPACE, "sai_test"));
+
+ for (int i = 0; i < 10; i++)
+ execute(String.format("INSERT INTO %s.%s (id, d) values (?, ?)", KEYSPACE, "sai_test"), i, i);
+
+ ColumnFamilyStore cfs = getColumnFamilyStore(KEYSPACE, "sai_test");
+ Util.flush(cfs);
+
+ Set<SSTableReader> sstables = cfs.getLiveSSTables();
+ cfs.clearUnsafe();
+
+ File backupDir = moveToBackupDir(sstables);
+
+ assertEquals(0, execute(String.format("SELECT * FROM %s.%s", KEYSPACE, "sai_test")).size());
+
+ SSTableImporter importer = new SSTableImporter(cfs);
+ SSTableImporter.Options options = SSTableImporter.Options.options(backupDir.toString())
+ .copyData(true)
+ .failOnMissingIndex(true)
+ .build();
+ assertTrue(importer.importNewSSTables(options).isEmpty());
Review Comment:
nit: Perhaps throw in an assertion on the actual index?
```
assertEquals(10, execute(String.format("SELECT * FROM %s.%s WHERE d >= 0", KEYSPACE, "sai_test")).size());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446766349
##########
src/java/org/apache/cassandra/index/sai/disk/format/IndexDescriptor.java:
##########
@@ -409,9 +409,9 @@ public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentif
}
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public void checksumPerSSTableComponents()
+ public void checksumPerSSTableComponents(boolean validateChecksum)
Review Comment:
Looking at this, we could probably merge w/ `validatePerSSTableComponents()` above, and we'd get some extra logging for free. We'd have to pass an extra boolean directing the method to throw or not on `UncheckedIOException` though.
Also, looking at `validatePerSSTableComponents()` itself, the log message isn't specific enough. It should probably be:
```
logger.info(logMessage("Validating per-sstable index components for SSTable {} using mode {}..."), sstableDescriptor, validation);
```
Otherwise, the log message will be the same for every SSTable we validate :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446766852
##########
src/java/org/apache/cassandra/index/sai/disk/format/IndexDescriptor.java:
##########
@@ -409,9 +409,9 @@ public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentif
}
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public void checksumPerSSTableComponents()
+ public void checksumPerSSTableComponents(boolean validateChecksum)
Review Comment:
Either way, probably can't continue to start the method name w/ `checksum*`...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445864664
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
I am open to different flags here, `-f` and `-i` just makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1449215366
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +101,39 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
+ {
+ Index.Group saiIndexGroup = cfs.indexManager.getIndexGroup(StorageAttachedIndexGroup.GROUP_KEY);
+ if (saiIndexGroup != null)
+ {
+ IndexDescriptor indexDescriptor = IndexDescriptor.create(descriptor,
+ cfs.getPartitioner(),
+ cfs.metadata().comparator);
+
+ String keyspace = cfs.getKeyspaceName();
+ String table = cfs.getTableName();
+
+ if (!indexDescriptor.isPerSSTableIndexBuildComplete())
+ throw new IllegalStateException(String.format("Missing SAI index to import for SSTable %s on %s.%s",
+ indexDescriptor.sstableDescriptor.toString(),
+ keyspace,
+ table));
+
+ for (Index index : saiIndexGroup.getIndexes())
+ {
+ IndexIdentifier indexIdentifier = new IndexIdentifier(keyspace, table, index.getIndexMetadata().name);
Review Comment:
nit: `StorageAttachedIndex` has an `IndexIdentifier` already if you want to cast the `Index`, which should be safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1449234740
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +101,39 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
+ {
+ Index.Group saiIndexGroup = cfs.indexManager.getIndexGroup(StorageAttachedIndexGroup.GROUP_KEY);
+ if (saiIndexGroup != null)
+ {
+ IndexDescriptor indexDescriptor = IndexDescriptor.create(descriptor,
+ cfs.getPartitioner(),
+ cfs.metadata().comparator);
+
+ String keyspace = cfs.getKeyspaceName();
+ String table = cfs.getTableName();
+
+ if (!indexDescriptor.isPerSSTableIndexBuildComplete())
+ throw new IllegalStateException(String.format("Missing SAI index to import for SSTable %s on %s.%s",
+ indexDescriptor.sstableDescriptor.toString(),
+ keyspace,
+ table));
+
+ for (Index index : saiIndexGroup.getIndexes())
+ {
+ IndexIdentifier indexIdentifier = new IndexIdentifier(keyspace, table, index.getIndexMetadata().name);
+ if (!indexDescriptor.isPerColumnIndexBuildComplete(indexIdentifier))
+ throw new IllegalStateException(String.format("Missing SAI index to import for index %s on %s.%s",
+ index.getIndexMetadata().name,
+ keyspace,
+ table));
Review Comment:
I dont like to mix sai specific logging stuff with sstable importer. Seems totally unrelated really ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1449242586
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +101,39 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
+ {
+ Index.Group saiIndexGroup = cfs.indexManager.getIndexGroup(StorageAttachedIndexGroup.GROUP_KEY);
+ if (saiIndexGroup != null)
+ {
+ IndexDescriptor indexDescriptor = IndexDescriptor.create(descriptor,
+ cfs.getPartitioner(),
+ cfs.metadata().comparator);
+
+ String keyspace = cfs.getKeyspaceName();
+ String table = cfs.getTableName();
+
+ if (!indexDescriptor.isPerSSTableIndexBuildComplete())
+ throw new IllegalStateException(String.format("Missing SAI index to import for SSTable %s on %s.%s",
+ indexDescriptor.sstableDescriptor.toString(),
+ keyspace,
+ table));
+
+ for (Index index : saiIndexGroup.getIndexes())
+ {
+ IndexIdentifier indexIdentifier = new IndexIdentifier(keyspace, table, index.getIndexMetadata().name);
Review Comment:
I rather leave this as it is ... casting this etc ... nah.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1449257001
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +101,39 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
+ {
+ Index.Group saiIndexGroup = cfs.indexManager.getIndexGroup(StorageAttachedIndexGroup.GROUP_KEY);
+ if (saiIndexGroup != null)
+ {
+ IndexDescriptor indexDescriptor = IndexDescriptor.create(descriptor,
+ cfs.getPartitioner(),
+ cfs.metadata().comparator);
+
+ String keyspace = cfs.getKeyspaceName();
+ String table = cfs.getTableName();
+
+ if (!indexDescriptor.isPerSSTableIndexBuildComplete())
+ throw new IllegalStateException(String.format("Missing SAI index to import for SSTable %s on %s.%s",
+ indexDescriptor.sstableDescriptor.toString(),
+ keyspace,
+ table));
+
+ for (Index index : saiIndexGroup.getIndexes())
+ {
+ IndexIdentifier indexIdentifier = new IndexIdentifier(keyspace, table, index.getIndexMetadata().name);
Review Comment:
Sure. `IndexIdentifier` itself could actually be an index implementation-agnostic class. There's nothing ASI-specific forcing it to be in `org.apache.cassandra.index.sai.utils`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446760976
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -481,6 +502,8 @@ public String toString()
", invalidateCaches=" + invalidateCaches +
", extendedVerify=" + extendedVerify +
", copyData= " + copyData +
+ ", failOnMissingIndex= " + failOnMissingIndex +
+ ", validateIndexChecksum= " + validateIndexChecksum +
Review Comment:
I'll come back and do a final review of all this stuff once we settle on naming ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1447102510
##########
src/java/org/apache/cassandra/index/sai/disk/format/IndexDescriptor.java:
##########
@@ -409,9 +409,9 @@ public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentif
}
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public void checksumPerSSTableComponents()
+ public void checksumPerSSTableComponents(boolean validateChecksum)
Review Comment:
@maedhroz `logMessage` in `validatePerSSTableComponents()` does print keyspace and table at least if you look into its implementation:
public String logMessage(String message)
{
// Index names are unique only within a keyspace.
return String.format("[%s.%s.*] %s",
sstableDescriptor.ksname,
sstableDescriptor.cfname,
message);
}
This however does not print Descrpitor's
@Override
public String toString()
{
return baseFile().absolutePath();
}
We could include it into `logMessage` argument here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446769373
##########
src/java/org/apache/cassandra/io/sstable/AbstractSSTableSimpleWriter.java:
##########
@@ -49,12 +51,14 @@ abstract class AbstractSSTableSimpleWriter implements Closeable
protected SSTableFormat<?, ?> format = DatabaseDescriptor.getSelectedSSTableFormat();
protected static final AtomicReference<SSTableId> id = new AtomicReference<>(SSTableIdFactory.instance.defaultBuilder().generator(Stream.empty()).get());
protected boolean makeRangeAware = false;
+ protected Collection<Index.Group> indexGroups;
Review Comment:
```suggestion
protected final Collection<Index.Group> indexGroups;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1447339733
##########
src/java/org/apache/cassandra/index/sai/disk/format/IndexDescriptor.java:
##########
@@ -409,9 +409,9 @@ public void checksumPerIndexComponents(IndexTermType indexTermType, IndexIdentif
}
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
- public void checksumPerSSTableComponents()
+ public void checksumPerSSTableComponents(boolean validateChecksum)
Review Comment:
I have a hard time to understand what should be done here ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445866575
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +99,19 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.verifySSTables || options.verifyTokens)
+ verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
Review Comment:
I put it here to be still in the "verification" stage of the import so we can fail fast before any sstables are actually imported. Also, I moved it before other verification in order to fail as fast as possible because just checking if index was built is way faster than `verifySSTableForImport` after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445866575
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +99,19 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.verifySSTables || options.verifyTokens)
+ verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
Review Comment:
I put it here to be still in the "verification" stage of the import so we can fail fast before any sstables are actually imported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1447339343
##########
src/java/org/apache/cassandra/index/sai/StorageAttachedIndexGroup.java:
##########
@@ -353,7 +353,7 @@ public boolean validateSSTableAttachedIndexes(Collection<SSTableReader> sstables
if (indexDescriptor.isPerSSTableIndexBuildComplete())
{
- indexDescriptor.checksumPerSSTableComponents();
+ indexDescriptor.checksumPerSSTableComponents(validateChecksum);
for (StorageAttachedIndex index : indexes)
{
Review Comment:
I passed it there too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446642239
##########
src/java/org/apache/cassandra/tools/nodetool/Import.java:
##########
@@ -80,6 +80,16 @@ public class Import extends NodeToolCmd
description = "Copy data from source directories instead of moving them")
private boolean copyData = false;
+ @Option(title = "fail_on_missing_index",
+ name = {"-f", "--fail-on-missing-index"},
Review Comment:
I'm fine w/ `--require-index-component` too, although I think it should be `--require-index-components`. (An SAI index is really a bag of separate components that enable related functionality, but perhaps I'm too in the weeds...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1445853626
##########
src/java/org/apache/cassandra/db/SSTableImporter.java:
##########
@@ -97,7 +99,19 @@ synchronized List<String> importNewSSTables(Options options)
try
{
abortIfDraining();
- verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.verifySSTables || options.verifyTokens)
+ verifySSTableForImport(descriptor, entry.getValue(), options.verifyTokens, options.verifySSTables, options.extendedVerify);
+
+ if (options.failOnMissingIndex)
+ {
+ IndexDescriptor indexDescriptor = IndexDescriptor.create(descriptor,
+ cfs.getPartitioner(),
Review Comment:
keep in mind that for SAI, the only partitioner currently supported is Murmur3Partitioner and it needs to be explicitly set on `CQLSSTableWriter` by `withPartitioner` (visible from tests in `CQLSSTableWriterTest`).
Anyway, I still think that this is the correct way to do this. We should not hardcode `Murmur3Partitioner.instance` here even it is the only one which makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446750391
##########
src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java:
##########
@@ -579,6 +599,18 @@ public Builder sorted()
return this;
}
+ /**
+ * Whether indexes should be built and serialized to disk along data. Defaults to true.
+ *
+ * @param buildIndexes true if indexes should be built, false otherwise
+ * @return this builder
+ */
+ public Builder withBuiltIndexes(boolean buildIndexes)
Review Comment:
```suggestion
public Builder withBuildIndexes(boolean buildIndexes)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "maedhroz (via GitHub)" <gi...@apache.org>.
maedhroz commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1449227920
##########
src/java/org/apache/cassandra/index/sai/StorageAttachedIndexGroup.java:
##########
@@ -353,12 +353,12 @@ public boolean validateSSTableAttachedIndexes(Collection<SSTableReader> sstables
if (indexDescriptor.isPerSSTableIndexBuildComplete())
{
- indexDescriptor.checksumPerSSTableComponents();
+ indexDescriptor.validatePerSSTableComponents(IndexValidation.CHECKSUM, true);
for (StorageAttachedIndex index : indexes)
{
if (indexDescriptor.isPerColumnIndexBuildComplete(index.identifier()))
- indexDescriptor.checksumPerIndexComponents(index.termType(), index.identifier());
+ indexDescriptor.checksumPerIndexComponents(index.termType(), index.identifier(), validateChecksum);
else if (throwOnIncomplete)
throw new IllegalStateException(indexDescriptor.logMessage("Incomplete per-column index build"));
Review Comment:
nit: I know this falls into the category of non-essential clean up, but I'm realizing that the two exception messages created in this method, much like the ones in `IndexDescriptor` before you fixed them, are missing SSTable info. They could probably be fixed in much the same way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-18714 CQLSSTableWriter writes SAI index [cassandra]
Posted by "JeetKunDoug (via GitHub)" <gi...@apache.org>.
JeetKunDoug commented on code in PR #3029:
URL: https://github.com/apache/cassandra/pull/3029#discussion_r1446287493
##########
src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java:
##########
@@ -457,6 +467,20 @@ public Builder forTable(String schema)
return this;
}
+ /**
+ * The schema (CREATE INDEX statement) for index to be created for the table. Only SAI indexes are supported.
+ *
+ * @param indexes CQL statements representing SAI indexes to be created.
+ * @return this builder
+ */
+ public Builder forIndex(String... indexes)
Review Comment:
NIT: I think this should be named `withIndexes` rather than `forIndex`
##########
src/java/org/apache/cassandra/cql3/statements/schema/CreateIndexStatement.java:
##########
@@ -80,11 +80,11 @@ public final class CreateIndexStatement extends AlterSchemaStatement
public static final String KEYSPACE_DOES_NOT_MATCH_INDEX = "Keyspace name '%s' doesn't match index name '%s'";
public static final String MUST_SPECIFY_INDEX_IMPLEMENTATION = "Must specify index implementation via USING";
- private final String indexName;
- private final String tableName;
- private final List<IndexTarget.Raw> rawIndexTargets;
- private final IndexAttributes attrs;
- private final boolean ifNotExists;
+ public final String indexName;
Review Comment:
with the `applyIndexes` change suggested in CQLSSTableWriter, you don't need these to be public any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org