You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "jacek-lewandowski (via GitHub)" <gi...@apache.org> on 2023/11/22 12:32:41 UTC
[PR] Cassandra 18963 4.0 [cassandra]
jacek-lewandowski opened a new pull request, #2928:
URL: https://github.com/apache/cassandra/pull/2928
Thanks for sending a pull request! Here are some tips if you're new here:
* Ensure you have added or run the [appropriate tests](https://cassandra.apache.org/_/development/testing.html) for your PR.
* Be sure to keep the PR description updated to reflect all changes.
* Write your PR title to summarize what this PR proposes.
* If possible, provide a concise example to reproduce the issue for a faster review.
* Read our [contributor guidelines](https://cassandra.apache.org/_/development/index.html)
* If you're making a documentation change, see our [guide to documentation contribution](https://cassandra.apache.org/_/development/documentation.html)
Commit messages should follow the following format:
```
<One sentence description, usually Jira title or CHANGES.txt summary>
<Optional lengthier description (context on patch)>
patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####
Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>
```
The [Cassandra Jira](https://issues.apache.org/jira/projects/CASSANDRA/issues/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] Cassandra 18963 4.0 [cassandra]
Posted by "jacek-lewandowski (via GitHub)" <gi...@apache.org>.
jacek-lewandowski commented on PR #2928:
URL: https://github.com/apache/cassandra/pull/2928#issuecomment-1822806195
https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/1099/workflows/d596e657-5543-46e3-a812-ab47c8f5177d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] Cassandra 18963 4.0 [cassandra]
Posted by "blerer (via GitHub)" <gi...@apache.org>.
blerer commented on code in PR #2928:
URL: https://github.com/apache/cassandra/pull/2928#discussion_r1402024636
##########
test/distributed/org/apache/cassandra/distributed/test/SystemKeyspacesDataLocationTest.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.test;
+
+import java.io.IOException;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.junit.Test;
+
+import org.apache.cassandra.cql3.QueryHandler;
+import org.apache.cassandra.cql3.QueryProcessor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.distributed.Cluster;
+import org.apache.cassandra.schema.Schema;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.schema.TableMetadata;
+
+import static com.datastax.driver.core.ParseUtils.doubleQuote;
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class SystemKeyspacesDataLocationTest extends TestBaseImpl
+{
+ @Test
+ public void preparedStatementsShouldNotBeMovedOnStartupTest() throws IOException, ExecutionException, InterruptedException
+ {
+ try (Cluster cluster = Cluster.build(1).withDataDirCount(3).start())
+ {
+ // to test the behaviour we need have sstables of prepared statements in more than one data directory
+ cluster.get(1).runOnInstance(() -> {
+ ColumnFamilyStore preparedStatementsCFS = ColumnFamilyStore.getIfExists(SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.PREPARED_STATEMENTS);
+ preparedStatementsCFS.disableAutoCompaction();
+ ColumnFamilyStore.getIfExists(SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.PREPARED_STATEMENTS).disableAutoCompaction();
+ for (String keyspaceName : SchemaConstants.LOCAL_SYSTEM_KEYSPACE_NAMES)
+ {
+ for (TableMetadata tableMetadata : Schema.instance.getKeyspaceMetadata(keyspaceName).tables)
+ {
+ String query = String.format("select * from %s.%s", keyspaceName, doubleQuote(tableMetadata.name));
+ QueryHandler.Prepared prepared = QueryProcessor.prepareInternal(query);
+ QueryProcessor.storePreparedStatement(query, keyspaceName, prepared);
+ preparedStatementsCFS.forceBlockingFlush();
+ }
+ }
Review Comment:
We could extract that code into a `populatePreparedStatementTable` method.
##########
test/distributed/org/apache/cassandra/distributed/test/SystemKeyspacesDataLocationTest.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.test;
+
+import java.io.IOException;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.junit.Test;
+
+import org.apache.cassandra.cql3.QueryHandler;
+import org.apache.cassandra.cql3.QueryProcessor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.distributed.Cluster;
+import org.apache.cassandra.schema.Schema;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.schema.TableMetadata;
+
+import static com.datastax.driver.core.ParseUtils.doubleQuote;
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class SystemKeyspacesDataLocationTest extends TestBaseImpl
+{
+ @Test
+ public void preparedStatementsShouldNotBeMovedOnStartupTest() throws IOException, ExecutionException, InterruptedException
+ {
+ try (Cluster cluster = Cluster.build(1).withDataDirCount(3).start())
+ {
+ // to test the behaviour we need have sstables of prepared statements in more than one data directory
+ cluster.get(1).runOnInstance(() -> {
+ ColumnFamilyStore preparedStatementsCFS = ColumnFamilyStore.getIfExists(SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.PREPARED_STATEMENTS);
+ preparedStatementsCFS.disableAutoCompaction();
+ ColumnFamilyStore.getIfExists(SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.PREPARED_STATEMENTS).disableAutoCompaction();
Review Comment:
Duplicate?
##########
src/java/org/apache/cassandra/service/CassandraDaemon.java:
##########
@@ -540,9 +538,7 @@ public void migrateSystemDataIfNeeded() throws IOException
try (Stream<Path> keyspaceChildren = Files.list(keyspaceDirectory))
{
Path[] tableDirectories = keyspaceChildren.filter(Files::isDirectory)
- .filter(p -> !SystemKeyspace.TABLES_SPLIT_ACROSS_MULTIPLE_DISKS
- .contains(p.getFileName()
- .toString()))
+ .filter(p -> SystemKeyspace.TABLES_SPLIT_ACROSS_MULTIPLE_DISKS.stream().noneMatch(t -> p.getFileName().toString().contains(t + '-')))
Review Comment:
Should we use `startWith` instead of `contains` ?
##########
test/distributed/org/apache/cassandra/distributed/test/SystemKeyspacesDataLocationTest.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.test;
+
+import java.io.IOException;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.junit.Test;
+
+import org.apache.cassandra.cql3.QueryHandler;
+import org.apache.cassandra.cql3.QueryProcessor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.distributed.Cluster;
+import org.apache.cassandra.schema.Schema;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.schema.TableMetadata;
+
+import static com.datastax.driver.core.ParseUtils.doubleQuote;
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class SystemKeyspacesDataLocationTest extends TestBaseImpl
+{
+ @Test
+ public void preparedStatementsShouldNotBeMovedOnStartupTest() throws IOException, ExecutionException, InterruptedException
+ {
+ try (Cluster cluster = Cluster.build(1).withDataDirCount(3).start())
+ {
+ // to test the behaviour we need have sstables of prepared statements in more than one data directory
+ cluster.get(1).runOnInstance(() -> {
+ ColumnFamilyStore preparedStatementsCFS = ColumnFamilyStore.getIfExists(SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.PREPARED_STATEMENTS);
+ preparedStatementsCFS.disableAutoCompaction();
+ ColumnFamilyStore.getIfExists(SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.PREPARED_STATEMENTS).disableAutoCompaction();
+ for (String keyspaceName : SchemaConstants.LOCAL_SYSTEM_KEYSPACE_NAMES)
+ {
+ for (TableMetadata tableMetadata : Schema.instance.getKeyspaceMetadata(keyspaceName).tables)
+ {
+ String query = String.format("select * from %s.%s", keyspaceName, doubleQuote(tableMetadata.name));
+ QueryHandler.Prepared prepared = QueryProcessor.prepareInternal(query);
+ QueryProcessor.storePreparedStatement(query, keyspaceName, prepared);
+ preparedStatementsCFS.forceBlockingFlush();
+ }
+ }
+ });
+ Function<Cluster, Set<String>> preparedStatementsDataLocations = c -> c.get(1).callOnInstance(() -> ColumnFamilyStore.getIfExists(SchemaConstants.SYSTEM_KEYSPACE_NAME, SystemKeyspace.PREPARED_STATEMENTS)
+ .getLiveSSTables().stream()
+ .map(sstr -> sstr.descriptor.directory.getParentFile().getParentFile().toString())
+ .collect(Collectors.toSet()));
Review Comment:
Using a method called `getPreparedStatementsDataLocations(Cluster)` would be better fro readability than a function.
It would be also good to document: ` sstr.descriptor.directory.getParentFile().getParentFile().toString()` so that people understand the logic behind it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] Cassandra 18963 4.0 [cassandra]
Posted by "pmcfadin (via GitHub)" <gi...@apache.org>.
pmcfadin commented on PR #2928:
URL: https://github.com/apache/cassandra/pull/2928#issuecomment-4227355216
Closing as [CASSANDRA-18963](https://issues.apache.org/jira/browse/CASSANDRA-18963) has been resolved as Fixed — if this PR contains changes not captured in the merged fix, please reopen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] Cassandra 18963 4.0 [cassandra]
Posted by "pmcfadin (via GitHub)" <gi...@apache.org>.
pmcfadin closed pull request #2928: Cassandra 18963 4.0
URL: https://github.com/apache/cassandra/pull/2928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] Cassandra 18963 4.0 [cassandra]
Posted by "jacek-lewandowski (via GitHub)" <gi...@apache.org>.
jacek-lewandowski commented on PR #2928:
URL: https://github.com/apache/cassandra/pull/2928#issuecomment-1829531028
https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/1127/workflows/4c460491-1f9e-4b85-8f9e-bd40d89ddf2e (j11)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] Cassandra 18963 4.0 [cassandra]
Posted by "jacek-lewandowski (via GitHub)" <gi...@apache.org>.
jacek-lewandowski commented on PR #2928:
URL: https://github.com/apache/cassandra/pull/2928#issuecomment-1829351166
https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/1125/workflows/a199ad8c-1a83-43c9-9218-a90c3472d113 (j8) - the failures are raised in https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/1125/workflows/a199ad8c-1a83-43c9-9218-a90c3472d113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org