You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by ad...@apache.org on 2020/07/10 17:30:36 UTC
[cassandra] branch cassandra-2.2 updated (f79d105 -> 257fb03)
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a change to branch cassandra-2.2
in repository https://gitbox.apache.org/repos/asf/cassandra.git.
from f79d105 Add bytebuddy support for in-jvm dtest
new 976096a Fix writing of snapshot manifest when the table has table-backed secondary indexes patch by Aleksandr Sorokoumov; reviewed by Andrés de la Peña for CASSANDRA-10968
new 257fb03 Merge branch 'cassandra-2.1' into cassandra-2.2
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.txt | 11 +++---
src/java/org/apache/cassandra/config/Config.java | 2 +-
.../org/apache/cassandra/db/ColumnFamilyStore.java | 5 ++-
src/java/org/apache/cassandra/db/Directories.java | 23 ++++++++-----
.../apache/cassandra/io/sstable/Descriptor.java | 7 +++-
.../apache/cassandra/db/ColumnFamilyStoreTest.java | 39 ++++++++++++++++++++++
6 files changed, 68 insertions(+), 19 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-2.1' into cassandra-2.2
Posted by ad...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-2.2
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 257fb0377343cbfdb58327da17f31d4eaed940f5
Merge: f79d105 976096a
Author: adelapena <a....@gmail.com>
AuthorDate: Fri Jul 10 17:52:23 2020 +0100
Merge branch 'cassandra-2.1' into cassandra-2.2
# Conflicts:
# CHANGES.txt
# src/java/org/apache/cassandra/db/ColumnFamilyStore.java
# test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
CHANGES.txt | 11 +++---
src/java/org/apache/cassandra/config/Config.java | 2 +-
.../org/apache/cassandra/db/ColumnFamilyStore.java | 5 ++-
src/java/org/apache/cassandra/db/Directories.java | 23 ++++++++-----
.../apache/cassandra/io/sstable/Descriptor.java | 7 +++-
.../apache/cassandra/db/ColumnFamilyStoreTest.java | 39 ++++++++++++++++++++++
6 files changed, 68 insertions(+), 19 deletions(-)
diff --cc CHANGES.txt
index b10a057,0a6268c..02de7c1
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,61 -1,8 +1,62 @@@
- 2.2.18
-2.1.21
++2.2.17
+ * Fix nomenclature of allow and deny lists (CASSANDRA-15862)
+ * Remove generated files from source artifact (CASSANDRA-15849)
+ * Remove duplicated tools binaries from tarballs (CASSANDRA-15768)
+ * Duplicate results with DISTINCT queries in mixed mode (CASSANDRA-15501)
+ * Disable JMX rebinding (CASSANDRA-15653)
- Merged from 2.1:
- * Fix parse error in cqlsh COPY FROM and formatting for map of blobs (CASSANDRA-15679)
-
- 2.2.17
+ * Fix Commit log replays when static column clustering keys are collections (CASSANDRA-14365)
+ * Fix Red Hat init script on newer systemd versions (CASSANDRA-15273)
+ * Allow EXTRA_CLASSPATH to work on tar/source installations (CASSANDRA-15567)
++Merged from 2.1:
+ * Fix writing of snapshot manifest when the table has table-backed secondary indexes (CASSANDRA-10968)
+ * Fix parse error in cqlsh COPY FROM and formatting for map of blobs (CASSANDRA-15679)
++
+
+2.2.16
+ * Fix SELECT JSON output for empty blobs (CASSANDRA-15435)
+ * In-JVM DTest: Set correct internode message version for upgrade test (CASSANDRA-15371)
+ * In-JVM DTest: Support NodeTool in dtest
+
++
+2.2.15
+ * Catch non-IOException in FileUtils.close to make sure that all resources are closed (CASSANDRA-15225)
+ * Handle exceptions during authentication/authorization (CASSANDRA-15041)
+ * Fix JDK7 compatibility broken in cassandra-2.2 (CASSANDRA-15050)
+ * Support cross version messaging in in-jvm upgrade dtests (CASSANDRA-15078)
+ * Fix index summary redistribution cancellation (CASSANDRA-15045)
+ * Refactor Circle CI configuration (CASSANDRA-14806)
+ * Fixing invalid CQL in security documentation (CASSANDRA-15020)
+ * Make tools/bin/token-generator py2/3 compatible (CASSANDRA-15012)
+ * Multi-version in-JVM dtests (CASSANDRA-14937)
+ * Allow instance class loaders to be garbage collected for inJVM dtest (CASSANDRA-15170)
+ * Add support for network topology and query tracing for inJVM dtest (CASSANDRA-15319)
+
+
+2.2.14
+ * CircleCI docker image should bake in more dependencies (CASSANDRA-14985)
+ * Don't enable client transports when bootstrap is pending (CASSANDRA-14525)
+ * MigrationManager attempts to pull schema from different major version nodes (CASSANDRA-14928)
+ * Don't skip entire sstables when reading backwards with mixed clustering column order
+ (CASSANDRA-14910)
+ * Cannot perform slice reads in reverse direction against tables with clustering columns
+ in mixed order (CASSANDRA-14899)
+ * Fix incorrect cqlsh results when selecting same columns multiple times (CASSANDRA-13262)
+ * Returns null instead of NaN or Infinity in JSON strings (CASSANDRA-14377)
+Merged from 2.1:
* Paged Range Slice queries with DISTINCT can drop rows from results (CASSANDRA-14956)
* Update release checksum algorithms to SHA-256, SHA-512 (CASSANDRA-14970)
+
+
+2.2.13
+ * Fix bug that prevented compaction of SSTables after full repairs (CASSANDRA-14423)
+ * Incorrect counting of pending messages in OutboundTcpConnection (CASSANDRA-11551)
+ * Fix compaction failure caused by reading un-flushed data (CASSANDRA-12743)
+ * Use Bounds instead of Range for sstables in anticompaction (CASSANDRA-14411)
+ * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286)
+ * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891)
+ * Fix query pager DEBUG log leak causing hit in paged reads throughput (CASSANDRA-14318)
+ * Backport circleci yaml (CASSANDRA-14240)
+Merged from 2.1:
* Check checksum before decompressing data (CASSANDRA-14284)
* CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt (CASSANDRA-14183)
diff --cc src/java/org/apache/cassandra/config/Config.java
index 7836396a,2c5e36a..ab79a08
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@@ -177,7 -154,7 +177,7 @@@ public class Confi
public volatile Integer stream_throughput_outbound_megabits_per_sec = 200;
public volatile Integer inter_dc_stream_throughput_outbound_megabits_per_sec = 200;
-- public String[] data_file_directories;
++ public String[] data_file_directories = new String[0];
public String saved_caches_directory;
diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 01330a6,df9b046..7f625e1
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@@ -2423,12 -2333,11 +2423,12 @@@ public class ColumnFamilyStore implemen
/**
* @param ephemeral If this flag is set to true, the snapshot will be cleaned during next startup
*/
- public void snapshotWithoutFlush(String snapshotName, Predicate<SSTableReader> predicate, boolean ephemeral)
+ public Set<SSTableReader> snapshotWithoutFlush(String snapshotName, Predicate<SSTableReader> predicate, boolean ephemeral)
{
+ Set<SSTableReader> snapshottedSSTables = new HashSet<>();
+ final JSONArray filesJSONArr = new JSONArray();
for (ColumnFamilyStore cfs : concatWithIndexes())
{
- final JSONArray filesJSONArr = new JSONArray();
try (RefViewFragment currentView = cfs.selectAndReference(CANONICAL_SSTABLES))
{
for (SSTableReader ssTable : currentView.sstables)
@@@ -2440,17 -2349,14 +2440,16 @@@
ssTable.createLinks(snapshotDirectory.getPath()); // hard links
filesJSONArr.add(ssTable.descriptor.relativeFilenameFor(Component.DATA));
- if (logger.isDebugEnabled())
- logger.debug("Snapshot for {} keyspace data file {} created in {}", keyspace, ssTable.getFilename(), snapshotDirectory);
+ if (logger.isTraceEnabled())
+ logger.trace("Snapshot for {} keyspace data file {} created in {}", keyspace, ssTable.getFilename(), snapshotDirectory);
+ snapshottedSSTables.add(ssTable);
}
-
- writeSnapshotManifest(filesJSONArr, snapshotName);
}
}
+ writeSnapshotManifest(filesJSONArr, snapshotName);
if (ephemeral)
createEphemeralSnapshotMarkerFile(snapshotName);
+ return snapshottedSSTables;
}
private void writeSnapshotManifest(final JSONArray filesJSONArr, final String snapshotName)
diff --cc src/java/org/apache/cassandra/db/Directories.java
index d1aa650,35aa447..5e02cd7
--- a/src/java/org/apache/cassandra/db/Directories.java
+++ b/src/java/org/apache/cassandra/db/Directories.java
@@@ -406,30 -368,7 +406,30 @@@ public class Directorie
public static File getSnapshotDirectory(Descriptor desc, String snapshotName)
{
- return getOrCreate(desc.directory, SNAPSHOT_SUBDIR, snapshotName);
+ return getSnapshotDirectory(desc.directory, snapshotName);
+ }
+
+ /**
+ * Returns directory to write snapshot. If directory does not exist, then one is created.
+ *
+ * If given {@code location} indicates secondary index, this will return
+ * {@code <cf dir>/snapshots/<snapshot name>/.<index name>}.
+ * Otherwise, this will return {@code <cf dir>/snapshots/<snapshot name>}.
+ *
+ * @param location base directory
+ * @param snapshotName snapshot name
+ * @return directory to write snapshot
+ */
+ public static File getSnapshotDirectory(File location, String snapshotName)
+ {
- if (location.getName().startsWith(SECONDARY_INDEX_NAME_SEPARATOR))
++ if (isSecondaryIndexFolder(location))
+ {
+ return getOrCreate(location.getParentFile(), SNAPSHOT_SUBDIR, snapshotName, location.getName());
+ }
+ else
+ {
+ return getOrCreate(location, SNAPSHOT_SUBDIR, snapshotName);
+ }
}
public File getSnapshotManifestFile(String snapshotName)
@@@ -451,19 -389,7 +451,19 @@@
public static File getBackupsDirectory(Descriptor desc)
{
- return getOrCreate(desc.directory, BACKUPS_SUBDIR);
+ return getBackupsDirectory(desc.directory);
+ }
+
+ public static File getBackupsDirectory(File location)
+ {
- if (location.getName().startsWith(SECONDARY_INDEX_NAME_SEPARATOR))
++ if (isSecondaryIndexFolder(location))
+ {
+ return getOrCreate(location.getParentFile(), BACKUPS_SUBDIR, location.getName());
+ }
+ else
+ {
+ return getOrCreate(location, BACKUPS_SUBDIR);
+ }
}
public SSTableLister sstableLister()
@@@ -680,9 -605,7 +680,9 @@@
final List<File> snapshots = new LinkedList<>();
for (final File dir : dataPaths)
{
- File snapshotDir = dir.getName().startsWith(SECONDARY_INDEX_NAME_SEPARATOR) ?
- new File(dir.getParent(), SNAPSHOT_SUBDIR) :
- new File(dir, SNAPSHOT_SUBDIR);
- final File snapshotDir = new File(dir,SNAPSHOT_SUBDIR);
++ File snapshotDir = isSecondaryIndexFolder(dir)
++ ? new File(dir.getParent(), SNAPSHOT_SUBDIR)
++ : new File(dir, SNAPSHOT_SUBDIR);
if (snapshotDir.exists() && snapshotDir.isDirectory())
{
final File[] snapshotDirs = snapshotDir.listFiles();
@@@ -704,15 -627,7 +704,15 @@@
{
for (File dir : dataPaths)
{
- File snapshotDir = new File(dir, join(SNAPSHOT_SUBDIR, snapshotName));
+ File snapshotDir;
- if (dir.getName().startsWith(SECONDARY_INDEX_NAME_SEPARATOR))
++ if (isSecondaryIndexFolder(dir))
+ {
+ snapshotDir = new File(dir.getParentFile(), join(SNAPSHOT_SUBDIR, snapshotName, dir.getName()));
+ }
+ else
+ {
+ snapshotDir = new File(dir, join(SNAPSHOT_SUBDIR, snapshotName));
+ }
if (snapshotDir.exists())
return true;
}
@@@ -763,12 -665,7 +763,12 @@@
{
long result = 0L;
for (File dir : dataPaths)
- result += getTrueAllocatedSizeIn(new File(dir, join(SNAPSHOT_SUBDIR)));
+ {
- File snapshotDir = dir.getName().startsWith(SECONDARY_INDEX_NAME_SEPARATOR) ?
- new File(dir.getParent(), SNAPSHOT_SUBDIR) :
- new File(dir, SNAPSHOT_SUBDIR);
++ File snapshotDir = isSecondaryIndexFolder(dir)
++ ? new File(dir.getParent(), SNAPSHOT_SUBDIR)
++ : new File(dir, SNAPSHOT_SUBDIR);
+ result += getTrueAllocatedSizeIn(snapshotDir);
+ }
return result;
}
@@@ -809,6 -711,6 +809,11 @@@
return result;
}
++ public static boolean isSecondaryIndexFolder(File dir)
++ {
++ return dir.getName().startsWith(SECONDARY_INDEX_NAME_SEPARATOR);
++ }
++
public List<File> getCFDirectories()
{
List<File> result = new ArrayList<>();
diff --cc src/java/org/apache/cassandra/io/sstable/Descriptor.java
index ed81616,4415db4..d9c7550
--- a/src/java/org/apache/cassandra/io/sstable/Descriptor.java
+++ b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
@@@ -151,6 -199,6 +151,11 @@@ public class Descripto
public String relativeFilenameFor(Component component)
{
final StringBuilder buff = new StringBuilder();
++ if (Directories.isSecondaryIndexFolder(directory))
++ {
++ buff.append(directory.getName()).append(File.separator);
++ }
++
appendFileName(buff);
buff.append(separator).append(component.name());
return buff.toString();
@@@ -220,81 -250,37 +225,81 @@@
StringTokenizer st = new StringTokenizer(name, String.valueOf(separator));
String nexttok;
- // all filenames must start with keyspace and column family
- String ksname = st.nextToken();
- String cfname = st.nextToken();
-
- // optional temporary marker
- nexttok = st.nextToken();
- Type type = Type.FINAL;
- if (nexttok.equals(Type.TEMP.marker))
+ // read tokens backwards to determine version
+ Deque<String> tokenStack = new ArrayDeque<>();
+ while (st.hasMoreTokens())
{
- type = Type.TEMP;
- nexttok = st.nextToken();
+ tokenStack.push(st.nextToken());
}
- else if (nexttok.equals(Type.TEMPLINK.marker))
+
+ // component suffix
+ String component = skipComponent ? null : tokenStack.pop();
+
+ nexttok = tokenStack.pop();
+ // generation OR Type
+ SSTableFormat.Type fmt = SSTableFormat.Type.LEGACY;
+ if (!CharMatcher.DIGIT.matchesAllOf(nexttok))
{
- type = Type.TEMPLINK;
- nexttok = st.nextToken();
+ fmt = SSTableFormat.Type.validate(nexttok);
+ nexttok = tokenStack.pop();
}
- if (!Version.validate(nexttok))
+ // generation
+ int generation = Integer.parseInt(nexttok);
+
+ // version
+ nexttok = tokenStack.pop();
+ Version version = fmt.info.getVersion(nexttok);
+
+ if (!version.validate(nexttok))
throw new UnsupportedOperationException("SSTable " + name + " is too old to open. Upgrade to 2.0 first, and run upgradesstables");
- Version version = new Version(nexttok);
- nexttok = st.nextToken();
- int generation = Integer.parseInt(nexttok);
+ // optional temporary marker
+ Type type = Descriptor.Type.FINAL;
+ nexttok = tokenStack.peek();
+ if (Descriptor.Type.TEMP.marker.equals(nexttok))
+ {
+ type = Descriptor.Type.TEMP;
+ tokenStack.pop();
+ }
+ else if (Descriptor.Type.TEMPLINK.marker.equals(nexttok))
+ {
+ type = Descriptor.Type.TEMPLINK;
+ tokenStack.pop();
+ }
- // component suffix
- String component = null;
- if (!skipComponent)
- component = st.nextToken();
- directory = directory != null ? directory : new File(".");
- return Pair.create(new Descriptor(version, directory, ksname, cfname, generation, type), component);
+ // ks/cf names
+ String ksname, cfname;
+ if (version.hasNewFileName())
+ {
+ // for 2.1+ read ks and cf names from directory
+ File cfDirectory = parentDirectory;
+ // check if this is secondary index
+ String indexName = "";
- if (cfDirectory.getName().startsWith(Directories.SECONDARY_INDEX_NAME_SEPARATOR))
++ if (Directories.isSecondaryIndexFolder(cfDirectory))
+ {
+ indexName = cfDirectory.getName();
+ cfDirectory = cfDirectory.getParentFile();
+ }
+ if (cfDirectory.getName().equals(Directories.BACKUPS_SUBDIR))
+ {
+ cfDirectory = cfDirectory.getParentFile();
+ }
+ else if (cfDirectory.getParentFile().getName().equals(Directories.SNAPSHOT_SUBDIR))
+ {
+ cfDirectory = cfDirectory.getParentFile().getParentFile();
+ }
+ cfname = cfDirectory.getName().split("-")[0] + indexName;
+ ksname = cfDirectory.getParentFile().getName();
+ }
+ else
+ {
+ cfname = tokenStack.pop();
+ ksname = tokenStack.pop();
+ }
+ assert tokenStack.isEmpty() : "Invalid file name " + name + " in " + directory;
+
+ return Pair.create(new Descriptor(version, parentDirectory, ksname, cfname, generation, type, fmt), component);
}
/**
diff --cc test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
index b21d3fa,1fc2029..4d452c6
--- a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
+++ b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
@@@ -91,10 -57,10 +92,13 @@@ import org.apache.cassandra.service.Sto
import org.apache.cassandra.thrift.SlicePredicate;
import org.apache.cassandra.thrift.SliceRange;
import org.apache.cassandra.thrift.ThriftValidation;
-import org.apache.cassandra.utils.*;
+import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.Pair;
+import org.apache.cassandra.utils.WrappedRunnable;
+ import org.json.simple.JSONArray;
+ import org.json.simple.JSONObject;
+ import org.json.simple.parser.JSONParser;
import static org.apache.cassandra.Util.cellname;
import static org.apache.cassandra.Util.column;
@@@ -2353,7 -2234,37 +2357,42 @@@ public class ColumnFamilyStoreTes
PerRowSecondaryIndexTest.TestIndex.ACTIVE = false;
ColumnFamilyStore.rebuildSecondaryIndex("PerRowSecondaryIndex", "Indexed1", PerRowSecondaryIndexTest.TestIndex.class.getSimpleName());
assertNull(PerRowSecondaryIndexTest.TestIndex.LAST_INDEXED_KEY);
-
+
PerRowSecondaryIndexTest.TestIndex.reset();
}
+
+ @Test
+ public void testSnapshotWithoutFlushWithSecondaryIndexes() throws Exception
+ {
- String keyspaceName = "Keyspace1";
- String cfName = "Indexed1";
- Keyspace keyspace = Keyspace.open(keyspaceName);
- ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(cfName);
++ Keyspace keyspace = Keyspace.open(KEYSPACE1);
++ ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(CF_INDEX1);
+ cfs.truncateBlocking();
+
+ List<Mutation> rms = new LinkedList<>();
+ Mutation rm;
+
- rm = new Mutation(keyspaceName, ByteBufferUtil.bytes("k1"));
- rm.add(cfName, cellname("birthdate"), ByteBufferUtil.bytes(1L), 0);
- rm.add(cfName, cellname("nobirthdate"), ByteBufferUtil.bytes(1L), 0);
++ rm = new Mutation(KEYSPACE1, ByteBufferUtil.bytes("k1"));
++ rm.add(CF_INDEX1, cellname("birthdate"), ByteBufferUtil.bytes(1L), 0);
++ rm.add(CF_INDEX1, cellname("nobirthdate"), ByteBufferUtil.bytes(1L), 0);
+ rms.add(rm);
+ Util.writeColumnFamily(rms);
+
+ String snapshotName = "newSnapshot";
+ cfs.snapshotWithoutFlush(snapshotName);
+
+ File snapshotManifestFile = cfs.directories.getSnapshotManifestFile(snapshotName);
+ JSONParser parser = new JSONParser();
+ JSONObject manifest = (JSONObject) parser.parse(new FileReader(snapshotManifestFile));
+ JSONArray files = (JSONArray) manifest.get("files");
+
- // Keyspace1-Indexed1 and the corresponding index CFS
++ // Keyspace1-Indexed1 and the corresponding index
+ assert files.size() == 2;
++
++ // Snapshot of the secondary index is stored in the subfolder with the same file name
++ String baseTableFile = (String) files.get(0);
++ String indexTableFile = (String) files.get(1);
++ assert !baseTableFile.equals(indexTableFile);
++ assert Directories.isSecondaryIndexFolder(new File(indexTableFile).getParentFile());
++ assert indexTableFile.endsWith(baseTableFile);
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org