You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by jb...@apache.org on 2014/05/03 05:49:50 UTC
[1/9] git commit: give some antlr targets more heap space
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 2658c90e6 -> af96d405b
refs/heads/cassandra-2.1 943287065 -> 8e897de68
refs/heads/trunk a44ba68cf -> b8c0199c3
give some antlr targets more heap space
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fab45571
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fab45571
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fab45571
Branch: refs/heads/cassandra-2.0
Commit: fab45571b851b118bb88e052e63980c09aa58a52
Parents: 2658c90
Author: Jonathan Ellis <jb...@apache.org>
Authored: Fri May 2 16:23:32 2014 -0500
Committer: Jonathan Ellis <jb...@apache.org>
Committed: Fri May 2 22:41:14 2014 -0500
----------------------------------------------------------------------
build.xml | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/fab45571/build.xml
----------------------------------------------------------------------
diff --git a/build.xml b/build.xml
index 5c6c736..62f980a 100644
--- a/build.xml
+++ b/build.xml
@@ -192,6 +192,7 @@
classpath="${build.lib}/antlr-3.2.jar"
fork="true"
failonerror="true">
+ <jvmarg value="-Xmx512M" />
<arg value="${build.src.java}/org/apache/cassandra/cli/Cli.g" />
<arg value="-fo" />
<arg value="${build.src.gen-java}/org/apache/cassandra/cli/" />
@@ -229,6 +230,7 @@
classpath="${build.lib}/antlr-3.2.jar"
fork="true"
failonerror="true">
+ <jvmarg value="-Xmx512M" />
<arg value="-Xconversiontimeout" />
<arg value="10000" />
<arg value="${build.src.java}/org/apache/cassandra/cql3/Cql.g" />
[9/9] git commit: Merge branch 'cassandra-2.1' into trunk
Posted by jb...@apache.org.
Merge branch 'cassandra-2.1' into trunk
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b8c0199c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b8c0199c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b8c0199c
Branch: refs/heads/trunk
Commit: b8c0199c33567184043afeff83924a313e69d22f
Parents: a44ba68 8e897de
Author: Jonathan Ellis <jb...@apache.org>
Authored: Fri May 2 22:49:44 2014 -0500
Committer: Jonathan Ellis <jb...@apache.org>
Committed: Fri May 2 22:49:44 2014 -0500
----------------------------------------------------------------------
CHANGES.txt | 4 +-
build.xml | 2 +
.../service/BatchlogEndpointSelector.java | 110 ++++++++++++++++++
.../apache/cassandra/service/StorageProxy.java | 21 +---
.../service/BatchlogEndpointSelectorTest.java | 116 +++++++++++++++++++
5 files changed, 236 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8c0199c/CHANGES.txt
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8c0199c/build.xml
----------------------------------------------------------------------
diff --cc build.xml
index 1f67aa3,898b141..8b47e4e
--- a/build.xml
+++ b/build.xml
@@@ -190,9 -190,10 +190,10 @@@
<target name="gen-cli-grammar" depends="check-gen-cli-grammar" unless="cliUpToDate">
<echo>Building Grammar ${build.src.java}/org/apache/cassandra/cli/Cli.g ....</echo>
<java classname="org.antlr.Tool"
- classpath="${build.lib}/antlr-3.2.jar"
+ classpath="${build.dir.lib}/jars/antlr-3.5.2.jar;${build.lib}/antlr-runtime-3.5.2.jar;${build.lib}/stringtemplate-4.0.2.jar"
fork="true"
failonerror="true">
+ <jvmarg value="-Xmx512M" />
<arg value="${build.src.java}/org/apache/cassandra/cli/Cli.g" />
<arg value="-fo" />
<arg value="${build.src.gen-java}/org/apache/cassandra/cli/" />
@@@ -211,9 -217,21 +212,10 @@@
<target name="gen-cql3-grammar" depends="check-gen-cql3-grammar" unless="cql3current">
<echo>Building Grammar ${build.src.java}/org/apache/cassandra/cql3/Cql.g ...</echo>
<java classname="org.antlr.Tool"
- classpath="${build.lib}/antlr-3.2.jar"
+ classpath="${build.dir.lib}/jars/antlr-3.5.2.jar;${build.lib}/antlr-runtime-3.5.2.jar;${build.lib}/stringtemplate-4.0.2.jar"
fork="true"
failonerror="true">
+ <jvmarg value="-Xmx512M" />
<arg value="-Xconversiontimeout" />
<arg value="10000" />
<arg value="${build.src.java}/org/apache/cassandra/cql3/Cql.g" />
[4/9] git commit: Make batchlog replica selection rack-aware patch by
Mikhail Stepura; reviewed by jbellis for CASSANDRA-6551
Posted by jb...@apache.org.
Make batchlog replica selection rack-aware
patch by Mikhail Stepura; reviewed by jbellis for CASSANDRA-6551
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/af96d405
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/af96d405
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/af96d405
Branch: refs/heads/cassandra-2.0
Commit: af96d405b42a9e4ae23cba841b7a5d83ee8f7ec8
Parents: fab4557
Author: Jonathan Ellis <jb...@apache.org>
Authored: Fri May 2 22:47:27 2014 -0500
Committer: Jonathan Ellis <jb...@apache.org>
Committed: Fri May 2 22:47:27 2014 -0500
----------------------------------------------------------------------
CHANGES.txt | 2 +-
.../service/BatchlogEndpointSelector.java | 110 ++++++++++++++++++
.../apache/cassandra/service/StorageProxy.java | 21 +---
.../service/BatchlogEndpointSelectorTest.java | 116 +++++++++++++++++++
4 files changed, 232 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 9ab1a5f..5799659 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,5 @@
2.0.8
-=======
+ * Make batchlog replica selection rack-aware (CASSANDRA-6551)
* Add Google Compute Engine snitch (CASSANDRA-7132)
* Allow overriding cassandra-rackdc.properties file (CASSANDRA-7072)
* Set JMX RMI port to 7199 (CASSANDRA-7087)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java
----------------------------------------------------------------------
diff --git a/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java b/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java
new file mode 100644
index 0000000..bf032f5
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.service;
+
+
+import java.net.InetAddress;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.cassandra.gms.FailureDetector;
+import org.apache.cassandra.utils.FBUtilities;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Multimap;
+
+public class BatchlogEndpointSelector
+{
+ private final String localRack;
+
+ public BatchlogEndpointSelector(String localRack)
+ {
+ this.localRack = localRack;
+ }
+
+ /**
+ * @param endpoints nodes in the local datacenter, grouped by rack name
+ * @return list of candidates for batchlog hosting. if possible these will be two nodes from different racks.
+ */
+ public Collection<InetAddress> chooseEndpoints(Multimap<String, InetAddress> endpoints)
+ {
+ // strip out dead endpoints and localhost
+ ListMultimap<String, InetAddress> validated = ArrayListMultimap.create();
+ for (Map.Entry<String, InetAddress> entry : endpoints.entries())
+ {
+ if (isValid(entry.getValue()))
+ validated.put(entry.getKey(), entry.getValue());
+ }
+ if (validated.size() <= 2)
+ return validated.values();
+
+ if ((validated.size() - validated.get(localRack).size()) >= 2)
+ {
+ // we have enough endpoints in other racks
+ validated.removeAll(localRack);
+ }
+
+ if (validated.keySet().size() == 1)
+ {
+ // we have only 1 `other` rack
+ Collection<InetAddress> otherRack = Iterables.getOnlyElement(validated.asMap().values());
+ return Lists.newArrayList(Iterables.limit(otherRack, 2));
+ }
+
+ // randomize which racks we pick from if more than 2 remaining
+ Collection<String> racks;
+ if (validated.keySet().size() == 2)
+ {
+ racks = validated.keySet();
+ }
+ else
+ {
+ racks = Lists.newArrayList(validated.keySet());
+ Collections.shuffle((List) racks);
+ }
+
+ // grab a random member of up to two racks
+ List<InetAddress> result = new ArrayList<>(2);
+ for (String rack : Iterables.limit(racks, 2))
+ {
+ List<InetAddress> rackMembers = validated.get(rack);
+ result.add(rackMembers.get(getRandomInt(rackMembers.size())));
+ }
+
+ return result;
+ }
+
+ @VisibleForTesting
+ protected boolean isValid(InetAddress input)
+ {
+ return !input.equals(FBUtilities.getBroadcastAddress()) && FailureDetector.instance.isAlive(input);
+ }
+
+ @VisibleForTesting
+ protected int getRandomInt(int bound)
+ {
+ return FBUtilities.threadLocalRandom().nextInt(bound);
+ }
+}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/src/java/org/apache/cassandra/service/StorageProxy.java
----------------------------------------------------------------------
diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java b/src/java/org/apache/cassandra/service/StorageProxy.java
index 14d5ee2..2bf8e7f 100644
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@ -22,7 +22,6 @@ import java.lang.management.ManagementFactory;
import java.net.InetAddress;
import java.nio.ByteBuffer;
import java.util.*;
-import java.util.Random;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import javax.management.MBeanServer;
@@ -729,24 +728,14 @@ public class StorageProxy implements StorageProxyMBean
throws UnavailableException
{
TokenMetadata.Topology topology = StorageService.instance.getTokenMetadata().cachedOnlyTokenMap().getTopology();
- List<InetAddress> localEndpoints = new ArrayList<>(topology.getDatacenterEndpoints().get(localDataCenter));
-
+ Multimap<String, InetAddress> localEndpoints = HashMultimap.create(topology.getDatacenterRacks().get(localDataCenter));
+
// special case for single-node datacenters
if (localEndpoints.size() == 1)
- return localEndpoints;
-
- List<InetAddress> chosenEndpoints = new ArrayList<>(2);
- int startOffset = new Random().nextInt(localEndpoints.size());
+ return localEndpoints.values();
- // starts at some random point in the list, advances forward until the end, then loops
- // around to the beginning, advancing again until it is back at the starting point again.
- for (int i = 0; i < localEndpoints.size() && chosenEndpoints.size() < 2; i++)
- {
- InetAddress endpoint = localEndpoints.get((i + startOffset) % localEndpoints.size());
- // skip localhost and non-alive nodes
- if (!endpoint.equals(FBUtilities.getBroadcastAddress()) && FailureDetector.instance.isAlive(endpoint))
- chosenEndpoints.add(endpoint);
- }
+ String localRack = DatabaseDescriptor.getEndpointSnitch().getRack(FBUtilities.getBroadcastAddress());
+ Collection<InetAddress> chosenEndpoints = new BatchlogEndpointSelector(localRack).chooseEndpoints(localEndpoints);
if (chosenEndpoints.isEmpty())
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java
----------------------------------------------------------------------
diff --git a/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java b/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java
new file mode 100644
index 0000000..293078d
--- /dev/null
+++ b/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.service;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.junit.Assert.assertThat;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.Collection;
+
+import org.junit.Test;
+import org.junit.matchers.JUnitMatchers;
+
+import com.google.common.collect.ImmutableMultimap;
+import com.google.common.collect.Multimap;
+
+public class BatchlogEndpointSelectorTest
+{
+ private final BatchlogEndpointSelector target;
+ private static final String LOCAL = "local";
+
+
+ public BatchlogEndpointSelectorTest() throws UnknownHostException
+ {
+ target = new BatchlogEndpointSelector(LOCAL)
+ {
+ @Override
+ protected boolean isValid(InetAddress input)
+ {
+ //we will use always alive non-localhost endpoints
+ return true;
+ }
+
+ @Override
+ protected int getRandomInt(int bound)
+ {
+ //we don't need a random behavior here
+ return bound - 1;
+ }
+ };
+ }
+
+ @Test
+ public void shouldSelect2hostsFromNonLocalRacks() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .put(LOCAL, InetAddress.getByName("00"))
+ .put("1", InetAddress.getByName("1"))
+ .put("1", InetAddress.getByName("11"))
+ .put("2", InetAddress.getByName("2"))
+ .put("2", InetAddress.getByName("22"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(2));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("11")));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("22")));
+ }
+
+ @Test
+ public void shouldSelectHostFromLocal() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .put(LOCAL, InetAddress.getByName("00"))
+ .put("1", InetAddress.getByName("1"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(2));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("1")));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("0")));
+ }
+
+ @Test
+ public void shouldReturnAsIsIfNoEnoughEndpoints() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(1));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("0")));
+ }
+
+ @Test
+ public void shouldSelectTwoFirstHostsFromSingleOtherRack() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .put(LOCAL, InetAddress.getByName("00"))
+ .put("1", InetAddress.getByName("1"))
+ .put("1", InetAddress.getByName("11"))
+ .put("1", InetAddress.getByName("111"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(2));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("1")));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("11")));
+ }
+}
[2/9] git commit: give some antlr targets more heap space
Posted by jb...@apache.org.
give some antlr targets more heap space
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fab45571
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fab45571
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fab45571
Branch: refs/heads/cassandra-2.1
Commit: fab45571b851b118bb88e052e63980c09aa58a52
Parents: 2658c90
Author: Jonathan Ellis <jb...@apache.org>
Authored: Fri May 2 16:23:32 2014 -0500
Committer: Jonathan Ellis <jb...@apache.org>
Committed: Fri May 2 22:41:14 2014 -0500
----------------------------------------------------------------------
build.xml | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/fab45571/build.xml
----------------------------------------------------------------------
diff --git a/build.xml b/build.xml
index 5c6c736..62f980a 100644
--- a/build.xml
+++ b/build.xml
@@ -192,6 +192,7 @@
classpath="${build.lib}/antlr-3.2.jar"
fork="true"
failonerror="true">
+ <jvmarg value="-Xmx512M" />
<arg value="${build.src.java}/org/apache/cassandra/cli/Cli.g" />
<arg value="-fo" />
<arg value="${build.src.gen-java}/org/apache/cassandra/cli/" />
@@ -229,6 +230,7 @@
classpath="${build.lib}/antlr-3.2.jar"
fork="true"
failonerror="true">
+ <jvmarg value="-Xmx512M" />
<arg value="-Xconversiontimeout" />
<arg value="10000" />
<arg value="${build.src.java}/org/apache/cassandra/cql3/Cql.g" />
[7/9] git commit: merge from 2.1
Posted by jb...@apache.org.
merge from 2.1
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8e897de6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8e897de6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8e897de6
Branch: refs/heads/cassandra-2.1
Commit: 8e897de68716e47c0341dc6aaa94a2ab4a7e519c
Parents: 9432870 af96d40
Author: Jonathan Ellis <jb...@apache.org>
Authored: Fri May 2 22:49:31 2014 -0500
Committer: Jonathan Ellis <jb...@apache.org>
Committed: Fri May 2 22:49:31 2014 -0500
----------------------------------------------------------------------
CHANGES.txt | 4 +-
build.xml | 2 +
.../service/BatchlogEndpointSelector.java | 110 ++++++++++++++++++
.../apache/cassandra/service/StorageProxy.java | 21 +---
.../service/BatchlogEndpointSelectorTest.java | 116 +++++++++++++++++++
5 files changed, 236 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8e897de6/CHANGES.txt
----------------------------------------------------------------------
diff --cc CHANGES.txt
index 82b0196,5799659..eece557
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,70 -1,6 +1,71 @@@
-2.0.8
+2.1.0-rc1
+ * Parallel streaming for sstableloader (CASSANDRA-3668)
+Merged from 2.0:
+ * Make batchlog replica selection rack-aware (CASSANDRA-6551)
* Add Google Compute Engine snitch (CASSANDRA-7132)
- Merged from 1.2:
+ * Update system.peers correctly when relocating tokens (CASSANDRA-7126)
+
++
+2.1.0-beta2
+ * Increase default CL space to 8GB (CASSANDRA-7031)
+ * Add range tombstones to read repair digests (CASSANDRA-6863)
+ * Fix BTree.clear for large updates (CASSANDRA-6943)
+ * Fail write instead of logging a warning when unable to append to CL
+ (CASSANDRA-6764)
+ * Eliminate possibility of CL segment appearing twice in active list
+ (CASSANDRA-6557)
+ * Apply DONTNEED fadvise to commitlog segments (CASSANDRA-6759)
+ * Switch CRC component to Adler and include it for compressed sstables
+ (CASSANDRA-4165)
+ * Allow cassandra-stress to set compaction strategy options (CASSANDRA-6451)
+ * Add broadcast_rpc_address option to cassandra.yaml (CASSANDRA-5899)
+ * Auto reload GossipingPropertyFileSnitch config (CASSANDRA-5897)
+ * Fix overflow of memtable_total_space_in_mb (CASSANDRA-6573)
+ * Fix ABTC NPE and apply update function correctly (CASSANDRA-6692)
+ * Allow nodetool to use a file or prompt for password (CASSANDRA-6660)
+ * Fix AIOOBE when concurrently accessing ABSC (CASSANDRA-6742)
+ * Fix assertion error in ALTER TYPE RENAME (CASSANDRA-6705)
+ * Scrub should not always clear out repaired status (CASSANDRA-5351)
+ * Improve handling of range tombstone for wide partitions (CASSANDRA-6446)
+ * Fix ClassCastException for compact table with composites (CASSANDRA-6738)
+ * Fix potentially repairing with wrong nodes (CASSANDRA-6808)
+ * Change caching option syntax (CASSANDRA-6745)
+ * Fix stress to do proper counter reads (CASSANDRA-6835)
+ * Fix help message for stress counter_write (CASSANDRA-6824)
+ * Fix stress smart Thrift client to pick servers correctly (CASSANDRA-6848)
+ * Add logging levels (minimal, normal or verbose) to stress tool (CASSANDRA-6849)
+ * Fix race condition in Batch CLE (CASSANDRA-6860)
+ * Improve cleanup/scrub/upgradesstables failure handling (CASSANDRA-6774)
+ * ByteBuffer write() methods for serializing sstables (CASSANDRA-6781)
+ * Proper compare function for CollectionType (CASSANDRA-6783)
+ * Update native server to Netty 4 (CASSANDRA-6236)
+ * Fix off-by-one error in stress (CASSANDRA-6883)
+ * Make OpOrder AutoCloseable (CASSANDRA-6901)
+ * Remove sync repair JMX interface (CASSANDRA-6900)
+ * Add multiple memory allocation options for memtables (CASSANDRA-6689, 6694)
+ * Remove adjusted op rate from stress output (CASSANDRA-6921)
+ * Add optimized CF.hasColumns() implementations (CASSANDRA-6941)
+ * Serialize batchlog mutations with the version of the target node
+ (CASSANDRA-6931)
+ * Optimize CounterColumn#reconcile() (CASSANDRA-6953)
+ * Properly remove 1.2 sstable support in 2.1 (CASSANDRA-6869)
+ * Lock counter cells, not partitions (CASSANDRA-6880)
+ * Track presence of legacy counter shards in sstables (CASSANDRA-6888)
+ * Ensure safe resource cleanup when replacing sstables (CASSANDRA-6912)
+ * Add failure handler to async callback (CASSANDRA-6747)
+ * Fix AE when closing SSTable without releasing reference (CASSANDRA-7000)
+ * Clean up IndexInfo on keyspace/table drops (CASSANDRA-6924)
+ * Only snapshot relative SSTables when sequential repair (CASSANDRA-7024)
+ * Require nodetool rebuild_index to specify index names (CASSANDRA-7038)
+ * fix cassandra stress errors on reads with native protocol (CASSANDRA-7033)
+ * Use OpOrder to guard sstable references for reads (CASSANDRA-6919)
+ * Preemptive opening of compaction result (CASSANDRA-6916)
+ * Multi-threaded scrub/cleanup/upgradesstables (CASSANDRA-5547)
+ * Optimize cellname comparison (CASSANDRA-6934)
+ * Native protocol v3 (CASSANDRA-6855)
+ * Optimize Cell liveness checks and clean up Cell (CASSANDRA-7119)
+ * Support consistent range movements (CASSANDRA-2434)
+Merged from 2.0:
* Allow overriding cassandra-rackdc.properties file (CASSANDRA-7072)
* Set JMX RMI port to 7199 (CASSANDRA-7087)
* Use LOCAL_QUORUM for data reads at LOCAL_SERIAL (CASSANDRA-6939)
@@@ -169,13 -95,9 +170,14 @@@ Merged from 1.2
* Schedule schema pulls on change (CASSANDRA-6971)
* Non-droppable verbs shouldn't be dropped from OTC (CASSANDRA-6980)
* Shutdown batchlog executor in SS#drain() (CASSANDRA-7025)
+ * Fix batchlog to account for CF truncation records (CASSANDRA-6999)
+ * Fix CQLSH parsing of functions and BLOB literals (CASSANDRA-7018)
* Properly load trustore in the native protocol (CASSANDRA-6847)
+ * Always clean up references in SerializingCache (CASSANDRA-6994)
+ * Don't shut MessagingService down when replacing a node (CASSANDRA-6476)
+ * fix npe when doing -Dcassandra.fd_initial_value_ms (CASSANDRA-6751)
+
2.0.6
* Avoid race-prone second "scrub" of system keyspace (CASSANDRA-6797)
* Pool CqlRecordWriter clients by inetaddress rather than Range
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8e897de6/build.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8e897de6/src/java/org/apache/cassandra/service/StorageProxy.java
----------------------------------------------------------------------
[3/9] git commit: give some antlr targets more heap space
Posted by jb...@apache.org.
give some antlr targets more heap space
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fab45571
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fab45571
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fab45571
Branch: refs/heads/trunk
Commit: fab45571b851b118bb88e052e63980c09aa58a52
Parents: 2658c90
Author: Jonathan Ellis <jb...@apache.org>
Authored: Fri May 2 16:23:32 2014 -0500
Committer: Jonathan Ellis <jb...@apache.org>
Committed: Fri May 2 22:41:14 2014 -0500
----------------------------------------------------------------------
build.xml | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/fab45571/build.xml
----------------------------------------------------------------------
diff --git a/build.xml b/build.xml
index 5c6c736..62f980a 100644
--- a/build.xml
+++ b/build.xml
@@ -192,6 +192,7 @@
classpath="${build.lib}/antlr-3.2.jar"
fork="true"
failonerror="true">
+ <jvmarg value="-Xmx512M" />
<arg value="${build.src.java}/org/apache/cassandra/cli/Cli.g" />
<arg value="-fo" />
<arg value="${build.src.gen-java}/org/apache/cassandra/cli/" />
@@ -229,6 +230,7 @@
classpath="${build.lib}/antlr-3.2.jar"
fork="true"
failonerror="true">
+ <jvmarg value="-Xmx512M" />
<arg value="-Xconversiontimeout" />
<arg value="10000" />
<arg value="${build.src.java}/org/apache/cassandra/cql3/Cql.g" />
[6/9] git commit: Make batchlog replica selection rack-aware patch by
Mikhail Stepura; reviewed by jbellis for CASSANDRA-6551
Posted by jb...@apache.org.
Make batchlog replica selection rack-aware
patch by Mikhail Stepura; reviewed by jbellis for CASSANDRA-6551
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/af96d405
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/af96d405
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/af96d405
Branch: refs/heads/trunk
Commit: af96d405b42a9e4ae23cba841b7a5d83ee8f7ec8
Parents: fab4557
Author: Jonathan Ellis <jb...@apache.org>
Authored: Fri May 2 22:47:27 2014 -0500
Committer: Jonathan Ellis <jb...@apache.org>
Committed: Fri May 2 22:47:27 2014 -0500
----------------------------------------------------------------------
CHANGES.txt | 2 +-
.../service/BatchlogEndpointSelector.java | 110 ++++++++++++++++++
.../apache/cassandra/service/StorageProxy.java | 21 +---
.../service/BatchlogEndpointSelectorTest.java | 116 +++++++++++++++++++
4 files changed, 232 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 9ab1a5f..5799659 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,5 @@
2.0.8
-=======
+ * Make batchlog replica selection rack-aware (CASSANDRA-6551)
* Add Google Compute Engine snitch (CASSANDRA-7132)
* Allow overriding cassandra-rackdc.properties file (CASSANDRA-7072)
* Set JMX RMI port to 7199 (CASSANDRA-7087)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java
----------------------------------------------------------------------
diff --git a/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java b/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java
new file mode 100644
index 0000000..bf032f5
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.service;
+
+
+import java.net.InetAddress;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.cassandra.gms.FailureDetector;
+import org.apache.cassandra.utils.FBUtilities;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Multimap;
+
+public class BatchlogEndpointSelector
+{
+ private final String localRack;
+
+ public BatchlogEndpointSelector(String localRack)
+ {
+ this.localRack = localRack;
+ }
+
+ /**
+ * @param endpoints nodes in the local datacenter, grouped by rack name
+ * @return list of candidates for batchlog hosting. if possible these will be two nodes from different racks.
+ */
+ public Collection<InetAddress> chooseEndpoints(Multimap<String, InetAddress> endpoints)
+ {
+ // strip out dead endpoints and localhost
+ ListMultimap<String, InetAddress> validated = ArrayListMultimap.create();
+ for (Map.Entry<String, InetAddress> entry : endpoints.entries())
+ {
+ if (isValid(entry.getValue()))
+ validated.put(entry.getKey(), entry.getValue());
+ }
+ if (validated.size() <= 2)
+ return validated.values();
+
+ if ((validated.size() - validated.get(localRack).size()) >= 2)
+ {
+ // we have enough endpoints in other racks
+ validated.removeAll(localRack);
+ }
+
+ if (validated.keySet().size() == 1)
+ {
+ // we have only 1 `other` rack
+ Collection<InetAddress> otherRack = Iterables.getOnlyElement(validated.asMap().values());
+ return Lists.newArrayList(Iterables.limit(otherRack, 2));
+ }
+
+ // randomize which racks we pick from if more than 2 remaining
+ Collection<String> racks;
+ if (validated.keySet().size() == 2)
+ {
+ racks = validated.keySet();
+ }
+ else
+ {
+ racks = Lists.newArrayList(validated.keySet());
+ Collections.shuffle((List) racks);
+ }
+
+ // grab a random member of up to two racks
+ List<InetAddress> result = new ArrayList<>(2);
+ for (String rack : Iterables.limit(racks, 2))
+ {
+ List<InetAddress> rackMembers = validated.get(rack);
+ result.add(rackMembers.get(getRandomInt(rackMembers.size())));
+ }
+
+ return result;
+ }
+
+ @VisibleForTesting
+ protected boolean isValid(InetAddress input)
+ {
+ return !input.equals(FBUtilities.getBroadcastAddress()) && FailureDetector.instance.isAlive(input);
+ }
+
+ @VisibleForTesting
+ protected int getRandomInt(int bound)
+ {
+ return FBUtilities.threadLocalRandom().nextInt(bound);
+ }
+}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/src/java/org/apache/cassandra/service/StorageProxy.java
----------------------------------------------------------------------
diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java b/src/java/org/apache/cassandra/service/StorageProxy.java
index 14d5ee2..2bf8e7f 100644
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@ -22,7 +22,6 @@ import java.lang.management.ManagementFactory;
import java.net.InetAddress;
import java.nio.ByteBuffer;
import java.util.*;
-import java.util.Random;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import javax.management.MBeanServer;
@@ -729,24 +728,14 @@ public class StorageProxy implements StorageProxyMBean
throws UnavailableException
{
TokenMetadata.Topology topology = StorageService.instance.getTokenMetadata().cachedOnlyTokenMap().getTopology();
- List<InetAddress> localEndpoints = new ArrayList<>(topology.getDatacenterEndpoints().get(localDataCenter));
-
+ Multimap<String, InetAddress> localEndpoints = HashMultimap.create(topology.getDatacenterRacks().get(localDataCenter));
+
// special case for single-node datacenters
if (localEndpoints.size() == 1)
- return localEndpoints;
-
- List<InetAddress> chosenEndpoints = new ArrayList<>(2);
- int startOffset = new Random().nextInt(localEndpoints.size());
+ return localEndpoints.values();
- // starts at some random point in the list, advances forward until the end, then loops
- // around to the beginning, advancing again until it is back at the starting point again.
- for (int i = 0; i < localEndpoints.size() && chosenEndpoints.size() < 2; i++)
- {
- InetAddress endpoint = localEndpoints.get((i + startOffset) % localEndpoints.size());
- // skip localhost and non-alive nodes
- if (!endpoint.equals(FBUtilities.getBroadcastAddress()) && FailureDetector.instance.isAlive(endpoint))
- chosenEndpoints.add(endpoint);
- }
+ String localRack = DatabaseDescriptor.getEndpointSnitch().getRack(FBUtilities.getBroadcastAddress());
+ Collection<InetAddress> chosenEndpoints = new BatchlogEndpointSelector(localRack).chooseEndpoints(localEndpoints);
if (chosenEndpoints.isEmpty())
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java
----------------------------------------------------------------------
diff --git a/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java b/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java
new file mode 100644
index 0000000..293078d
--- /dev/null
+++ b/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.service;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.junit.Assert.assertThat;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.Collection;
+
+import org.junit.Test;
+import org.junit.matchers.JUnitMatchers;
+
+import com.google.common.collect.ImmutableMultimap;
+import com.google.common.collect.Multimap;
+
+public class BatchlogEndpointSelectorTest
+{
+ private final BatchlogEndpointSelector target;
+ private static final String LOCAL = "local";
+
+
+ public BatchlogEndpointSelectorTest() throws UnknownHostException
+ {
+ target = new BatchlogEndpointSelector(LOCAL)
+ {
+ @Override
+ protected boolean isValid(InetAddress input)
+ {
+ //we will use always alive non-localhost endpoints
+ return true;
+ }
+
+ @Override
+ protected int getRandomInt(int bound)
+ {
+ //we don't need a random behavior here
+ return bound - 1;
+ }
+ };
+ }
+
+ @Test
+ public void shouldSelect2hostsFromNonLocalRacks() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .put(LOCAL, InetAddress.getByName("00"))
+ .put("1", InetAddress.getByName("1"))
+ .put("1", InetAddress.getByName("11"))
+ .put("2", InetAddress.getByName("2"))
+ .put("2", InetAddress.getByName("22"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(2));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("11")));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("22")));
+ }
+
+ @Test
+ public void shouldSelectHostFromLocal() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .put(LOCAL, InetAddress.getByName("00"))
+ .put("1", InetAddress.getByName("1"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(2));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("1")));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("0")));
+ }
+
+ @Test
+ public void shouldReturnAsIsIfNoEnoughEndpoints() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(1));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("0")));
+ }
+
+ @Test
+ public void shouldSelectTwoFirstHostsFromSingleOtherRack() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .put(LOCAL, InetAddress.getByName("00"))
+ .put("1", InetAddress.getByName("1"))
+ .put("1", InetAddress.getByName("11"))
+ .put("1", InetAddress.getByName("111"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(2));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("1")));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("11")));
+ }
+}
[5/9] git commit: Make batchlog replica selection rack-aware patch by
Mikhail Stepura; reviewed by jbellis for CASSANDRA-6551
Posted by jb...@apache.org.
Make batchlog replica selection rack-aware
patch by Mikhail Stepura; reviewed by jbellis for CASSANDRA-6551
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/af96d405
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/af96d405
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/af96d405
Branch: refs/heads/cassandra-2.1
Commit: af96d405b42a9e4ae23cba841b7a5d83ee8f7ec8
Parents: fab4557
Author: Jonathan Ellis <jb...@apache.org>
Authored: Fri May 2 22:47:27 2014 -0500
Committer: Jonathan Ellis <jb...@apache.org>
Committed: Fri May 2 22:47:27 2014 -0500
----------------------------------------------------------------------
CHANGES.txt | 2 +-
.../service/BatchlogEndpointSelector.java | 110 ++++++++++++++++++
.../apache/cassandra/service/StorageProxy.java | 21 +---
.../service/BatchlogEndpointSelectorTest.java | 116 +++++++++++++++++++
4 files changed, 232 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 9ab1a5f..5799659 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,5 @@
2.0.8
-=======
+ * Make batchlog replica selection rack-aware (CASSANDRA-6551)
* Add Google Compute Engine snitch (CASSANDRA-7132)
* Allow overriding cassandra-rackdc.properties file (CASSANDRA-7072)
* Set JMX RMI port to 7199 (CASSANDRA-7087)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java
----------------------------------------------------------------------
diff --git a/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java b/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java
new file mode 100644
index 0000000..bf032f5
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.service;
+
+
+import java.net.InetAddress;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.cassandra.gms.FailureDetector;
+import org.apache.cassandra.utils.FBUtilities;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Multimap;
+
+public class BatchlogEndpointSelector
+{
+ private final String localRack;
+
+ public BatchlogEndpointSelector(String localRack)
+ {
+ this.localRack = localRack;
+ }
+
+ /**
+ * @param endpoints nodes in the local datacenter, grouped by rack name
+ * @return list of candidates for batchlog hosting. if possible these will be two nodes from different racks.
+ */
+ public Collection<InetAddress> chooseEndpoints(Multimap<String, InetAddress> endpoints)
+ {
+ // strip out dead endpoints and localhost
+ ListMultimap<String, InetAddress> validated = ArrayListMultimap.create();
+ for (Map.Entry<String, InetAddress> entry : endpoints.entries())
+ {
+ if (isValid(entry.getValue()))
+ validated.put(entry.getKey(), entry.getValue());
+ }
+ if (validated.size() <= 2)
+ return validated.values();
+
+ if ((validated.size() - validated.get(localRack).size()) >= 2)
+ {
+ // we have enough endpoints in other racks
+ validated.removeAll(localRack);
+ }
+
+ if (validated.keySet().size() == 1)
+ {
+ // we have only 1 `other` rack
+ Collection<InetAddress> otherRack = Iterables.getOnlyElement(validated.asMap().values());
+ return Lists.newArrayList(Iterables.limit(otherRack, 2));
+ }
+
+ // randomize which racks we pick from if more than 2 remaining
+ Collection<String> racks;
+ if (validated.keySet().size() == 2)
+ {
+ racks = validated.keySet();
+ }
+ else
+ {
+ racks = Lists.newArrayList(validated.keySet());
+ Collections.shuffle((List) racks);
+ }
+
+ // grab a random member of up to two racks
+ List<InetAddress> result = new ArrayList<>(2);
+ for (String rack : Iterables.limit(racks, 2))
+ {
+ List<InetAddress> rackMembers = validated.get(rack);
+ result.add(rackMembers.get(getRandomInt(rackMembers.size())));
+ }
+
+ return result;
+ }
+
+ @VisibleForTesting
+ protected boolean isValid(InetAddress input)
+ {
+ return !input.equals(FBUtilities.getBroadcastAddress()) && FailureDetector.instance.isAlive(input);
+ }
+
+ @VisibleForTesting
+ protected int getRandomInt(int bound)
+ {
+ return FBUtilities.threadLocalRandom().nextInt(bound);
+ }
+}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/src/java/org/apache/cassandra/service/StorageProxy.java
----------------------------------------------------------------------
diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java b/src/java/org/apache/cassandra/service/StorageProxy.java
index 14d5ee2..2bf8e7f 100644
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@ -22,7 +22,6 @@ import java.lang.management.ManagementFactory;
import java.net.InetAddress;
import java.nio.ByteBuffer;
import java.util.*;
-import java.util.Random;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import javax.management.MBeanServer;
@@ -729,24 +728,14 @@ public class StorageProxy implements StorageProxyMBean
throws UnavailableException
{
TokenMetadata.Topology topology = StorageService.instance.getTokenMetadata().cachedOnlyTokenMap().getTopology();
- List<InetAddress> localEndpoints = new ArrayList<>(topology.getDatacenterEndpoints().get(localDataCenter));
-
+ Multimap<String, InetAddress> localEndpoints = HashMultimap.create(topology.getDatacenterRacks().get(localDataCenter));
+
// special case for single-node datacenters
if (localEndpoints.size() == 1)
- return localEndpoints;
-
- List<InetAddress> chosenEndpoints = new ArrayList<>(2);
- int startOffset = new Random().nextInt(localEndpoints.size());
+ return localEndpoints.values();
- // starts at some random point in the list, advances forward until the end, then loops
- // around to the beginning, advancing again until it is back at the starting point again.
- for (int i = 0; i < localEndpoints.size() && chosenEndpoints.size() < 2; i++)
- {
- InetAddress endpoint = localEndpoints.get((i + startOffset) % localEndpoints.size());
- // skip localhost and non-alive nodes
- if (!endpoint.equals(FBUtilities.getBroadcastAddress()) && FailureDetector.instance.isAlive(endpoint))
- chosenEndpoints.add(endpoint);
- }
+ String localRack = DatabaseDescriptor.getEndpointSnitch().getRack(FBUtilities.getBroadcastAddress());
+ Collection<InetAddress> chosenEndpoints = new BatchlogEndpointSelector(localRack).chooseEndpoints(localEndpoints);
if (chosenEndpoints.isEmpty())
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/af96d405/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java
----------------------------------------------------------------------
diff --git a/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java b/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java
new file mode 100644
index 0000000..293078d
--- /dev/null
+++ b/test/unit/org/apache/cassandra/service/BatchlogEndpointSelectorTest.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.service;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.junit.Assert.assertThat;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.Collection;
+
+import org.junit.Test;
+import org.junit.matchers.JUnitMatchers;
+
+import com.google.common.collect.ImmutableMultimap;
+import com.google.common.collect.Multimap;
+
+public class BatchlogEndpointSelectorTest
+{
+ private final BatchlogEndpointSelector target;
+ private static final String LOCAL = "local";
+
+
+ public BatchlogEndpointSelectorTest() throws UnknownHostException
+ {
+ target = new BatchlogEndpointSelector(LOCAL)
+ {
+ @Override
+ protected boolean isValid(InetAddress input)
+ {
+ //we will use always alive non-localhost endpoints
+ return true;
+ }
+
+ @Override
+ protected int getRandomInt(int bound)
+ {
+ //we don't need a random behavior here
+ return bound - 1;
+ }
+ };
+ }
+
+ @Test
+ public void shouldSelect2hostsFromNonLocalRacks() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .put(LOCAL, InetAddress.getByName("00"))
+ .put("1", InetAddress.getByName("1"))
+ .put("1", InetAddress.getByName("11"))
+ .put("2", InetAddress.getByName("2"))
+ .put("2", InetAddress.getByName("22"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(2));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("11")));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("22")));
+ }
+
+ @Test
+ public void shouldSelectHostFromLocal() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .put(LOCAL, InetAddress.getByName("00"))
+ .put("1", InetAddress.getByName("1"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(2));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("1")));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("0")));
+ }
+
+ @Test
+ public void shouldReturnAsIsIfNoEnoughEndpoints() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(1));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("0")));
+ }
+
+ @Test
+ public void shouldSelectTwoFirstHostsFromSingleOtherRack() throws UnknownHostException
+ {
+ Multimap<String, InetAddress> endpoints = ImmutableMultimap.<String, InetAddress> builder()
+ .put(LOCAL, InetAddress.getByName("0"))
+ .put(LOCAL, InetAddress.getByName("00"))
+ .put("1", InetAddress.getByName("1"))
+ .put("1", InetAddress.getByName("11"))
+ .put("1", InetAddress.getByName("111"))
+ .build();
+ Collection<InetAddress> result = target.chooseEndpoints(endpoints);
+ assertThat(result.size(), is(2));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("1")));
+ assertThat(result, JUnitMatchers.hasItem(InetAddress.getByName("11")));
+ }
+}
[8/9] git commit: merge from 2.1
Posted by jb...@apache.org.
merge from 2.1
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8e897de6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8e897de6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8e897de6
Branch: refs/heads/trunk
Commit: 8e897de68716e47c0341dc6aaa94a2ab4a7e519c
Parents: 9432870 af96d40
Author: Jonathan Ellis <jb...@apache.org>
Authored: Fri May 2 22:49:31 2014 -0500
Committer: Jonathan Ellis <jb...@apache.org>
Committed: Fri May 2 22:49:31 2014 -0500
----------------------------------------------------------------------
CHANGES.txt | 4 +-
build.xml | 2 +
.../service/BatchlogEndpointSelector.java | 110 ++++++++++++++++++
.../apache/cassandra/service/StorageProxy.java | 21 +---
.../service/BatchlogEndpointSelectorTest.java | 116 +++++++++++++++++++
5 files changed, 236 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8e897de6/CHANGES.txt
----------------------------------------------------------------------
diff --cc CHANGES.txt
index 82b0196,5799659..eece557
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,70 -1,6 +1,71 @@@
-2.0.8
+2.1.0-rc1
+ * Parallel streaming for sstableloader (CASSANDRA-3668)
+Merged from 2.0:
+ * Make batchlog replica selection rack-aware (CASSANDRA-6551)
* Add Google Compute Engine snitch (CASSANDRA-7132)
- Merged from 1.2:
+ * Update system.peers correctly when relocating tokens (CASSANDRA-7126)
+
++
+2.1.0-beta2
+ * Increase default CL space to 8GB (CASSANDRA-7031)
+ * Add range tombstones to read repair digests (CASSANDRA-6863)
+ * Fix BTree.clear for large updates (CASSANDRA-6943)
+ * Fail write instead of logging a warning when unable to append to CL
+ (CASSANDRA-6764)
+ * Eliminate possibility of CL segment appearing twice in active list
+ (CASSANDRA-6557)
+ * Apply DONTNEED fadvise to commitlog segments (CASSANDRA-6759)
+ * Switch CRC component to Adler and include it for compressed sstables
+ (CASSANDRA-4165)
+ * Allow cassandra-stress to set compaction strategy options (CASSANDRA-6451)
+ * Add broadcast_rpc_address option to cassandra.yaml (CASSANDRA-5899)
+ * Auto reload GossipingPropertyFileSnitch config (CASSANDRA-5897)
+ * Fix overflow of memtable_total_space_in_mb (CASSANDRA-6573)
+ * Fix ABTC NPE and apply update function correctly (CASSANDRA-6692)
+ * Allow nodetool to use a file or prompt for password (CASSANDRA-6660)
+ * Fix AIOOBE when concurrently accessing ABSC (CASSANDRA-6742)
+ * Fix assertion error in ALTER TYPE RENAME (CASSANDRA-6705)
+ * Scrub should not always clear out repaired status (CASSANDRA-5351)
+ * Improve handling of range tombstone for wide partitions (CASSANDRA-6446)
+ * Fix ClassCastException for compact table with composites (CASSANDRA-6738)
+ * Fix potentially repairing with wrong nodes (CASSANDRA-6808)
+ * Change caching option syntax (CASSANDRA-6745)
+ * Fix stress to do proper counter reads (CASSANDRA-6835)
+ * Fix help message for stress counter_write (CASSANDRA-6824)
+ * Fix stress smart Thrift client to pick servers correctly (CASSANDRA-6848)
+ * Add logging levels (minimal, normal or verbose) to stress tool (CASSANDRA-6849)
+ * Fix race condition in Batch CLE (CASSANDRA-6860)
+ * Improve cleanup/scrub/upgradesstables failure handling (CASSANDRA-6774)
+ * ByteBuffer write() methods for serializing sstables (CASSANDRA-6781)
+ * Proper compare function for CollectionType (CASSANDRA-6783)
+ * Update native server to Netty 4 (CASSANDRA-6236)
+ * Fix off-by-one error in stress (CASSANDRA-6883)
+ * Make OpOrder AutoCloseable (CASSANDRA-6901)
+ * Remove sync repair JMX interface (CASSANDRA-6900)
+ * Add multiple memory allocation options for memtables (CASSANDRA-6689, 6694)
+ * Remove adjusted op rate from stress output (CASSANDRA-6921)
+ * Add optimized CF.hasColumns() implementations (CASSANDRA-6941)
+ * Serialize batchlog mutations with the version of the target node
+ (CASSANDRA-6931)
+ * Optimize CounterColumn#reconcile() (CASSANDRA-6953)
+ * Properly remove 1.2 sstable support in 2.1 (CASSANDRA-6869)
+ * Lock counter cells, not partitions (CASSANDRA-6880)
+ * Track presence of legacy counter shards in sstables (CASSANDRA-6888)
+ * Ensure safe resource cleanup when replacing sstables (CASSANDRA-6912)
+ * Add failure handler to async callback (CASSANDRA-6747)
+ * Fix AE when closing SSTable without releasing reference (CASSANDRA-7000)
+ * Clean up IndexInfo on keyspace/table drops (CASSANDRA-6924)
+ * Only snapshot relative SSTables when sequential repair (CASSANDRA-7024)
+ * Require nodetool rebuild_index to specify index names (CASSANDRA-7038)
+ * fix cassandra stress errors on reads with native protocol (CASSANDRA-7033)
+ * Use OpOrder to guard sstable references for reads (CASSANDRA-6919)
+ * Preemptive opening of compaction result (CASSANDRA-6916)
+ * Multi-threaded scrub/cleanup/upgradesstables (CASSANDRA-5547)
+ * Optimize cellname comparison (CASSANDRA-6934)
+ * Native protocol v3 (CASSANDRA-6855)
+ * Optimize Cell liveness checks and clean up Cell (CASSANDRA-7119)
+ * Support consistent range movements (CASSANDRA-2434)
+Merged from 2.0:
* Allow overriding cassandra-rackdc.properties file (CASSANDRA-7072)
* Set JMX RMI port to 7199 (CASSANDRA-7087)
* Use LOCAL_QUORUM for data reads at LOCAL_SERIAL (CASSANDRA-6939)
@@@ -169,13 -95,9 +170,14 @@@ Merged from 1.2
* Schedule schema pulls on change (CASSANDRA-6971)
* Non-droppable verbs shouldn't be dropped from OTC (CASSANDRA-6980)
* Shutdown batchlog executor in SS#drain() (CASSANDRA-7025)
+ * Fix batchlog to account for CF truncation records (CASSANDRA-6999)
+ * Fix CQLSH parsing of functions and BLOB literals (CASSANDRA-7018)
* Properly load trustore in the native protocol (CASSANDRA-6847)
+ * Always clean up references in SerializingCache (CASSANDRA-6994)
+ * Don't shut MessagingService down when replacing a node (CASSANDRA-6476)
+ * fix npe when doing -Dcassandra.fd_initial_value_ms (CASSANDRA-6751)
+
2.0.6
* Avoid race-prone second "scrub" of system keyspace (CASSANDRA-6797)
* Pool CqlRecordWriter clients by inetaddress rather than Range
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8e897de6/build.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8e897de6/src/java/org/apache/cassandra/service/StorageProxy.java
----------------------------------------------------------------------