You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by brian-lininger <gi...@git.apache.org> on 2017/11/16 18:03:07 UTC
[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...
GitHub user brian-lininger opened a pull request:
https://github.com/apache/zookeeper/pull/419
ZOOKEEPER-2849: Exponential back-off retry for Quorum port binding
Added BackoffStrategy interface with ExponentialBackoffStrategy as a concrete implementation. Enhanced
QuorumCnxManager to utilize the provided BackoffStrategy to determine retry of port binding as opposed to
a fixed interval/count.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/brian-lininger/zookeeper ZOOKEEPER-2849
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zookeeper/pull/419.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #419
----
commit 0675e09a5d6122bff6b6543d80076b03d05d5f85
Author: Brian Lininger <br...@veeva.com>
Date: 2017-11-16T17:58:13Z
ZOOKEEPER-2849:
Added BackoffStrategy interface with ExponentialBackoffStrategy as a concrete implementation. Enhanced
QuorumCnxManager to utilize the provided BackoffStrategy to determine retry of port binding as opposed to
a fixed interval/count.
----
---
[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...
Posted by afine <gi...@git.apache.org>.
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/419#discussion_r151812103
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/ExponentialBackoffStrategy.java ---
@@ -0,0 +1,192 @@
+package org.apache.zookeeper.server.quorum;
+
+/**
+ * A {@link BackoffStrategy} that increases the wait time between each
+ * interval up to the configured maximum wait time.
+ */
+public class ExponentialBackoffStrategy implements BackoffStrategy {
+
+ // Sensible default values to use if not set by the user
+ private static final long DEFAULT_INITIAL_BACKOFF_MILLIS = 500L; // 0.5s
+ private static final long DEFAULT_MAX_BACKOFF_MILLIS = 30_000L; // 30s
+ private static final long DEFAULT_MAX_ELAPSED_MILLIS = 5 * 60_000L; // 10m
+ private static final double DEFAULT_BACKOFF_MULTIPLE = 1.5;
+
+ // internal values per instance
+ private final long initialBackoffMillis;
+ private final long maxBackoffMillis;
+ private final long maxElapsedMillis;
+ private final double backoffMultiple;
+
+ // internal state
+ private long nextWait;
+ private long totalElapsed;
+ private final boolean limitBackoffMillis;
+ private final boolean checkElapsedTime;
+
+ /**
+ * Construct a new instance.
+ * @param builder the Builder to use for configuring this BackoffStrategy
+ */
+ private ExponentialBackoffStrategy(Builder builder) {
+ this.initialBackoffMillis = builder.initialBackoffMillis;
+ this.maxBackoffMillis = builder.maxBackoffMillis;
+ this.maxElapsedMillis = builder.maxElapsedMillis;
+ this.backoffMultiple = builder.backoffMultiple;
+
+ if(maxBackoffMillis == -1) {
+ limitBackoffMillis = false;
+ } else {
+ limitBackoffMillis = true;
+ }
+
+ if(maxElapsedMillis == -1) {
+ checkElapsedTime = false;
+ } else {
+ checkElapsedTime = true;
+ }
+
+ reset();
+ }
+
+
+ @Override
+ public long nextWaitMillis() throws IllegalStateException {
+ // check if we have exceeded the allowed maximum elapsed time
+ if(checkElapsedTime && totalElapsed > maxElapsedMillis) {
+ return BackoffStrategy.STOP;
+ }
+
+ long waitMillis = nextWait;
+
+ // calculate the next wait milliseconds
+ nextWait = Math.round(nextWait * backoffMultiple);
+
+ // don't exceed the allowed maximum wait milliseconds
+ // if a maximum was configured
+ if(limitBackoffMillis && nextWait > maxBackoffMillis) {
+ nextWait = maxBackoffMillis;
+ }
+
+ // track total elapsed time, even if we don't wait we have to assume
+ // that some amount of time passed outside of the wait or we'll never
+ // hit the elapsed time limit
+ totalElapsed += waitMillis != 0 ? waitMillis : 1L;
+ return waitMillis;
+ }
+
+ @Override
+ public void reset() {
+ nextWait = this.initialBackoffMillis;
+ totalElapsed = 0;
+ }
+
+ /**
+ *
+ * @return a new {@link Builder} instance.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Builder for instances of {@link ExponentialBackoffStrategy}.
+ */
+ public static final class Builder {
--- End diff --
So I probably wasn't clear here. It would be great if it was also possible, since backoff strategy is such a nicely defined interface, if the user could include jar with their own backoff strategy in the classpath and tell zookeeper to use that. What do you think?
---
[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...
Posted by brian-lininger <gi...@git.apache.org>.
Github user brian-lininger commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/419#discussion_r152119989
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/ExponentialBackoffStrategy.java ---
@@ -0,0 +1,192 @@
+package org.apache.zookeeper.server.quorum;
+
+/**
+ * A {@link BackoffStrategy} that increases the wait time between each
+ * interval up to the configured maximum wait time.
+ */
+public class ExponentialBackoffStrategy implements BackoffStrategy {
+
+ // Sensible default values to use if not set by the user
+ private static final long DEFAULT_INITIAL_BACKOFF_MILLIS = 500L; // 0.5s
+ private static final long DEFAULT_MAX_BACKOFF_MILLIS = 30_000L; // 30s
+ private static final long DEFAULT_MAX_ELAPSED_MILLIS = 5 * 60_000L; // 10m
+ private static final double DEFAULT_BACKOFF_MULTIPLE = 1.5;
+
+ // internal values per instance
+ private final long initialBackoffMillis;
+ private final long maxBackoffMillis;
+ private final long maxElapsedMillis;
+ private final double backoffMultiple;
+
+ // internal state
+ private long nextWait;
+ private long totalElapsed;
+ private final boolean limitBackoffMillis;
+ private final boolean checkElapsedTime;
+
+ /**
+ * Construct a new instance.
+ * @param builder the Builder to use for configuring this BackoffStrategy
+ */
+ private ExponentialBackoffStrategy(Builder builder) {
+ this.initialBackoffMillis = builder.initialBackoffMillis;
+ this.maxBackoffMillis = builder.maxBackoffMillis;
+ this.maxElapsedMillis = builder.maxElapsedMillis;
+ this.backoffMultiple = builder.backoffMultiple;
+
+ if(maxBackoffMillis == -1) {
+ limitBackoffMillis = false;
+ } else {
+ limitBackoffMillis = true;
+ }
+
+ if(maxElapsedMillis == -1) {
+ checkElapsedTime = false;
+ } else {
+ checkElapsedTime = true;
+ }
+
+ reset();
+ }
+
+
+ @Override
+ public long nextWaitMillis() throws IllegalStateException {
+ // check if we have exceeded the allowed maximum elapsed time
+ if(checkElapsedTime && totalElapsed > maxElapsedMillis) {
+ return BackoffStrategy.STOP;
+ }
+
+ long waitMillis = nextWait;
+
+ // calculate the next wait milliseconds
+ nextWait = Math.round(nextWait * backoffMultiple);
+
+ // don't exceed the allowed maximum wait milliseconds
+ // if a maximum was configured
+ if(limitBackoffMillis && nextWait > maxBackoffMillis) {
+ nextWait = maxBackoffMillis;
+ }
+
+ // track total elapsed time, even if we don't wait we have to assume
+ // that some amount of time passed outside of the wait or we'll never
+ // hit the elapsed time limit
+ totalElapsed += waitMillis != 0 ? waitMillis : 1L;
+ return waitMillis;
+ }
+
+ @Override
+ public void reset() {
+ nextWait = this.initialBackoffMillis;
+ totalElapsed = 0;
+ }
+
+ /**
+ *
+ * @return a new {@link Builder} instance.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Builder for instances of {@link ExponentialBackoffStrategy}.
+ */
+ public static final class Builder {
--- End diff --
That should be doable.
---
[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...
Posted by afine <gi...@git.apache.org>.
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/419#discussion_r151560463
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java ---
@@ -946,7 +946,8 @@ protected Election createElectionAlgorithm(int electionAlgorithm){
le = new AuthFastLeaderElection(this, true);
break;
case 3:
- qcm = new QuorumCnxManager(this);
+ qcm = new QuorumCnxManager(this,
+ ExponentialBackoffStrategy.builder().build());
--- End diff --
For instance, it may be nice to have the default strategy perform the existing behavior.
---
[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...
Posted by afine <gi...@git.apache.org>.
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/419#discussion_r151553425
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/ExponentialBackoffStrategy.java ---
@@ -0,0 +1,192 @@
+package org.apache.zookeeper.server.quorum;
+
+/**
+ * A {@link BackoffStrategy} that increases the wait time between each
+ * interval up to the configured maximum wait time.
+ */
+public class ExponentialBackoffStrategy implements BackoffStrategy {
+
+ // Sensible default values to use if not set by the user
+ private static final long DEFAULT_INITIAL_BACKOFF_MILLIS = 500L; // 0.5s
+ private static final long DEFAULT_MAX_BACKOFF_MILLIS = 30_000L; // 30s
+ private static final long DEFAULT_MAX_ELAPSED_MILLIS = 5 * 60_000L; // 10m
+ private static final double DEFAULT_BACKOFF_MULTIPLE = 1.5;
+
+ // internal values per instance
+ private final long initialBackoffMillis;
+ private final long maxBackoffMillis;
+ private final long maxElapsedMillis;
+ private final double backoffMultiple;
+
+ // internal state
+ private long nextWait;
+ private long totalElapsed;
+ private final boolean limitBackoffMillis;
+ private final boolean checkElapsedTime;
+
+ /**
+ * Construct a new instance.
+ * @param builder the Builder to use for configuring this BackoffStrategy
+ */
+ private ExponentialBackoffStrategy(Builder builder) {
+ this.initialBackoffMillis = builder.initialBackoffMillis;
+ this.maxBackoffMillis = builder.maxBackoffMillis;
+ this.maxElapsedMillis = builder.maxElapsedMillis;
+ this.backoffMultiple = builder.backoffMultiple;
+
+ if(maxBackoffMillis == -1) {
+ limitBackoffMillis = false;
+ } else {
+ limitBackoffMillis = true;
+ }
+
+ if(maxElapsedMillis == -1) {
+ checkElapsedTime = false;
+ } else {
+ checkElapsedTime = true;
+ }
+
+ reset();
+ }
+
+
+ @Override
+ public long nextWaitMillis() throws IllegalStateException {
+ // check if we have exceeded the allowed maximum elapsed time
+ if(checkElapsedTime && totalElapsed > maxElapsedMillis) {
+ return BackoffStrategy.STOP;
+ }
+
+ long waitMillis = nextWait;
+
+ // calculate the next wait milliseconds
+ nextWait = Math.round(nextWait * backoffMultiple);
+
+ // don't exceed the allowed maximum wait milliseconds
+ // if a maximum was configured
+ if(limitBackoffMillis && nextWait > maxBackoffMillis) {
+ nextWait = maxBackoffMillis;
+ }
+
+ // track total elapsed time, even if we don't wait we have to assume
+ // that some amount of time passed outside of the wait or we'll never
+ // hit the elapsed time limit
+ totalElapsed += waitMillis != 0 ? waitMillis : 1L;
+ return waitMillis;
+ }
+
+ @Override
+ public void reset() {
+ nextWait = this.initialBackoffMillis;
+ totalElapsed = 0;
+ }
+
+ /**
+ *
+ * @return a new {@link Builder} instance.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Builder for instances of {@link ExponentialBackoffStrategy}.
+ */
+ public static final class Builder {
--- End diff --
I'm not sure a builder is the best way to handle this. Since I think it would be nice to have much of this be user configurable, perhaps we can just pull the config from system properties?
---
[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...
Posted by afine <gi...@git.apache.org>.
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/419#discussion_r151532589
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java ---
@@ -946,7 +946,8 @@ protected Election createElectionAlgorithm(int electionAlgorithm){
le = new AuthFastLeaderElection(this, true);
break;
case 3:
- qcm = new QuorumCnxManager(this);
+ qcm = new QuorumCnxManager(this,
+ ExponentialBackoffStrategy.builder().build());
--- End diff --
Perhaps we should find a way to make this pluggable? Perhaps we can read which backoff strategy to use from a java system property?
---
[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...
Posted by afine <gi...@git.apache.org>.
Github user afine commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/419#discussion_r151560170
--- Diff: src/java/test/org/apache/zookeeper/server/quorum/ExponentialBackoffStrategyTest.java ---
@@ -0,0 +1,180 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zookeeper.server.quorum;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+
+import org.junit.Test;
+
+/**
+ * Unit tests for {@link ExponentialBackoffStrategy}.
+ */
+public class ExponentialBackoffStrategyTest {
--- End diff --
This should extend ZKTestCase
---
[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...
Posted by brian-lininger <gi...@git.apache.org>.
Github user brian-lininger commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/419#discussion_r151766248
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/ExponentialBackoffStrategy.java ---
@@ -0,0 +1,192 @@
+package org.apache.zookeeper.server.quorum;
+
+/**
+ * A {@link BackoffStrategy} that increases the wait time between each
+ * interval up to the configured maximum wait time.
+ */
+public class ExponentialBackoffStrategy implements BackoffStrategy {
+
+ // Sensible default values to use if not set by the user
+ private static final long DEFAULT_INITIAL_BACKOFF_MILLIS = 500L; // 0.5s
+ private static final long DEFAULT_MAX_BACKOFF_MILLIS = 30_000L; // 30s
+ private static final long DEFAULT_MAX_ELAPSED_MILLIS = 5 * 60_000L; // 10m
+ private static final double DEFAULT_BACKOFF_MULTIPLE = 1.5;
+
+ // internal values per instance
+ private final long initialBackoffMillis;
+ private final long maxBackoffMillis;
+ private final long maxElapsedMillis;
+ private final double backoffMultiple;
+
+ // internal state
+ private long nextWait;
+ private long totalElapsed;
+ private final boolean limitBackoffMillis;
+ private final boolean checkElapsedTime;
+
+ /**
+ * Construct a new instance.
+ * @param builder the Builder to use for configuring this BackoffStrategy
+ */
+ private ExponentialBackoffStrategy(Builder builder) {
+ this.initialBackoffMillis = builder.initialBackoffMillis;
+ this.maxBackoffMillis = builder.maxBackoffMillis;
+ this.maxElapsedMillis = builder.maxElapsedMillis;
+ this.backoffMultiple = builder.backoffMultiple;
+
+ if(maxBackoffMillis == -1) {
+ limitBackoffMillis = false;
+ } else {
+ limitBackoffMillis = true;
+ }
+
+ if(maxElapsedMillis == -1) {
+ checkElapsedTime = false;
+ } else {
+ checkElapsedTime = true;
+ }
+
+ reset();
+ }
+
+
+ @Override
+ public long nextWaitMillis() throws IllegalStateException {
+ // check if we have exceeded the allowed maximum elapsed time
+ if(checkElapsedTime && totalElapsed > maxElapsedMillis) {
+ return BackoffStrategy.STOP;
+ }
+
+ long waitMillis = nextWait;
+
+ // calculate the next wait milliseconds
+ nextWait = Math.round(nextWait * backoffMultiple);
+
+ // don't exceed the allowed maximum wait milliseconds
+ // if a maximum was configured
+ if(limitBackoffMillis && nextWait > maxBackoffMillis) {
+ nextWait = maxBackoffMillis;
+ }
+
+ // track total elapsed time, even if we don't wait we have to assume
+ // that some amount of time passed outside of the wait or we'll never
+ // hit the elapsed time limit
+ totalElapsed += waitMillis != 0 ? waitMillis : 1L;
+ return waitMillis;
+ }
+
+ @Override
+ public void reset() {
+ nextWait = this.initialBackoffMillis;
+ totalElapsed = 0;
+ }
+
+ /**
+ *
+ * @return a new {@link Builder} instance.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Builder for instances of {@link ExponentialBackoffStrategy}.
+ */
+ public static final class Builder {
--- End diff --
I agree that it should be configurable, but I don't think using a Builder pattern precludes this. I'm about to push changes that adds support for using system properties, that's a good idea.
---
[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...
Posted by brian-lininger <gi...@git.apache.org>.
Github user brian-lininger commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/419#discussion_r151766328
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java ---
@@ -946,7 +946,8 @@ protected Election createElectionAlgorithm(int electionAlgorithm){
le = new AuthFastLeaderElection(this, true);
break;
case 3:
- qcm = new QuorumCnxManager(this);
+ qcm = new QuorumCnxManager(this,
+ ExponentialBackoffStrategy.builder().build());
--- End diff --
Agreed, included in my upcoming push.
---