You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by brian-lininger <gi...@git.apache.org> on 2017/11/16 18:03:07 UTC

[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...

GitHub user brian-lininger opened a pull request:

    https://github.com/apache/zookeeper/pull/419

    ZOOKEEPER-2849: Exponential back-off retry for Quorum port binding

      Added BackoffStrategy interface with ExponentialBackoffStrategy as a concrete implementation. Enhanced
      QuorumCnxManager to utilize the provided BackoffStrategy to determine retry of port binding as opposed to
      a fixed interval/count.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/brian-lininger/zookeeper ZOOKEEPER-2849

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zookeeper/pull/419.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #419
    
----
commit 0675e09a5d6122bff6b6543d80076b03d05d5f85
Author: Brian Lininger <br...@veeva.com>
Date:   2017-11-16T17:58:13Z

    ZOOKEEPER-2849:
      Added BackoffStrategy interface with ExponentialBackoffStrategy as a concrete implementation. Enhanced
      QuorumCnxManager to utilize the provided BackoffStrategy to determine retry of port binding as opposed to
      a fixed interval/count.

----


---

[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...

Posted by afine <gi...@git.apache.org>.
Github user afine commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/419#discussion_r151812103
  
    --- Diff: src/java/main/org/apache/zookeeper/server/quorum/ExponentialBackoffStrategy.java ---
    @@ -0,0 +1,192 @@
    +package org.apache.zookeeper.server.quorum;
    +
    +/**
    + * A {@link BackoffStrategy} that increases the wait time between each
    + * interval up to the configured maximum wait time.
    + */
    +public class ExponentialBackoffStrategy implements BackoffStrategy {
    +
    +    // Sensible default values to use if not set by the user
    +    private static final long DEFAULT_INITIAL_BACKOFF_MILLIS = 500L;  // 0.5s
    +    private static final long DEFAULT_MAX_BACKOFF_MILLIS = 30_000L;  // 30s
    +    private static final long DEFAULT_MAX_ELAPSED_MILLIS = 5 * 60_000L; // 10m
    +    private static final double DEFAULT_BACKOFF_MULTIPLE = 1.5;
    +
    +    // internal values per instance
    +    private final long initialBackoffMillis;
    +    private final long maxBackoffMillis;
    +    private final long maxElapsedMillis;
    +    private final double backoffMultiple;
    +
    +    // internal state
    +    private long nextWait;
    +    private long totalElapsed;
    +    private final boolean limitBackoffMillis;
    +    private final boolean checkElapsedTime;
    +
    +    /**
    +     * Construct a new instance.
    +     * @param builder the Builder to use for configuring this BackoffStrategy
    +     */
    +    private ExponentialBackoffStrategy(Builder builder) {
    +        this.initialBackoffMillis = builder.initialBackoffMillis;
    +        this.maxBackoffMillis = builder.maxBackoffMillis;
    +        this.maxElapsedMillis = builder.maxElapsedMillis;
    +        this.backoffMultiple = builder.backoffMultiple;
    +
    +        if(maxBackoffMillis == -1) {
    +            limitBackoffMillis = false;
    +        } else {
    +            limitBackoffMillis = true;
    +        }
    +
    +        if(maxElapsedMillis == -1) {
    +            checkElapsedTime = false;
    +        } else {
    +            checkElapsedTime = true;
    +        }
    +
    +        reset();
    +    }
    +
    +
    +    @Override
    +    public long nextWaitMillis() throws IllegalStateException {
    +        // check if we have exceeded the allowed maximum elapsed time
    +        if(checkElapsedTime && totalElapsed > maxElapsedMillis) {
    +            return BackoffStrategy.STOP;
    +        }
    +
    +        long waitMillis = nextWait;
    +
    +        // calculate the next wait milliseconds
    +        nextWait = Math.round(nextWait * backoffMultiple);
    +
    +        // don't exceed the allowed maximum wait milliseconds
    +        // if a maximum was configured
    +        if(limitBackoffMillis && nextWait > maxBackoffMillis) {
    +            nextWait = maxBackoffMillis;
    +        }
    +
    +        // track total elapsed time, even if we don't wait we have to assume
    +        // that some amount of time passed outside of the wait or we'll never
    +        // hit the elapsed time limit
    +        totalElapsed += waitMillis != 0 ? waitMillis : 1L;
    +        return waitMillis;
    +    }
    +
    +    @Override
    +    public void reset() {
    +        nextWait = this.initialBackoffMillis;
    +        totalElapsed = 0;
    +    }
    +
    +    /**
    +     *
    +     * @return a new {@link Builder} instance.
    +     */
    +    public static Builder builder() {
    +        return new Builder();
    +    }
    +
    +    /**
    +     * Builder for instances of {@link ExponentialBackoffStrategy}.
    +     */
    +    public static final class Builder {
    --- End diff --
    
    So I probably wasn't clear here. It would be great if it was also possible, since backoff strategy is such a nicely defined interface, if the user could include jar with their own backoff strategy in the classpath and tell zookeeper to use that. What do you think?


---

[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...

Posted by brian-lininger <gi...@git.apache.org>.
Github user brian-lininger commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/419#discussion_r152119989
  
    --- Diff: src/java/main/org/apache/zookeeper/server/quorum/ExponentialBackoffStrategy.java ---
    @@ -0,0 +1,192 @@
    +package org.apache.zookeeper.server.quorum;
    +
    +/**
    + * A {@link BackoffStrategy} that increases the wait time between each
    + * interval up to the configured maximum wait time.
    + */
    +public class ExponentialBackoffStrategy implements BackoffStrategy {
    +
    +    // Sensible default values to use if not set by the user
    +    private static final long DEFAULT_INITIAL_BACKOFF_MILLIS = 500L;  // 0.5s
    +    private static final long DEFAULT_MAX_BACKOFF_MILLIS = 30_000L;  // 30s
    +    private static final long DEFAULT_MAX_ELAPSED_MILLIS = 5 * 60_000L; // 10m
    +    private static final double DEFAULT_BACKOFF_MULTIPLE = 1.5;
    +
    +    // internal values per instance
    +    private final long initialBackoffMillis;
    +    private final long maxBackoffMillis;
    +    private final long maxElapsedMillis;
    +    private final double backoffMultiple;
    +
    +    // internal state
    +    private long nextWait;
    +    private long totalElapsed;
    +    private final boolean limitBackoffMillis;
    +    private final boolean checkElapsedTime;
    +
    +    /**
    +     * Construct a new instance.
    +     * @param builder the Builder to use for configuring this BackoffStrategy
    +     */
    +    private ExponentialBackoffStrategy(Builder builder) {
    +        this.initialBackoffMillis = builder.initialBackoffMillis;
    +        this.maxBackoffMillis = builder.maxBackoffMillis;
    +        this.maxElapsedMillis = builder.maxElapsedMillis;
    +        this.backoffMultiple = builder.backoffMultiple;
    +
    +        if(maxBackoffMillis == -1) {
    +            limitBackoffMillis = false;
    +        } else {
    +            limitBackoffMillis = true;
    +        }
    +
    +        if(maxElapsedMillis == -1) {
    +            checkElapsedTime = false;
    +        } else {
    +            checkElapsedTime = true;
    +        }
    +
    +        reset();
    +    }
    +
    +
    +    @Override
    +    public long nextWaitMillis() throws IllegalStateException {
    +        // check if we have exceeded the allowed maximum elapsed time
    +        if(checkElapsedTime && totalElapsed > maxElapsedMillis) {
    +            return BackoffStrategy.STOP;
    +        }
    +
    +        long waitMillis = nextWait;
    +
    +        // calculate the next wait milliseconds
    +        nextWait = Math.round(nextWait * backoffMultiple);
    +
    +        // don't exceed the allowed maximum wait milliseconds
    +        // if a maximum was configured
    +        if(limitBackoffMillis && nextWait > maxBackoffMillis) {
    +            nextWait = maxBackoffMillis;
    +        }
    +
    +        // track total elapsed time, even if we don't wait we have to assume
    +        // that some amount of time passed outside of the wait or we'll never
    +        // hit the elapsed time limit
    +        totalElapsed += waitMillis != 0 ? waitMillis : 1L;
    +        return waitMillis;
    +    }
    +
    +    @Override
    +    public void reset() {
    +        nextWait = this.initialBackoffMillis;
    +        totalElapsed = 0;
    +    }
    +
    +    /**
    +     *
    +     * @return a new {@link Builder} instance.
    +     */
    +    public static Builder builder() {
    +        return new Builder();
    +    }
    +
    +    /**
    +     * Builder for instances of {@link ExponentialBackoffStrategy}.
    +     */
    +    public static final class Builder {
    --- End diff --
    
    That should be doable.


---

[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...

Posted by afine <gi...@git.apache.org>.
Github user afine commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/419#discussion_r151560463
  
    --- Diff: src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java ---
    @@ -946,7 +946,8 @@ protected Election createElectionAlgorithm(int electionAlgorithm){
                     le = new AuthFastLeaderElection(this, true);
                     break;
                 case 3:
    -                qcm = new QuorumCnxManager(this);
    +                qcm = new QuorumCnxManager(this,
    +                    ExponentialBackoffStrategy.builder().build());
    --- End diff --
    
    For instance, it may be nice to have the default strategy perform the existing behavior.


---

[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...

Posted by afine <gi...@git.apache.org>.
Github user afine commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/419#discussion_r151553425
  
    --- Diff: src/java/main/org/apache/zookeeper/server/quorum/ExponentialBackoffStrategy.java ---
    @@ -0,0 +1,192 @@
    +package org.apache.zookeeper.server.quorum;
    +
    +/**
    + * A {@link BackoffStrategy} that increases the wait time between each
    + * interval up to the configured maximum wait time.
    + */
    +public class ExponentialBackoffStrategy implements BackoffStrategy {
    +
    +    // Sensible default values to use if not set by the user
    +    private static final long DEFAULT_INITIAL_BACKOFF_MILLIS = 500L;  // 0.5s
    +    private static final long DEFAULT_MAX_BACKOFF_MILLIS = 30_000L;  // 30s
    +    private static final long DEFAULT_MAX_ELAPSED_MILLIS = 5 * 60_000L; // 10m
    +    private static final double DEFAULT_BACKOFF_MULTIPLE = 1.5;
    +
    +    // internal values per instance
    +    private final long initialBackoffMillis;
    +    private final long maxBackoffMillis;
    +    private final long maxElapsedMillis;
    +    private final double backoffMultiple;
    +
    +    // internal state
    +    private long nextWait;
    +    private long totalElapsed;
    +    private final boolean limitBackoffMillis;
    +    private final boolean checkElapsedTime;
    +
    +    /**
    +     * Construct a new instance.
    +     * @param builder the Builder to use for configuring this BackoffStrategy
    +     */
    +    private ExponentialBackoffStrategy(Builder builder) {
    +        this.initialBackoffMillis = builder.initialBackoffMillis;
    +        this.maxBackoffMillis = builder.maxBackoffMillis;
    +        this.maxElapsedMillis = builder.maxElapsedMillis;
    +        this.backoffMultiple = builder.backoffMultiple;
    +
    +        if(maxBackoffMillis == -1) {
    +            limitBackoffMillis = false;
    +        } else {
    +            limitBackoffMillis = true;
    +        }
    +
    +        if(maxElapsedMillis == -1) {
    +            checkElapsedTime = false;
    +        } else {
    +            checkElapsedTime = true;
    +        }
    +
    +        reset();
    +    }
    +
    +
    +    @Override
    +    public long nextWaitMillis() throws IllegalStateException {
    +        // check if we have exceeded the allowed maximum elapsed time
    +        if(checkElapsedTime && totalElapsed > maxElapsedMillis) {
    +            return BackoffStrategy.STOP;
    +        }
    +
    +        long waitMillis = nextWait;
    +
    +        // calculate the next wait milliseconds
    +        nextWait = Math.round(nextWait * backoffMultiple);
    +
    +        // don't exceed the allowed maximum wait milliseconds
    +        // if a maximum was configured
    +        if(limitBackoffMillis && nextWait > maxBackoffMillis) {
    +            nextWait = maxBackoffMillis;
    +        }
    +
    +        // track total elapsed time, even if we don't wait we have to assume
    +        // that some amount of time passed outside of the wait or we'll never
    +        // hit the elapsed time limit
    +        totalElapsed += waitMillis != 0 ? waitMillis : 1L;
    +        return waitMillis;
    +    }
    +
    +    @Override
    +    public void reset() {
    +        nextWait = this.initialBackoffMillis;
    +        totalElapsed = 0;
    +    }
    +
    +    /**
    +     *
    +     * @return a new {@link Builder} instance.
    +     */
    +    public static Builder builder() {
    +        return new Builder();
    +    }
    +
    +    /**
    +     * Builder for instances of {@link ExponentialBackoffStrategy}.
    +     */
    +    public static final class Builder {
    --- End diff --
    
    I'm not sure a builder is the best way to handle this. Since I think it would be nice to have much of this be user configurable, perhaps we can just pull the config from system properties?


---

[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...

Posted by afine <gi...@git.apache.org>.
Github user afine commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/419#discussion_r151532589
  
    --- Diff: src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java ---
    @@ -946,7 +946,8 @@ protected Election createElectionAlgorithm(int electionAlgorithm){
                     le = new AuthFastLeaderElection(this, true);
                     break;
                 case 3:
    -                qcm = new QuorumCnxManager(this);
    +                qcm = new QuorumCnxManager(this,
    +                    ExponentialBackoffStrategy.builder().build());
    --- End diff --
    
    Perhaps we should find a way to make this pluggable? Perhaps we can read which backoff strategy to use from a java system property?


---

[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...

Posted by afine <gi...@git.apache.org>.
Github user afine commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/419#discussion_r151560170
  
    --- Diff: src/java/test/org/apache/zookeeper/server/quorum/ExponentialBackoffStrategyTest.java ---
    @@ -0,0 +1,180 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.zookeeper.server.quorum;
    +
    +import static org.junit.Assert.assertEquals;
    +import static org.junit.Assert.assertNotEquals;
    +import static org.junit.Assert.assertTrue;
    +
    +import org.junit.Test;
    +
    +/**
    + * Unit tests for {@link ExponentialBackoffStrategy}.
    + */
    +public class ExponentialBackoffStrategyTest {
    --- End diff --
    
    This should extend ZKTestCase


---

[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...

Posted by brian-lininger <gi...@git.apache.org>.
Github user brian-lininger commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/419#discussion_r151766248
  
    --- Diff: src/java/main/org/apache/zookeeper/server/quorum/ExponentialBackoffStrategy.java ---
    @@ -0,0 +1,192 @@
    +package org.apache.zookeeper.server.quorum;
    +
    +/**
    + * A {@link BackoffStrategy} that increases the wait time between each
    + * interval up to the configured maximum wait time.
    + */
    +public class ExponentialBackoffStrategy implements BackoffStrategy {
    +
    +    // Sensible default values to use if not set by the user
    +    private static final long DEFAULT_INITIAL_BACKOFF_MILLIS = 500L;  // 0.5s
    +    private static final long DEFAULT_MAX_BACKOFF_MILLIS = 30_000L;  // 30s
    +    private static final long DEFAULT_MAX_ELAPSED_MILLIS = 5 * 60_000L; // 10m
    +    private static final double DEFAULT_BACKOFF_MULTIPLE = 1.5;
    +
    +    // internal values per instance
    +    private final long initialBackoffMillis;
    +    private final long maxBackoffMillis;
    +    private final long maxElapsedMillis;
    +    private final double backoffMultiple;
    +
    +    // internal state
    +    private long nextWait;
    +    private long totalElapsed;
    +    private final boolean limitBackoffMillis;
    +    private final boolean checkElapsedTime;
    +
    +    /**
    +     * Construct a new instance.
    +     * @param builder the Builder to use for configuring this BackoffStrategy
    +     */
    +    private ExponentialBackoffStrategy(Builder builder) {
    +        this.initialBackoffMillis = builder.initialBackoffMillis;
    +        this.maxBackoffMillis = builder.maxBackoffMillis;
    +        this.maxElapsedMillis = builder.maxElapsedMillis;
    +        this.backoffMultiple = builder.backoffMultiple;
    +
    +        if(maxBackoffMillis == -1) {
    +            limitBackoffMillis = false;
    +        } else {
    +            limitBackoffMillis = true;
    +        }
    +
    +        if(maxElapsedMillis == -1) {
    +            checkElapsedTime = false;
    +        } else {
    +            checkElapsedTime = true;
    +        }
    +
    +        reset();
    +    }
    +
    +
    +    @Override
    +    public long nextWaitMillis() throws IllegalStateException {
    +        // check if we have exceeded the allowed maximum elapsed time
    +        if(checkElapsedTime && totalElapsed > maxElapsedMillis) {
    +            return BackoffStrategy.STOP;
    +        }
    +
    +        long waitMillis = nextWait;
    +
    +        // calculate the next wait milliseconds
    +        nextWait = Math.round(nextWait * backoffMultiple);
    +
    +        // don't exceed the allowed maximum wait milliseconds
    +        // if a maximum was configured
    +        if(limitBackoffMillis && nextWait > maxBackoffMillis) {
    +            nextWait = maxBackoffMillis;
    +        }
    +
    +        // track total elapsed time, even if we don't wait we have to assume
    +        // that some amount of time passed outside of the wait or we'll never
    +        // hit the elapsed time limit
    +        totalElapsed += waitMillis != 0 ? waitMillis : 1L;
    +        return waitMillis;
    +    }
    +
    +    @Override
    +    public void reset() {
    +        nextWait = this.initialBackoffMillis;
    +        totalElapsed = 0;
    +    }
    +
    +    /**
    +     *
    +     * @return a new {@link Builder} instance.
    +     */
    +    public static Builder builder() {
    +        return new Builder();
    +    }
    +
    +    /**
    +     * Builder for instances of {@link ExponentialBackoffStrategy}.
    +     */
    +    public static final class Builder {
    --- End diff --
    
    I agree that it should be configurable, but I don't think using a Builder pattern precludes this.  I'm about to push changes that adds support for using system properties, that's a good idea.


---

[GitHub] zookeeper pull request #419: ZOOKEEPER-2849: Exponential back-off retry for ...

Posted by brian-lininger <gi...@git.apache.org>.
Github user brian-lininger commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/419#discussion_r151766328
  
    --- Diff: src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java ---
    @@ -946,7 +946,8 @@ protected Election createElectionAlgorithm(int electionAlgorithm){
                     le = new AuthFastLeaderElection(this, true);
                     break;
                 case 3:
    -                qcm = new QuorumCnxManager(this);
    +                qcm = new QuorumCnxManager(this,
    +                    ExponentialBackoffStrategy.builder().build());
    --- End diff --
    
    Agreed, included in my upcoming push.


---