You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Karthik Anantha Padmanabhan <tr...@gmail.com> on 2017/03/23 08:22:18 UTC

Re: Review Request 54957: Add option to not retry api calls to the scheduler.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/
-----------------------------------------------------------

(Updated March 23, 2017, 8:22 a.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.


Changes
-------

CR Comments.


Repository: aurora


Description
-------

This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.

If there is consensus, this can be added to all commands as well.


Diffs (updated)
-----

  src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/admin/aurora_admin.py fbebbab8c827b5695042d18770d850e31fc38122 
  src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
  src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 


Diff: https://reviews.apache.org/r/54957/diff/2/

Changes: https://reviews.apache.org/r/54957/diff/1-2/


Testing
-------

* Manuall testing
* ./build-support/jenkins/build.sh passes


Thanks,

Karthik Anantha Padmanabhan


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/#review169847
-----------------------------------------------------------



Master (33acb89) is red with this patch.
  ./build-support/jenkins/build.sh

               (To run a reporting server: ./pants server)
08:38:11 00:00   [setup]
08:38:11 00:00     [parse]
               Executing tasks in goals: compile
08:38:11 00:00   [compile]
08:38:11 00:00     [compile-prep-command]
08:38:11 00:00       [prep_command]
08:38:14 00:03     [compile]
08:38:14 00:03     [python-eval]
08:38:14 00:03     [pythonstyle]
08:38:14 00:03       [cache]                                          
                   No cached artifacts for 42 targets.
                   Invalidated 42 targets.
T302:ERROR   src/test/python/apache/aurora/client/api/test_scheduler_client.py:059 Expected 2 blank lines, found 1
     |def test_coverage():

T301:ERROR   src/test/python/apache/aurora/client/api/test_scheduler_client.py:598-599 Expected 1 blank lines, found 2
     |  @mock.patch('apache.aurora.client.api.scheduler_client.SchedulerClient',
     |              spec=scheduler_client.SchedulerClient)

E302:ERROR   PythonFile(src/test/python/apache/aurora/client/api/test_scheduler_client.py):059 expected 2 blank lines, found 1
     |def test_coverage():

E303:ERROR   PythonFile(src/test/python/apache/aurora/client/api/test_scheduler_client.py):598-599 too many blank lines (2)
     |  @mock.patch('apache.aurora.client.api.scheduler_client.SchedulerClient',
     |              spec=scheduler_client.SchedulerClient)


T302:ERROR   src/main/python/apache/aurora/admin/aurora_admin.py:050 Expected 2 blank lines, found 1
     |def proxy_main():

E302:ERROR   PythonFile(src/main/python/apache/aurora/admin/aurora_admin.py):050 expected 2 blank lines, found 1
     |def proxy_main():


FAILURE: 6 Python Style issues found. For import order related issues, please try `./pants fmt.isort <targets>`


08:38:39 00:28   [complete]
               FAILURE


I will refresh this build result if you post a review containing "@ReviewBot retry"

- Aurora ReviewBot


On March 23, 2017, 8:23 a.m., Karthik Anantha Padmanabhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54957/
> -----------------------------------------------------------
> 
> (Updated March 23, 2017, 8:23 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.
> 
> If there is consensus, this can be added to all commands as well.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/admin/aurora_admin.py fbebbab8c827b5695042d18770d850e31fc38122 
>   src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 
> 
> 
> Diff: https://reviews.apache.org/r/54957/diff/3/
> 
> 
> Testing
> -------
> 
> * Manual testing
> * ./build-support/jenkins/build.sh passes
> 
> 
> Thanks,
> 
> Karthik Anantha Padmanabhan
> 
>


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/#review169889
-----------------------------------------------------------


Ship it!




Master (33acb89) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot retry"

- Aurora ReviewBot


On March 23, 2017, 4:23 p.m., Karthik Anantha Padmanabhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54957/
> -----------------------------------------------------------
> 
> (Updated March 23, 2017, 4:23 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.
> 
> If there is consensus, this can be added to all commands as well.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 
> 
> 
> Diff: https://reviews.apache.org/r/54957/diff/4/
> 
> 
> Testing
> -------
> 
> * Manual testing
> * ./build-support/jenkins/build.sh passes
> 
> 
> Thanks,
> 
> Karthik Anantha Padmanabhan
> 
>


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Karthik Anantha Padmanabhan <tr...@gmail.com>.

> On March 23, 2017, 4:52 p.m., David McLaughlin wrote:
> > src/main/python/apache/aurora/admin/admin.py
> > Lines 244 (patched)
> > <https://reviews.apache.org/r/54957/diff/4/?file=1672873#file1672873line244>
> >
> >     Where is options.no_retry defined?

duh ! Bad push my bad.


- Karthik


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/#review169894
-----------------------------------------------------------


On March 23, 2017, 4:23 p.m., Karthik Anantha Padmanabhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54957/
> -----------------------------------------------------------
> 
> (Updated March 23, 2017, 4:23 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.
> 
> If there is consensus, this can be added to all commands as well.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 
> 
> 
> Diff: https://reviews.apache.org/r/54957/diff/4/
> 
> 
> Testing
> -------
> 
> * Manual testing
> * ./build-support/jenkins/build.sh passes
> 
> 
> Thanks,
> 
> Karthik Anantha Padmanabhan
> 
>


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by David McLaughlin <da...@dmclaughlin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/#review169894
-----------------------------------------------------------




src/main/python/apache/aurora/admin/admin.py
Lines 244 (patched)
<https://reviews.apache.org/r/54957/#comment242595>

    Where is options.no_retry defined?


- David McLaughlin


On March 23, 2017, 4:23 p.m., Karthik Anantha Padmanabhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54957/
> -----------------------------------------------------------
> 
> (Updated March 23, 2017, 4:23 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.
> 
> If there is consensus, this can be added to all commands as well.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 
> 
> 
> Diff: https://reviews.apache.org/r/54957/diff/4/
> 
> 
> Testing
> -------
> 
> * Manual testing
> * ./build-support/jenkins/build.sh passes
> 
> 
> Thanks,
> 
> Karthik Anantha Padmanabhan
> 
>


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/#review169977
-----------------------------------------------------------



Master (c32f14c) is red with this patch.
  ./build-support/jenkins/build.sh

  Running setup.py bdist_wheel for twitter.common.log: finished with status 'done'
  Stored in directory: /home/jenkins/jenkins-slave/workspace/AuroraBot/.home/.cache/pip/wheels/80/e3/3a/297d8950fcbd822ab5a6a62175818fab38b668cc5a86dbd0b0
  Running setup.py bdist_wheel for pycparser: started
  Running setup.py bdist_wheel for pycparser: finished with status 'done'
  Stored in directory: /home/jenkins/jenkins-slave/workspace/AuroraBot/.home/.cache/pip/wheels/a8/0b/41/dc95621f9d3a0da7bc191b8a71f0e8182ffd3cc5f33ac55005
  Running setup.py bdist_wheel for twitter.common.options: started
  Running setup.py bdist_wheel for twitter.common.options: finished with status 'done'
  Stored in directory: /home/jenkins/jenkins-slave/workspace/AuroraBot/.home/.cache/pip/wheels/17/41/80/c4811d8c1c7ca7007e520c3399872fc340f45c3a26a6a23e6a
Successfully built pantsbuild.pants twitter.common.collections setproctitle ansicolors pathspec scandir twitter.common.dirutil psutil pystache docutils Markdown Pygments twitter.common.confluence coverage lmdb pywatchman twitter.common.lang twitter.common.log pycparser twitter.common.options
Installing collected packages: twitter.common.lang, twitter.common.collections, setproctitle, setuptools, six, ansicolors, pyparsing, packaging, pathspec, scandir, twitter.common.dirutil, psutil, requests, pystache, pex, docutils, Markdown, Pygments, twitter.common.options, twitter.common.log, twitter.common.confluence, monotonic, fasteners, coverage, pycparser, cffi, lmdb, pywatchman, futures, pantsbuild.pants
  Found existing installation: setuptools 21.2.1
    Uninstalling setuptools-21.2.1:
      Successfully uninstalled setuptools-21.2.1
Successfully installed Markdown-2.1.1 Pygments-1.4 ansicolors-1.0.2 cffi-1.7.0 coverage-3.7.1 docutils-0.12 fasteners-0.14.1 futures-3.0.5 lmdb-0.89 monotonic-1.3 packaging-16.7 pantsbuild.pants-1.3.0.dev3 pathspec-0.3.4 pex-1.1.16 psutil-4.3.0 pycparser-2.17 pyparsing-2.2.0 pystache-0.5.3 pywatchman-1.3.0 requests-2.5.3 scandir-1.2 setproctitle-1.1.10 setuptools-30.0.0 six-1.10.0 twitter.common.collections-0.3.9 twitter.common.confluence-0.3.9 twitter.common.dirutil-0.3.9 twitter.common.lang-0.3.9 twitter.common.log-0.3.9 twitter.common.options-0.3.9
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

04:47:22 00:00 [main]
               (To run a reporting server: ./pants server)
04:47:22 00:00   [setup]
04:47:22 00:00     [parse]
               Executing tasks in goals: compile
04:47:23 00:01   [compile]
04:47:23 00:01     [compile-prep-command]
04:47:23 00:01       [prep_command]
04:47:25 00:03     [compile]
04:47:25 00:03     [python-eval]
04:47:25 00:03     [pythonstyle]
04:47:26 00:04       [cache]                                          
                   No cached artifacts for 42 targets.
                   Invalidated 42 targets.
F841:ERROR   src/main/python/apache/aurora/admin/admin.py:274 local variable 'options' is assigned to but never used
     |  options = app.get_options()


FAILURE: 1 Python Style issues found. For import order related issues, please try `./pants fmt.isort <targets>`


04:47:49 00:27   [complete]
               FAILURE


I will refresh this build result if you post a review containing "@ReviewBot retry"

- Aurora ReviewBot


On March 24, 2017, 4:38 a.m., Karthik Anantha Padmanabhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54957/
> -----------------------------------------------------------
> 
> (Updated March 24, 2017, 4:38 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.
> 
> If there is consensus, this can be added to all commands as well.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 
> 
> 
> Diff: https://reviews.apache.org/r/54957/diff/5/
> 
> 
> Testing
> -------
> 
> * Manual testing
> * ./build-support/jenkins/build.sh passes
> 
> 
> Thanks,
> 
> Karthik Anantha Padmanabhan
> 
>


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Santhosh Kumar Shanmugham <sa...@gmail.com>.

> On March 27, 2017, 3:22 p.m., Stephan Erb wrote:
> > src/main/python/apache/aurora/client/api/scheduler_client.py
> > Line 324 (original), 324 (patched)
> > <https://reviews.apache.org/r/54957/diff/6/?file=1674018#file1674018line325>
> >
> >     Could you please elaborate why TransportException and TransientError require special retry handling, whereas a TimeoutError can always be retried? Sorry if I have missed some previous discussion.

Seeing that there is some subtly in the choice to retry - can you capture the same in a comment as well? Thanks.


- Santhosh Kumar


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/#review170233
-----------------------------------------------------------


On March 24, 2017, 11:30 a.m., Karthik Anantha Padmanabhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54957/
> -----------------------------------------------------------
> 
> (Updated March 24, 2017, 11:30 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.
> 
> If there is consensus, this can be added to all commands as well.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 
> 
> 
> Diff: https://reviews.apache.org/r/54957/diff/6/
> 
> 
> Testing
> -------
> 
> * Manual testing
> * ./build-support/jenkins/build.sh passes
> 
> 
> Thanks,
> 
> Karthik Anantha Padmanabhan
> 
>


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Stephan Erb <se...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/#review170233
-----------------------------------------------------------




src/main/python/apache/aurora/client/api/scheduler_client.py
Line 324 (original), 324 (patched)
<https://reviews.apache.org/r/54957/#comment243023>

    Could you please elaborate why TransportException and TransientError require special retry handling, whereas a TimeoutError can always be retried? Sorry if I have missed some previous discussion.


- Stephan Erb


On March 24, 2017, 7:30 p.m., Karthik Anantha Padmanabhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54957/
> -----------------------------------------------------------
> 
> (Updated March 24, 2017, 7:30 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.
> 
> If there is consensus, this can be added to all commands as well.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 
> 
> 
> Diff: https://reviews.apache.org/r/54957/diff/6/
> 
> 
> Testing
> -------
> 
> * Manual testing
> * ./build-support/jenkins/build.sh passes
> 
> 
> Thanks,
> 
> Karthik Anantha Padmanabhan
> 
>


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Mehrdad Nurolahzade <me...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/#review172990
-----------------------------------------------------------



I see three catgories of timeouts here:

(a) Aurora scheduler cannot be reached

If a connection between Aurora client and scheduler has not been established yet or has been invalidated/terminated due to error (e.g., transport exception) then client makes an attempt to stablish the connection before submitting an operation to the scheduler. If stablishing the connection takes longer than 1 minute, a `TimeoutError` is raised in the following context in `method_wrapper()` that is then trapped, leading to an automatic retry of the operation.

```
    @functools.wraps(method)
    def method_wrapper(*args):
      with self._lock:
        start = time.time()
        while not self._terminating.is_set() and (
            time.time() - start) < self.RPC_MAXIMUM_WAIT.as_(Time.SECONDS):

          try:
            // self.client() raises self.TimeoutError after 1 minute if it cannot establish a connection
            method = getattr(self.client(), method_name)
            if not callable(method):
              return method
            // raises TTransport.TTransportException for HTTP response code in the ranges 4XX and 5XX
            resp = method(*args)
            if resp is not None and resp.responseCode == ResponseCode.ERROR_TRANSIENT:
              raise self.TransientError(", ".join(
                  [m.message for m in resp.details] if resp.details else []))            
          ...
          except (TTransport.TTransportException, self.TimeoutError, self.TransientError) as e:
            if not self._terminating.is_set():
              log.warning('Connection error with scheduler: %s, reconnecting...' % e)
              self.invalidate()
              self._terminating.wait(self.RPC_RETRY_INTERVAL.as_(Time.SECONDS))
        ...
```

(b) Transport exception occured in the communication

A transport exception (e.g., HTTP 503 Service Temporarily Unavailable) can be raised in the communication link between Aurora and scheduler in `TTransportException.flush()`. 

```
  def flush(self):
    ...
    try:
      response = self._session.post(
          self.__uri,
          data=data,
          timeout=self.__timeout,
          auth=self.__auth)
      response.raise_for_status()
    ...
```

This `TTransport.TTransportException` is trapped in `SchedulerClient.method_wrapper()`, leading to an automatic retry of the operation.

(c) Transient state error in scheduler

If Aurora scheduler discovers that its storage is not in the `READY` state to process a storage operation (i.e., read, write, or snapshot) it throws `Storage.TransientException`. This is caught by `LoggingInterceptor` and translated to a thrift response with `ResponseCode.ERROR_TRANSIENT`.

```
  @Override
  public Object invoke(MethodInvocation invocation) throws Throwable {
    ...
    Response response = null;
    try {
      // casting is safe, interception happens on methods that return Response or its subclasses
      response = (Response) invocation.proceed();
    } catch (Storage.TransientStorageException e) {
      LOG.warn("Uncaught transient exception while handling {}({})", methodName, messageArgs, e);
      response = Responses.addMessage(Responses.empty(), ResponseCode.ERROR_TRANSIENT, e);
    } 
    ...
```

Aurora client receives `ResponseCode.ERROR_TRANSIENT` and raises `TransientError` which is trapped in `SchedulerClient.method_wrapper()`, leading to an automatic retry of the operation.

Based on the above analysis, I believe:
- (a) is always safe to retry; it just establishes a connection with the scheduler.
- (b) is not safe to retry; the operation may or may not have been processed by the scheduler.
- (c) is always safe to retry; previous attempt to modify storage has failed.

- Mehrdad Nurolahzade


On March 24, 2017, 11:30 a.m., Karthik Anantha Padmanabhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54957/
> -----------------------------------------------------------
> 
> (Updated March 24, 2017, 11:30 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.
> 
> If there is consensus, this can be added to all commands as well.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 
> 
> 
> Diff: https://reviews.apache.org/r/54957/diff/6/
> 
> 
> Testing
> -------
> 
> * Manual testing
> * ./build-support/jenkins/build.sh passes
> 
> 
> Thanks,
> 
> Karthik Anantha Padmanabhan
> 
>


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Santhosh Kumar Shanmugham <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/#review170227
-----------------------------------------------------------


Ship it!





src/main/python/apache/aurora/admin/admin.py
Line 242 (original), 242-243 (patched)
<https://reviews.apache.org/r/54957/#comment243010>

    kill ?


- Santhosh Kumar Shanmugham


On March 24, 2017, 11:30 a.m., Karthik Anantha Padmanabhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54957/
> -----------------------------------------------------------
> 
> (Updated March 24, 2017, 11:30 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.
> 
> If there is consensus, this can be added to all commands as well.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 
> 
> 
> Diff: https://reviews.apache.org/r/54957/diff/6/
> 
> 
> Testing
> -------
> 
> * Manual testing
> * ./build-support/jenkins/build.sh passes
> 
> 
> Thanks,
> 
> Karthik Anantha Padmanabhan
> 
>


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/#review170047
-----------------------------------------------------------


Ship it!




Master (c32f14c) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot retry"

- Aurora ReviewBot


On March 24, 2017, 6:30 p.m., Karthik Anantha Padmanabhan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54957/
> -----------------------------------------------------------
> 
> (Updated March 24, 2017, 6:30 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.
> 
> If there is consensus, this can be added to all commands as well.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 
> 
> 
> Diff: https://reviews.apache.org/r/54957/diff/6/
> 
> 
> Testing
> -------
> 
> * Manual testing
> * ./build-support/jenkins/build.sh passes
> 
> 
> Thanks,
> 
> Karthik Anantha Padmanabhan
> 
>


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Karthik Anantha Padmanabhan <tr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/
-----------------------------------------------------------

(Updated March 24, 2017, 6:30 p.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.


Repository: aurora


Description
-------

This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.

If there is consensus, this can be added to all commands as well.


Diffs (updated)
-----

  src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
  src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 


Diff: https://reviews.apache.org/r/54957/diff/6/

Changes: https://reviews.apache.org/r/54957/diff/5-6/


Testing
-------

* Manual testing
* ./build-support/jenkins/build.sh passes


Thanks,

Karthik Anantha Padmanabhan


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Karthik Anantha Padmanabhan <tr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/
-----------------------------------------------------------

(Updated March 24, 2017, 4:38 a.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.


Repository: aurora


Description
-------

This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.

If there is consensus, this can be added to all commands as well.


Diffs (updated)
-----

  src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
  src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 


Diff: https://reviews.apache.org/r/54957/diff/5/

Changes: https://reviews.apache.org/r/54957/diff/4-5/


Testing
-------

* Manual testing
* ./build-support/jenkins/build.sh passes


Thanks,

Karthik Anantha Padmanabhan


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Karthik Anantha Padmanabhan <tr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/
-----------------------------------------------------------

(Updated March 23, 2017, 4:23 p.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.


Changes
-------

Style changes.


Repository: aurora


Description
-------

This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.

If there is consensus, this can be added to all commands as well.


Diffs (updated)
-----

  src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
  src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 


Diff: https://reviews.apache.org/r/54957/diff/4/

Changes: https://reviews.apache.org/r/54957/diff/3-4/


Testing
-------

* Manual testing
* ./build-support/jenkins/build.sh passes


Thanks,

Karthik Anantha Padmanabhan


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Karthik Anantha Padmanabhan <tr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/
-----------------------------------------------------------

(Updated March 23, 2017, 8:23 a.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.


Repository: aurora


Description
-------

This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.

If there is consensus, this can be added to all commands as well.


Diffs
-----

  src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/admin/aurora_admin.py fbebbab8c827b5695042d18770d850e31fc38122 
  src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
  src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 


Diff: https://reviews.apache.org/r/54957/diff/3/


Testing (updated)
-------

* Manual testing
* ./build-support/jenkins/build.sh passes


Thanks,

Karthik Anantha Padmanabhan


Re: Review Request 54957: Add option to not retry api calls to the scheduler.

Posted by Karthik Anantha Padmanabhan <tr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54957/
-----------------------------------------------------------

(Updated March 23, 2017, 8:23 a.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji.


Changes
-------

Rename tests.


Repository: aurora


Description
-------

This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command.

If there is consensus, this can be added to all commands as well.


Diffs (updated)
-----

  src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/admin/aurora_admin.py fbebbab8c827b5695042d18770d850e31fc38122 
  src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc 
  src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef 


Diff: https://reviews.apache.org/r/54957/diff/3/

Changes: https://reviews.apache.org/r/54957/diff/2-3/


Testing
-------

* Manuall testing
* ./build-support/jenkins/build.sh passes


Thanks,

Karthik Anantha Padmanabhan