[jira] [Commented] (BEAM-2556) Client-side throttling for Datastore connector

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076334#comment-16076334 ] 

ASF GitHub Bot commented on BEAM-2556:
--------------------------------------

GitHub user cph6 opened a pull request:

    https://github.com/apache/beam/pull/3507

    [BEAM-2556] Add client-side throttling.

    The approach used is as described in
    https://landing.google.com/sre/book/chapters/handling-overload.html#client-side-throttling-a7sYUg
    . By backing off individual workers in response to high error rates, we relieve
    pressure on the Datastore service, increasing the chance that the workload can
    complete successfully.
    
    The exported cumulativeThrottledSeconds could also be used as an autoscaling
    signal in future.
    
    R: @vikkyrk 
    R: @ssisk 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cph6/beam datastore_adaptive

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3507
    
----
commit 152fa5c8b8b37e3ecf1b28bacd4d636936a3ea04
Author: Colin Phipps <fi...@google.com>
Date:   2017-06-26T13:34:19Z

    Add client-side throttling.
    
    The approach used is as described in
    https://landing.google.com/sre/book/chapters/handling-overload.html#client-side-throttling-a7sYUg
    . By backing off individual workers in response to high error rates, we relieve
    pressure on the Datastore service, increasing the chance that the workload can
    complete successfully.
    
    The exported cumulativeThrottledSeconds could also be used as an autoscaling
    signal in future.

----


> Client-side throttling for Datastore connector
> ----------------------------------------------
>
>                 Key: BEAM-2556
>                 URL: https://issues.apache.org/jira/browse/BEAM-2556
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-gcp
>            Reporter: Colin Phipps
>            Assignee: Colin Phipps
>            Priority: Minor
>              Labels: datastore
>
> The Datastore connector currently has exponential backoff on errors, which is good. But it does not do any other throttling of its write load in response to errors; once a request succeeds, it resumes writing as quickly as it can.
> Write loads will be more stable and more likely to compete if the client throttles itself in the event that it receives high rates of errors from the Datastore service; specifically 
> https://landing.google.com/sre/book/chapters/handling-overload.html#client-side-throttling-a7sYUg is a technique that Google has had success with on other services.
> We (Datastore) have a patch in progress to add this behaviour to the connector.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)