You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by tonysun83 <gi...@git.apache.org> on 2016/09/21 18:37:33 UTC

[GitHub] couchdb-couch-replicator pull request #48: Port HTTP 429 Commits

GitHub user tonysun83 opened a pull request:

    https://github.com/apache/couchdb-couch-replicator/pull/48

    Port HTTP 429 Commits

    These commits allow us to not crash when a server returns a HTTP 429 response during replication.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cloudant/couchdb-couch-replicator 3010-port-429

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/couchdb-couch-replicator/pull/48.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #48
    
----

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request #48: Port HTTP 429 Commits

Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on a diff in the pull request:

    https://github.com/apache/couchdb-couch-replicator/pull/48#discussion_r79905411
  
    --- Diff: src/couch_replicator_httpc.erl ---
    @@ -27,6 +27,9 @@
     
     -define(replace(L, K, V), lists:keystore(K, 1, L, {K, V})).
     -define(MAX_WAIT, 5 * 60 * 1000).
    +-define(MAX_BACKOFF_WAIT, 250 * 32768).
    --- End diff --
    
    Shouldn't these be configurable?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request #48: Port HTTP 429 Commits

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/couchdb-couch-replicator/pull/48


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator issue #48: Port HTTP 429 Commits

Posted by nickva <gi...@git.apache.org>.
Github user nickva commented on the issue:

    https://github.com/apache/couchdb-couch-replicator/pull/48
  
    @tonysun83 I think that is a flaky test, will try it locally see what it does


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator issue #48: Port HTTP 429 Commits

Posted by tonysun83 <gi...@git.apache.org>.
Github user tonysun83 commented on the issue:

    https://github.com/apache/couchdb-couch-replicator/pull/48
  
    @kxepal: made every macro configurable


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator issue #48: Port HTTP 429 Commits

Posted by nickva <gi...@git.apache.org>.
Github user nickva commented on the issue:

    https://github.com/apache/couchdb-couch-replicator/pull/48
  
    @tonysun83 looks like it is flaky and it is in the couch repo anyway, not even in replicator, so go ahead and merge if you want


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator issue #48: Port HTTP 429 Commits

Posted by nickva <gi...@git.apache.org>.
Github user nickva commented on the issue:

    https://github.com/apache/couchdb-couch-replicator/pull/48
  
    +1 (after rebase + lower case the config constants). 
    
    I tested with a rate limited account. Changes replicated and made progress. I lowered the max backoff value to see how errors look like. Replications were dying kind of messy sometimes. Depending which request failed, they'd be kill all procs linked to it, showing crashes `{'EXIT',<0.14124.12>,{process_died,<0.14698.12>,...}` but I think that's fine.
    
    Nice job, Tony
    
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator issue #48: Port HTTP 429 Commits

Posted by tonysun83 <gi...@git.apache.org>.
Github user tonysun83 commented on the issue:

    https://github.com/apache/couchdb-couch-replicator/pull/48
  
    @nickva : thx for the review. rebased and lowered cased the names. do the builds fail sometimes for CI? Log shows 
    ```
    in function couch_changes_tests:'-should_select_with_continuous/1-fun-0-'/1 (test/couch_changes_tests.erl, line 461)
    in call from couch_changes_tests:'-should_select_with_continuous/1-fun-3-'/2 (test/couch_changes_tests.erl, line 460)
    **error:{assertMatch_failed,[{module,couch_changes_tests},
                         {line,461},
                         {expression,"Rows"},
                         {pattern,"[ # row { seq = 10 , id = << \"doc8\" >> , deleted = false } ]"},
                         {value,[]}]}
    ```
    
    I just want to be careful before I merge.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch-replicator pull request #48: Port HTTP 429 Commits

Posted by nickva <gi...@git.apache.org>.
Github user nickva commented on a diff in the pull request:

    https://github.com/apache/couchdb-couch-replicator/pull/48#discussion_r81839898
  
    --- Diff: src/couch_replicator_httpc.erl ---
    @@ -440,3 +440,12 @@ after_redirect(RedirectUrl, _Code, HttpDb, Params) ->
     after_redirect(RedirectUrl, HttpDb, Params) ->
         Params2 = lists:keydelete(path, 1, lists:keydelete(qs, 1, Params)),
         {HttpDb#httpdb{url = RedirectUrl}, Params2}.
    +
    +get_max_back_off() ->
    +    config:get_integer("replicator", "MAX_BACKOFF_WAIT", 250 * 32768).
    --- End diff --
    
    Let's make the settings lower case. Just to confirm with most other settings.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---