You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by robertkowalski <gi...@git.apache.org> on 2015/05/29 19:29:24 UTC

[GitHub] couchdb-couch pull request: perf: http layer - cache couchdb versi...

GitHub user robertkowalski opened a pull request:

    https://github.com/apache/couchdb-couch/pull/56

    perf: http layer - cache couchdb version

    I have taken a look at our http stack with erlang:trace and flamegraphs.
    It turned out that we spend a lot of our time for ever request in the
    function couch_server:get_version which simply gets the current CouchDB
    version. [1]
    
    ![http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/flame--unpatched-second-request.png](http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/flame--unpatched-second-request.png)
    
    So I tried to memoize it, with pretty good results [2]. 
    
    ![http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/flame--patched-second-request.png](http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/flame--patched-second-request.png)
    
    I benchmarked the unclustered interface, but (almost?) all our 
    requests are accessing this function so it should also be beneficial 
    for chttpd & friends.
    
    After a better looking flamegraph i tried to run Apache ab [[3]](http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/results-ab.txt) on it.
    
    All benchmarks (also the flamegraph creation) were run using this test protocol:
    
    1. turn off all auto starting apps, especially dropbox & co
    2. run make with path
    3. reboot
    4. wait 60secs
    5. boot cluster and wait until successful connected
    6. wait 60secs
    7. run test
    
    Apache „ab“:
    
    10000 requests, concurrency 130, reading one doc, unclustered
    interface:
    
    old version, overall run times:
    test 1: 3.846
    test 2: 3.830
    
    patched version, overall run times:
    test 1: 3.439
    test 2: 3.577
    
    mean difference:
    
    (3.846 + 3.830) / (3.439 + 3.577) * 100
    
    patched version is ~9.4% faster
    
    The implementation of the cache itself is a simple orddict that is
    meant for caching a small amount of these time consuming operations.
    
    [1] http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/flame--unpatched-second-request.png
    [2] http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/flame--patched-second-request.png
    [3] http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/results-ab.txt

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/robertkowalski/couchdb-couch increase-perf

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/couchdb-couch/pull/56.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #56
    
----
commit 4f205dc0f6210ce342ea65994c1eef46674ee8f0
Author: Robert Kowalski <ro...@kowalski.gd>
Date:   2015-05-29T13:25:00Z

    perf: http layer - cache couchdb version
    
    I have taken a look at our http stack with erlang:trace and flamegraphs.
    It turned out that we spend a lot of our time for ever request in the
    function couch_server:get_version which simply gets the current CouchDB
    version. [1]
    
    So I tried to memoize it, with pretty good results [2]. I benchmarked
    the unclustered interface, but (almost?) all our requests are accessing
    this function so it should also be beneficial for chttpd & friends.
    
    After a better looking flamegraph i tried to run Apache ab [3] on it.
    
    All benchmarks (also the flamegraph creation) were run using this test
    protocol:
    
    1. turn off all auto starting apps, especially dropbox & co
    2. run make with path
    3. reboot
    4. wait 60secs
    5. boot cluster and wait until successful connected
    6. wait 60secs
    7. run test
    
    Apache „ab“:
    
    10000 requests, concurrency 130, reading one doc, unclustered
    interface:
    
    old version, overall run times:
    test 1: 3.846
    test 2: 3.830
    
    patched version, overall run times:
    test 1: 3.439
    test 2: 3.577
    
    mean difference:
    
    (3.846 + 3.830) / (3.439 + 3.577) * 100
    
    patched version is ~9.4% faster
    
    The implementation of the cache itself is a simple orddict that is
    meant for caching a small amount of these time consuming operations.
    
    [1] http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/flame--unpatched-second-request.png
    [2] http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/flame--patched-second-request.png
    [3] http://robert-kowalski.de/assets/data/2015-05-29-erlang-perf/results-ab.txt

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: perf: http layer - cache couchdb versi...

Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:

    https://github.com/apache/couchdb-couch/pull/56#issuecomment-106885439
  
    Alternative solution:
    
    ```
    diff --git a/src/couch_server.erl b/src/couch_server.erl
    index 228edc3..48046c6 100644
    --- a/src/couch_server.erl
    +++ b/src/couch_server.erl
    @@ -45,12 +45,9 @@ dev_start() ->
         couch:start().
     
     get_version() ->
    -    Apps = application:loaded_applications(),
    -    case lists:keysearch(couch, 1, Apps) of
    -    {value, {_, _, Vsn}} ->
    -        Vsn;
    -    false ->
    -        "0.0.0"
    +    case application:get_key(couch, vsn) of
    +        {ok, Vsn} -> Vsn;
    +        undefined -> "0.0.0"
         end.
     get_version(short) ->
       %% strip git hash from version string
    ```
    
    No need to fetch all the apps from the ets table and filter them twice (by load status and by name) since we already know our app name and it's already loaded.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: perf: http layer - cache couchdb versi...

Posted by robertkowalski <gi...@git.apache.org>.
Github user robertkowalski commented on the pull request:

    https://github.com/apache/couchdb-couch/pull/56#issuecomment-106922467
  
    @kxepal hehe - ETOOMUCHDOCUMENTATION? :)
    
    I made the commit message shorter. no blogging for me in git commits this we :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: perf: http layer - cache couchdb versi...

Posted by robertkowalski <gi...@git.apache.org>.
Github user robertkowalski closed the pull request at:

    https://github.com/apache/couchdb-couch/pull/56


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: perf: http layer - cache couchdb versi...

Posted by robertkowalski <gi...@git.apache.org>.
Github user robertkowalski commented on the pull request:

    https://github.com/apache/couchdb-couch/pull/56#issuecomment-108123513
  
    Cool, merged with your recommendations I just got on IRC.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: perf: http layer - cache couchdb versi...

Posted by robertkowalski <gi...@git.apache.org>.
Github user robertkowalski commented on the pull request:

    https://github.com/apache/couchdb-couch/pull/56#issuecomment-106906420
  
    thanks for all the input and suggestions! updated!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: perf: http layer - cache couchdb versi...

Posted by banjiewen <gi...@git.apache.org>.
Github user banjiewen commented on the pull request:

    https://github.com/apache/couchdb-couch/pull/56#issuecomment-106882885
  
    Using the application environment would be faster than the cache implemented here. Plus it wouldn't represent a potential failure due to mailbox overflow.
    
    You would, of course, need to hack the version in to the app env somehow - I don't think it's there already.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: perf: http layer - cache couchdb versi...

Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:

    https://github.com/apache/couchdb-couch/pull/56#issuecomment-106906983
  
    @robertkowalski if you update commit message to make it actual and shorter (no need to explain there about flamegraphs and benchmark, but about what it does and why it makes things better) then +1 from me (:


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---