You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Filipe Manana (JIRA)" <ji...@apache.org> on 2010/12/04 20:11:10 UTC
[jira] Created: (COUCHDB-976) Improve database read and write
performance using 2 couch_files
Improve database read and write performance using 2 couch_files
---------------------------------------------------------------
Key: COUCHDB-976
URL: https://issues.apache.org/jira/browse/COUCHDB-976
Project: CouchDB
Issue Type: Improvement
Reporter: Filipe Manana
Assignee: Filipe Manana
Right now we use a single couch_file server for both updating a database and reading from a database.
This is a contention point, as concurrent read/write access to a database implies having processes waiting for access to the couch_file server.
The following patches add a couch_file server that is used only by the DB updater process and another couch_file meant to be used by anyone else only for read operations:
https://github.com/fdmanana/couchdb/compare/updater_dedicated_fd
Some performance measurements:
# updater_fd vs trunk (small docs, 1Kb each)
$ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
-name1 updater_fd_small_docs -name2 trunk \
-url1 http://localhost:5984/ -url2 http://localhost:5985/ \
--duration 300
http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a616
# updater_fd vs trunk (large docs, 100Kb each)
$ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
-name1 updater_fd_large_docs -name2 trunk \
-url1 http://localhost:5984/ -url2 http://localhost:5985/ \
--duration 300 --doc large
http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a7a7
We can see that both the response time and throughput gets significantly better for both read and writes.
If no objections I'll commit it to trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-976) Improve database read and write
performance using 2 couch_files
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966927#action_12966927 ]
Filipe Manana commented on COUCHDB-976:
---------------------------------------
No significant gains?
For small docs:
- for writes I see in average response times reduced by 100 to 200ms;
- for reads I see in average response times reduced by 100ms.
For large docs:
- writes get in average the response time reduced by 1s to 1.5s;
- idem for reads
And the cache patch forget it, I don't think it's worth it, perhaps only for large docs (100kb), however it doesn't boost performance as much as this patch. Its gains are generally smaller than 5ms, which is almost insignificant, not to mention the additional complexity it brings: more code, more gen_servers, 1 ets per cache, etc.
I have an analysis of the caching at https://issues.apache.org/jira/browse/COUCHDB-913
> Improve database read and write performance using 2 couch_files
> ---------------------------------------------------------------
>
> Key: COUCHDB-976
> URL: https://issues.apache.org/jira/browse/COUCHDB-976
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Right now we use a single couch_file server for both updating a database and reading from a database.
> This is a contention point, as concurrent read/write access to a database implies having processes waiting for access to the couch_file server.
> The following patches add a couch_file server that is used only by the DB updater process and another couch_file meant to be used by anyone else only for read operations:
> https://github.com/fdmanana/couchdb/compare/updater_dedicated_fd
> Some performance measurements:
> # updater_fd vs trunk (small docs, 1Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_small_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a616
> # updater_fd vs trunk (large docs, 100Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_large_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300 --doc large
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a7a7
> We can see that both the response time and throughput gets significantly better for both read and writes.
> If no objections I'll commit it to trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (COUCHDB-976) Improve database read and write
performance using 2 couch_files
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Filipe Manana closed COUCHDB-976.
---------------------------------
Resolution: Fixed
Committed to trunk (revision 1043352).
Thanks everyone.
> Improve database read and write performance using 2 couch_files
> ---------------------------------------------------------------
>
> Key: COUCHDB-976
> URL: https://issues.apache.org/jira/browse/COUCHDB-976
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Right now we use a single couch_file server for both updating a database and reading from a database.
> This is a contention point, as concurrent read/write access to a database implies having processes waiting for access to the couch_file server.
> The following patches add a couch_file server that is used only by the DB updater process and another couch_file meant to be used by anyone else only for read operations:
> https://github.com/fdmanana/couchdb/compare/updater_dedicated_fd
> Some performance measurements:
> # updater_fd vs trunk (small docs, 1Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_small_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a616
> # updater_fd vs trunk (large docs, 100Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_large_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300 --doc large
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a7a7
> We can see that both the response time and throughput gets significantly better for both read and writes.
> If no objections I'll commit it to trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-976) Improve database read and write
performance using 2 couch_files
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12967028#action_12967028 ]
Randall Leeds commented on COUCHDB-976:
---------------------------------------
Yeah, you're right. The gains are there.
As for the caching, I think maybe you're right that we should leave it out. It was a workaround for the same problem this addresses: too many reads blocking the write path on couch_file. Problem solved. Great work, Filipe. Thanks for doing this patch.
> Improve database read and write performance using 2 couch_files
> ---------------------------------------------------------------
>
> Key: COUCHDB-976
> URL: https://issues.apache.org/jira/browse/COUCHDB-976
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Right now we use a single couch_file server for both updating a database and reading from a database.
> This is a contention point, as concurrent read/write access to a database implies having processes waiting for access to the couch_file server.
> The following patches add a couch_file server that is used only by the DB updater process and another couch_file meant to be used by anyone else only for read operations:
> https://github.com/fdmanana/couchdb/compare/updater_dedicated_fd
> Some performance measurements:
> # updater_fd vs trunk (small docs, 1Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_small_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a616
> # updater_fd vs trunk (large docs, 100Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_large_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300 --doc large
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a7a7
> We can see that both the response time and throughput gets significantly better for both read and writes.
> If no objections I'll commit it to trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-976) Improve database read and write
performance using 2 couch_files
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966906#action_12966906 ]
Randall Leeds commented on COUCHDB-976:
---------------------------------------
Actually, I read the graphs wrong.
I keep accidentally comparing the reads to the writes thinking I'm comparing trunk to the patch. Oops.
Can't actually tell if it's significantly better, but I'll try to run some tests myself because I expect it to be.
I think the difference will be more dramatic when you crank the number of clients up much higher.
> Improve database read and write performance using 2 couch_files
> ---------------------------------------------------------------
>
> Key: COUCHDB-976
> URL: https://issues.apache.org/jira/browse/COUCHDB-976
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Right now we use a single couch_file server for both updating a database and reading from a database.
> This is a contention point, as concurrent read/write access to a database implies having processes waiting for access to the couch_file server.
> The following patches add a couch_file server that is used only by the DB updater process and another couch_file meant to be used by anyone else only for read operations:
> https://github.com/fdmanana/couchdb/compare/updater_dedicated_fd
> Some performance measurements:
> # updater_fd vs trunk (small docs, 1Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_small_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a616
> # updater_fd vs trunk (large docs, 100Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_large_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300 --doc large
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a7a7
> We can see that both the response time and throughput gets significantly better for both read and writes.
> If no objections I'll commit it to trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-976) Improve database read and write
performance using 2 couch_files
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966854#action_12966854 ]
Randall Leeds commented on COUCHDB-976:
---------------------------------------
Wonderful!!
Your results confirm all my expectations about this patch.
It looks great to me.
I would keep your ets-based cache patch in mind and be sure that it will be easy enough to share one cache even with two FDs, but otherwise I have zero concerns.
> Improve database read and write performance using 2 couch_files
> ---------------------------------------------------------------
>
> Key: COUCHDB-976
> URL: https://issues.apache.org/jira/browse/COUCHDB-976
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Right now we use a single couch_file server for both updating a database and reading from a database.
> This is a contention point, as concurrent read/write access to a database implies having processes waiting for access to the couch_file server.
> The following patches add a couch_file server that is used only by the DB updater process and another couch_file meant to be used by anyone else only for read operations:
> https://github.com/fdmanana/couchdb/compare/updater_dedicated_fd
> Some performance measurements:
> # updater_fd vs trunk (small docs, 1Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_small_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a616
> # updater_fd vs trunk (large docs, 100Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_large_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300 --doc large
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a7a7
> We can see that both the response time and throughput gets significantly better for both read and writes.
> If no objections I'll commit it to trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-976) Improve database read and write
performance using 2 couch_files
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12967228#action_12967228 ]
Filipe Manana commented on COUCHDB-976:
---------------------------------------
Adam,
Look when T is:
- between 50 000 and 75 000, most points have differences of about 100ms (or slightly more)
- when T > 200 000
In the last result I posted (the shorter one of 120secs, easier to analyse) when T > 110 000 for e.g. or when T is between 20 000 and 30 000 (between 50ms and 80ms less than trunk)
> Improve database read and write performance using 2 couch_files
> ---------------------------------------------------------------
>
> Key: COUCHDB-976
> URL: https://issues.apache.org/jira/browse/COUCHDB-976
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Right now we use a single couch_file server for both updating a database and reading from a database.
> This is a contention point, as concurrent read/write access to a database implies having processes waiting for access to the couch_file server.
> The following patches add a couch_file server that is used only by the DB updater process and another couch_file meant to be used by anyone else only for read operations:
> https://github.com/fdmanana/couchdb/compare/updater_dedicated_fd
> Some performance measurements:
> # updater_fd vs trunk (small docs, 1Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_small_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a616
> # updater_fd vs trunk (large docs, 100Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_large_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300 --doc large
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a7a7
> We can see that both the response time and throughput gets significantly better for both read and writes.
> If no objections I'll commit it to trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-976) Improve database read and write
performance using 2 couch_files
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12967218#action_12967218 ]
Adam Kocoloski commented on COUCHDB-976:
----------------------------------------
Hi Filipe, I can't see gains of the magnitude you describe in this ticket on those graphs. A 100ms improvement in reader response time for small docs? I just don't see it.
Nevertheless, the patch is a good idea.
> Improve database read and write performance using 2 couch_files
> ---------------------------------------------------------------
>
> Key: COUCHDB-976
> URL: https://issues.apache.org/jira/browse/COUCHDB-976
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Right now we use a single couch_file server for both updating a database and reading from a database.
> This is a contention point, as concurrent read/write access to a database implies having processes waiting for access to the couch_file server.
> The following patches add a couch_file server that is used only by the DB updater process and another couch_file meant to be used by anyone else only for read operations:
> https://github.com/fdmanana/couchdb/compare/updater_dedicated_fd
> Some performance measurements:
> # updater_fd vs trunk (small docs, 1Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_small_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a616
> # updater_fd vs trunk (large docs, 100Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_large_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300 --doc large
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a7a7
> We can see that both the response time and throughput gets significantly better for both read and writes.
> If no objections I'll commit it to trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-976) Improve database read and write
performance using 2 couch_files
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12967029#action_12967029 ]
Filipe Manana commented on COUCHDB-976:
---------------------------------------
Thanks.
A relaximation test for 2 minutes (instead of 5) helps out verifying the gains, since point samples are not so close to each other:
http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104b99e
regards
> Improve database read and write performance using 2 couch_files
> ---------------------------------------------------------------
>
> Key: COUCHDB-976
> URL: https://issues.apache.org/jira/browse/COUCHDB-976
> Project: CouchDB
> Issue Type: Improvement
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Right now we use a single couch_file server for both updating a database and reading from a database.
> This is a contention point, as concurrent read/write access to a database implies having processes waiting for access to the couch_file server.
> The following patches add a couch_file server that is used only by the DB updater process and another couch_file meant to be used by anyone else only for read operations:
> https://github.com/fdmanana/couchdb/compare/updater_dedicated_fd
> Some performance measurements:
> # updater_fd vs trunk (small docs, 1Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_small_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a616
> # updater_fd vs trunk (large docs, 100Kb each)
> $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
> -name1 updater_fd_large_docs -name2 trunk \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 300 --doc large
> http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a7a7
> We can see that both the response time and throughput gets significantly better for both read and writes.
> If no objections I'll commit it to trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.