You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Adam Kocoloski (JIRA)" <ji...@apache.org> on 2010/05/14 17:21:41 UTC
[jira] Created: (COUCHDB-762) Faster implementation of
couch_file:pread_iolist
Faster implementation of couch_file:pread_iolist
------------------------------------------------
Key: COUCHDB-762
URL: https://issues.apache.org/jira/browse/COUCHDB-762
Project: CouchDB
Issue Type: Improvement
Components: Database Core
Affects Versions: 0.11
Environment: any
Reporter: Adam Kocoloski
Priority: Minor
Fix For: 1.1
couch_file's pread_iolist function is used every time we read anything from disk. It makes 2-3 gen_server calls to the couch_file process to do its work.
This patch moves the work done by the read_raw_iolist function into the gen_server itself and adds a pread_iolist handler. This means that one gen_server call is sufficient in every case.
Here are some benchmarks comparing the current method with the patch that reduces everything to one call. I write a number of 10k binaries to a file, then read them back in a random order from 1/5/10/20 concurrent reader processes. I report the median/90/95/99 percentile response times in microseconds. In almost every case the patch is an improvement.
The data was fully cached for these tests; I think that in a real-world concurrent reader scenario the performance improvement may be greater. The patch ensures that the 2-3 pread calls reading sequential bits of data (term length, MD5, and term) are always submitted without interruption. Previously, two concurrent readers could race to read different terms and cause some extra disk head movement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-762) Faster implementation of
couch_file:pread_iolist
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski updated COUCHDB-762:
-----------------------------------
Attachment: 762-pread_iolist-v2.patch
An even better patch which does exactly 2 pread() calls in all cases, even for MD5-prefixed terms. Here are updated timings, with this approach termed 'pread_iolist_3':
4> pread_iolist_bench:go(5000, 10000, 1, pread_iolist).
Median 96
90% 103
95% 109
99% 153
ok
5> pread_iolist_bench:go(5000, 10000, 1, pread_iolist2).
Median 82
90% 90
95% 94
99% 107
ok
6> pread_iolist_bench:go(5000, 10000, 1, pread_iolist3).
Median 71
90% 78
95% 81
99% 93
ok
> Faster implementation of couch_file:pread_iolist
> ------------------------------------------------
>
> Key: COUCHDB-762
> URL: https://issues.apache.org/jira/browse/COUCHDB-762
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Environment: any
> Reporter: Adam Kocoloski
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 762-pread_iolist-v2.patch, 762-pread_iolist.patch, patch-to-reproduce-benchmarks.txt, pread_iolist_bench.erl, pread_iolist_results.txt
>
>
> couch_file's pread_iolist function is used every time we read anything from disk. It makes 2-3 gen_server calls to the couch_file process to do its work.
> This patch moves the work done by the read_raw_iolist function into the gen_server itself and adds a pread_iolist handler. This means that one gen_server call is sufficient in every case.
> Here are some benchmarks comparing the current method with the patch that reduces everything to one call. I write a number of 10k binaries to a file, then read them back in a random order from 1/5/10/20 concurrent reader processes. I report the median/90/95/99 percentile response times in microseconds. In almost every case the patch is an improvement.
> The data was fully cached for these tests; I think that in a real-world concurrent reader scenario the performance improvement may be greater. The patch ensures that the 2-3 pread calls reading sequential bits of data (term length, MD5, and term) are always submitted without interruption. Previously, two concurrent readers could race to read different terms and cause some extra disk head movement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-762) Faster implementation of
couch_file:pread_iolist
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867602#action_12867602 ]
Adam Kocoloski commented on COUCHDB-762:
----------------------------------------
Just a comment about the breakdown of time spent reading a term. We're looking at median response times of 70 µs to read a 10K binary. I think this is roughly distributed as
crypto:md5 - 30 µs
pread()*2 - 20 µs
everything else - µs
> Faster implementation of couch_file:pread_iolist
> ------------------------------------------------
>
> Key: COUCHDB-762
> URL: https://issues.apache.org/jira/browse/COUCHDB-762
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Environment: any
> Reporter: Adam Kocoloski
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 762-pread_iolist-v2.patch, 762-pread_iolist.patch, patch-to-reproduce-benchmarks.txt, pread_iolist_bench.erl, pread_iolist_results.txt
>
>
> couch_file's pread_iolist function is used every time we read anything from disk. It makes 2-3 gen_server calls to the couch_file process to do its work.
> This patch moves the work done by the read_raw_iolist function into the gen_server itself and adds a pread_iolist handler. This means that one gen_server call is sufficient in every case.
> Here are some benchmarks comparing the current method with the patch that reduces everything to one call. I write a number of 10k binaries to a file, then read them back in a random order from 1/5/10/20 concurrent reader processes. I report the median/90/95/99 percentile response times in microseconds. In almost every case the patch is an improvement.
> The data was fully cached for these tests; I think that in a real-world concurrent reader scenario the performance improvement may be greater. The patch ensures that the 2-3 pread calls reading sequential bits of data (term length, MD5, and term) are always submitted without interruption. Previously, two concurrent readers could race to read different terms and cause some extra disk head movement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (COUCHDB-762) Faster implementation of
couch_file:pread_iolist
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski closed COUCHDB-762.
----------------------------------
Resolution: Fixed
applied v2 of the patch
> Faster implementation of couch_file:pread_iolist
> ------------------------------------------------
>
> Key: COUCHDB-762
> URL: https://issues.apache.org/jira/browse/COUCHDB-762
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Environment: any
> Reporter: Adam Kocoloski
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 762-pread_iolist-v2.patch, 762-pread_iolist.patch, patch-to-reproduce-benchmarks.txt, pread_iolist_bench.erl, pread_iolist_results.txt
>
>
> couch_file's pread_iolist function is used every time we read anything from disk. It makes 2-3 gen_server calls to the couch_file process to do its work.
> This patch moves the work done by the read_raw_iolist function into the gen_server itself and adds a pread_iolist handler. This means that one gen_server call is sufficient in every case.
> Here are some benchmarks comparing the current method with the patch that reduces everything to one call. I write a number of 10k binaries to a file, then read them back in a random order from 1/5/10/20 concurrent reader processes. I report the median/90/95/99 percentile response times in microseconds. In almost every case the patch is an improvement.
> The data was fully cached for these tests; I think that in a real-world concurrent reader scenario the performance improvement may be greater. The patch ensures that the 2-3 pread calls reading sequential bits of data (term length, MD5, and term) are always submitted without interruption. Previously, two concurrent readers could race to read different terms and cause some extra disk head movement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-762) Faster implementation of
couch_file:pread_iolist
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski updated COUCHDB-762:
-----------------------------------
Attachment: pread_iolist_bench.erl
Benchmarking script
> Faster implementation of couch_file:pread_iolist
> ------------------------------------------------
>
> Key: COUCHDB-762
> URL: https://issues.apache.org/jira/browse/COUCHDB-762
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Environment: any
> Reporter: Adam Kocoloski
> Priority: Minor
> Fix For: 1.1
>
> Attachments: patch-to-reproduce-benchmarks.txt, pread_iolist_bench.erl, pread_iolist_results.txt
>
>
> couch_file's pread_iolist function is used every time we read anything from disk. It makes 2-3 gen_server calls to the couch_file process to do its work.
> This patch moves the work done by the read_raw_iolist function into the gen_server itself and adds a pread_iolist handler. This means that one gen_server call is sufficient in every case.
> Here are some benchmarks comparing the current method with the patch that reduces everything to one call. I write a number of 10k binaries to a file, then read them back in a random order from 1/5/10/20 concurrent reader processes. I report the median/90/95/99 percentile response times in microseconds. In almost every case the patch is an improvement.
> The data was fully cached for these tests; I think that in a real-world concurrent reader scenario the performance improvement may be greater. The patch ensures that the 2-3 pread calls reading sequential bits of data (term length, MD5, and term) are always submitted without interruption. Previously, two concurrent readers could race to read different terms and cause some extra disk head movement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-762) Faster implementation of
couch_file:pread_iolist
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski updated COUCHDB-762:
-----------------------------------
Attachment: pread_iolist_results.txt
Benchmark results
> Faster implementation of couch_file:pread_iolist
> ------------------------------------------------
>
> Key: COUCHDB-762
> URL: https://issues.apache.org/jira/browse/COUCHDB-762
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Environment: any
> Reporter: Adam Kocoloski
> Priority: Minor
> Fix For: 1.1
>
> Attachments: patch-to-reproduce-benchmarks.txt, pread_iolist_results.txt
>
>
> couch_file's pread_iolist function is used every time we read anything from disk. It makes 2-3 gen_server calls to the couch_file process to do its work.
> This patch moves the work done by the read_raw_iolist function into the gen_server itself and adds a pread_iolist handler. This means that one gen_server call is sufficient in every case.
> Here are some benchmarks comparing the current method with the patch that reduces everything to one call. I write a number of 10k binaries to a file, then read them back in a random order from 1/5/10/20 concurrent reader processes. I report the median/90/95/99 percentile response times in microseconds. In almost every case the patch is an improvement.
> The data was fully cached for these tests; I think that in a real-world concurrent reader scenario the performance improvement may be greater. The patch ensures that the 2-3 pread calls reading sequential bits of data (term length, MD5, and term) are always submitted without interruption. Previously, two concurrent readers could race to read different terms and cause some extra disk head movement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-762) Faster implementation of
couch_file:pread_iolist
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski updated COUCHDB-762:
-----------------------------------
Attachment: 762-pread_iolist.patch
> Faster implementation of couch_file:pread_iolist
> ------------------------------------------------
>
> Key: COUCHDB-762
> URL: https://issues.apache.org/jira/browse/COUCHDB-762
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Environment: any
> Reporter: Adam Kocoloski
> Priority: Minor
> Fix For: 1.1
>
> Attachments: 762-pread_iolist.patch, patch-to-reproduce-benchmarks.txt, pread_iolist_bench.erl, pread_iolist_results.txt
>
>
> couch_file's pread_iolist function is used every time we read anything from disk. It makes 2-3 gen_server calls to the couch_file process to do its work.
> This patch moves the work done by the read_raw_iolist function into the gen_server itself and adds a pread_iolist handler. This means that one gen_server call is sufficient in every case.
> Here are some benchmarks comparing the current method with the patch that reduces everything to one call. I write a number of 10k binaries to a file, then read them back in a random order from 1/5/10/20 concurrent reader processes. I report the median/90/95/99 percentile response times in microseconds. In almost every case the patch is an improvement.
> The data was fully cached for these tests; I think that in a real-world concurrent reader scenario the performance improvement may be greater. The patch ensures that the 2-3 pread calls reading sequential bits of data (term length, MD5, and term) are always submitted without interruption. Previously, two concurrent readers could race to read different terms and cause some extra disk head movement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-762) Faster implementation of
couch_file:pread_iolist
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski updated COUCHDB-762:
-----------------------------------
Attachment: patch-to-reproduce-benchmarks.txt
This is not the patch I would submit, but rather one that exports both new and old methods for benchmarking.
> Faster implementation of couch_file:pread_iolist
> ------------------------------------------------
>
> Key: COUCHDB-762
> URL: https://issues.apache.org/jira/browse/COUCHDB-762
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 0.11
> Environment: any
> Reporter: Adam Kocoloski
> Priority: Minor
> Fix For: 1.1
>
> Attachments: patch-to-reproduce-benchmarks.txt, pread_iolist_results.txt
>
>
> couch_file's pread_iolist function is used every time we read anything from disk. It makes 2-3 gen_server calls to the couch_file process to do its work.
> This patch moves the work done by the read_raw_iolist function into the gen_server itself and adds a pread_iolist handler. This means that one gen_server call is sufficient in every case.
> Here are some benchmarks comparing the current method with the patch that reduces everything to one call. I write a number of 10k binaries to a file, then read them back in a random order from 1/5/10/20 concurrent reader processes. I report the median/90/95/99 percentile response times in microseconds. In almost every case the patch is an improvement.
> The data was fully cached for these tests; I think that in a real-world concurrent reader scenario the performance improvement may be greater. The patch ensures that the 2-3 pread calls reading sequential bits of data (term length, MD5, and term) are always submitted without interruption. Previously, two concurrent readers could race to read different terms and cause some extra disk head movement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.